How to troubleshoot RavenDB’s high CPU usage

What you will learn

• Common high CPU usage problem causes
• Difference between Current CPU Usage and Total CPU Usage Time
• How to use tools to investigate your problem of high CPU usage

Introduction

High CPU usage can be a challenging issue for anyone. Whether you noticed it on the RavenDB dashboard or started investigating why your tasks take so long to finish. In this article, we will do our best to find the cause. Whether your issue is with Current CPU Usage or Total CPU Usage, this guide can help you identify the root of the problem or give you the tools to do it yourself.

Common Causes

Before we attempt to detect the reason for your server’s high CPU usage, let’s check if one of the following common cases isn’t happening.

Indexing Workload

During indexing, your CPU usage can skyrocket if you set up many indexing requests for your database. Review your index definitions to find problems in the code itself.

Creating a backup

If your database periodically experiences high CPU usage, periodic backups may be a factor causing these problems. Consider changing the frequency of your backups. You can check your database backup options at ‘Tasks’ and ‘Backups’.

External factors and bugs

Bugs in your connected application or other server processes can influence your CPU usage. Check your machine’s other running processes and applications.

Current CPU Usage and Total CPU Time

Now let’s break down what we’re looking for through the analysis. Two metrics will be helpful for us: Current CPU Usage and Total CPU Time. 

Current CPU Usage (In RavenDB called Current CPU %)\ informs us of the percentage CPU on your machine used by our processes. This metric is continuously monitored and updated. It is helpful in detecting “spikes” of CPU usage (short-term usage bursts). In extreme cases, it will also clearly show overall high usage.

On the other hand, Total CPU Time (Overall CPU Time) is a metric that informs us about CPU usage in the long term. It measures the time consumed by the process since it started. Depending on the tool, it can be as precise as measuring milliseconds of CPU work. Since it is calculated from the start of the process, it can be used to monitor the overall performance of your server.

As we can see, they are two different metrics for different uses. Both complement each other, allowing us to investigate the database server performance.

Investigating

Depending on if you are hosting your database on your machine or you are using RavenDB Cloud, you can monitor your CPU using different tools. 

Linux

If you are running your machine on Linux, you can do it with two programs. Both are launched using the terminal and its top -H and htop -H. -H  immediately launches those programs in threads mode. If you forgot to add -H, just press SHIFT+H to switch it. In this article, we are using Ubuntu-based Linux. If you are using a different Linux distribution, please check what commands will work for you.

Top

The top command is a part of the procps package, a common component of Linux systems. Type top in the terminal to check if it’s already there. 

If it’s not, let’s break down how to install it. First, update your packages. You can do this with the following command:

sudo apt update && sudo apt upgrade

Then, install the procps package by executing the following command (use sudo to elevate permissions):

sudo apt install procps

As mentioned previously, this will give you access to the top program, but you want to use top -H. -H allows you to view threads instead of full processes. -H can be triggered with a command, or you can press SHIFT+H to change it when the application is already running.

Htop

Command htop is a bit more advanced and easy to read. It’s not a part of the procps package, it’s installed separately. Update your packages if you haven’t done that recently:

sudo apt update && sudo apt upgrade

Now, you can install htop by executing the following command:

sudo apt install htop

This will download and install the htop on your machine. It is a bit more comprehensive than top so you might prefer it. As mentioned before, use the command htop -H to check threads instead of just processes, or press SHIFT+H to change it when the application is already running.

Investigating on Linux

Let’s continue the investigation. Try top and htop to find out which interface suits you better, as both give most of the information we need… 

top -H
htop -H

Take a look at two columns (1 and 2). First, you have %CPU (top) or CPU% (htop). Current CPU Usage is expressed in percentage. 

TIME+ indicates Total CPU Time. Additionally, htop shows CPU usage of every core of your CPU  at the top of your window. 

Look at the pictures and notice high CPU usage—something CPU-demanding is happening right now. However, in the COMMAND (3) column, we can see that I set up some indexing (Idx is the indexing thread) for example purposes. This way, using standard tooling, we can properly investigate the cause of our problem and conclude

Windows 

Sadly, Windows has no easy way to access specific thread information. We have access to threads in Task Manager and Resource Manager, but those two are too imprecise, and usage is too time-consuming to be worth it. Your best shot is to use the RavenDB tools explained below. They give you easily readable and complete information about your database. You can also search for a third-party tool suited to your needs.

RavenDB

RavenDB has ready-to-go solutions: use RavenDB’s Studio widgets. We have prepared a dashboard where you can use widgets to monitor your database. 

The ‘CPU’ widget monitors CPU usage and is the first place to spot potential issues.

It shows the current CPU usage plot, which informs us about current and overall performance. This widget, combined with others like traffic or indexing, can help you find problematic threads.

Let’s gather more information about your CPU usage and threads. Select ‘Manage Server’ on the left bar and find two useful tools there. The first tool you can use for more precise analysis can be found in the ‘Manage Server’ category on the left bar. It’s called ‘Advanced’.

We are interested in the ‘Thread Runtime Info’ section that was selected by default. It gives us access to the useful chart of our threads.

Here you can find two useful columns, just like in top commands. ‘Current CPU %’ (1) and ‘Overall CPU Time’ (2). In the screenshot above, you can see that Current CPU Usage is high. Looking at the name in the left column, you can say that I set up indexing to create this example. You can also notice that it may be eating a lot of CPU currently, but Overall CPU Time shows that it’s not using much in the long term.

The second tool you can use is Stack Traces. This tool allows you to analyze your data visually. To get there, you again choose ‘Manage Server’ on the left bar and select ‘Stack Traces’.

To use Stack Traces, you need to click ‘Capture’. After a short wait, you can see the thread tree. This tree cannot show you your Overall CPU Time, but it can benefit you in other ways. It shows all connections more precisely for the human eye and allows you to not only find the root of the problem but also to find bottlenecks or where many threads create one. If you need, you can even zoom in and precisely see what each box is doing. For example:

This gives you the exact percentage of what the stack does so you can analyze it. This case is the same as previous ones. The cause of high CPU usage is threads doing indexing. You can notice that by looking at the gray text at the top of two top stacks. Because it’s indexing, I can review my indexing definitions or just wait for them to finish, so it will stop using this much CPU.

Summary

This article explored ways to identify your CPU problems, introducing common issues that users can experience. We also explained tools like top, htop, and in-built RavenDB tools for quick and efficient CPU usage checks.

The table below summarizes tools you can use for your investigation.

For Linux.
Type Top -H in console		Type Htop -H in console
For Windows
Use RavenDB tools. Default Windows tools are too imprecise.
RavenDB Native Tools
RavenDB Widgets on the main page.	Advanced Table in ‘Manage Server’ options.		Stack Traces in ‘Manage Server’ options.