Both solutions performed admirably when accessing by key for less than 30k requests/sec at almost the same price point. Couchbase has a slight advantage due to keeping all the data in memory, but this comes at a higher cost of vastly increased startup time as well as much higher disk usage. Beyond 30k requests/sec the conclusion is that a deeper investigation and an advanced distributed testing infrastructure would be needed to be done to rule out measurement artifacts.

When indexing is required, the conclusion is that an alternative solution like Elasticsearch would be required when using Couchbase.

Under the recommendation of the RavenDB Performance Team, noticing the extra capacity on the 10k request/sec scenario, a few more tests were run with downsized hardware to understand scaling costs.

Because of how Couchbase is designed, the minimum cloud hardware required to handle the benchmark on AWS is a cluster of 3 nodes of m5a.8xlarge instances (32 cores and 128 GB of RAM) with the data sharded among the nodes, without replication. That isn’t a viable production configuration, of course.

Looking at the sizing guide for Couchbase, and taking into account that a production environment cannot run with a single copy of the data, we estimate that a production cluster to serve this scenario would take 5 servers with 192 GB each. This assumes a working set of 20% of the data to reside in memory and three replicas for each document.

On premise, adding more RAM has a rather marginal cost, but on the cloud the relevant instance for the requirement is m5a.12xlarge, which comes at a 48% premium. A cluster of 5 such machines would have a total of 240 cores and 960 GB of RAM and a total disk usage of 30 TB.

The recommended setup for production would be bigger than the one showcased here and it would cost significantly more as well. When selecting a database solution, one does not look simply at the performance numbers, but also at what resources it takes to achieve them.

This benchmark puts a very high bar for passing. Handling thousands of queries per second with low latency is a load very few applications need to face. Therefore, we decided to see how far we could step down the hardware requirements for RavenDB and what would be the acceptable performance at each price point.

For RavenDB we selected the default setup for the cluster as reference which matches Couchbase performance to compare against: m5a.xlarge (4 cores, 16GB) and a Graviton ARM m6g.large (2 cores, 8GB). (See Fig. 10)

Fig. 10 Latency distribution in milliseconds at 10k requests/sec for get-by-id query (lower is better)

Fig. 10 Latency distribution in milliseconds at 10k requests/sec for get-by-id query (lower is better)

RavenDB is able to handle gracefully 10,000 requests per second at the 99-percentile with a latency below 200ms on a 2TB database using an ARM Graviton processor (2 cores, 8 GB of RAM) at an annualized cost of $1,185 + storage costs for the entire cluster.

For more performance conscious applications at the 99.9-percentile, a cluster composed of three m5a.xlarge nodes is able to serve 10,000 requests per second with latencies below 110ms at an annualized cost of $2,658 + storage costs. The reference cluster would cost $24,699 per year without including storage cost at comparable latency.

For the cases of the Books and Users highlight queries, comparisons were performed against a RavenDB reference cluster. Couchbase was not able to complete a test run at 250 requests per second without timeouts.

If the solution requires the usage of queries, there is not enough spare capacity at the 1,000 requests per second to be able to downsize the environment up to 2 cores Graviton range (See Fig. 11). However, as shown in the mid size cluster, it is possible to sustain up to 1,000 requests per second within the threshold even using queries. The memory size and limited core count of the smallest Graviton cluster cannot sustain 1k request/sec.

Fig. 11 RavenDB latency distribution in milliseconds at 1k requests/sec for users query (lower is better)

Fig. 11 RavenDB latency distribution in milliseconds at 1k requests/sec for users query (lower is better)

The absolute downsizing limit for half the load at 500 requests per second was found to be even lower than the ARM system. The system which was used for comparison in this report was a single Raspberry PI 4 with 4GB and a USB connected SSD. At that reference load, it is able to sustain at the 99-percentile a latency well under 50ms as shown below. (See Fig. 12)

Fig. 12 RavenDB latency distribution in milliseconds at 500 requests/sec for users query (lower is better)

Fig. 12 RavenDB latency distribution in milliseconds at 500 requests/sec for users query (lower is better)

Couchbase cluster of three nodes with a total of 96 cores and 384 GB RAM was unable to sustain 250 queries per second. A single RavenDB node running on a Raspberry PI 4 with 4 cores and 4 GB of RAM was able to answer 500 queries / second in under 200ms in the 99 percentile.

If the system can be engineered to ensure that requests are dominated by high performance prefix queries, even at the smallest cluster composed with nodes of 2 cores with 8GB of memory, the querying system can sustain 10,000 requests per second within the 200ms threshold for the 99-percentile. (See Fig. 13)

Fig. 13 RavenDB latency distribution in milliseconds at 10k requests/sec for books query (lower is better)

Fig. 13 RavenDB latency distribution in milliseconds at 10k requests/sec for books query (lower is better)

Contents