Sharding: Overview

As a database grows very large, storing and managing it may become too demanding for any single node.
System performance may suffer as resources like RAM, CPU, and storage are exhausted, routine chores like indexing and backup become massive tasks, responsiveness to client requests and queries slows down, and the system's throughput spreads thin serving an ever-growing number of clients.

As the volume of stored data grows, the database can be scaled out by splitting it into shards, allowing it to be handled by multiple nodes and presenting practically no limit to its growth.
The size of the overall database, comprised of all shards, can reach in this fashion dozens of terabytes and more while keeping the resources of each shard in check and maintaining its high performance and throughput.

Licensing

• On a Developer license, the replication factor is restricted to 1.
• On Community and Professional licenses, all shards need to be on the same node.

Learn more about licensing here.

Client-Server Communication

As a client connects a sharded database, it is appointed a RavenDB server that functions as an orchestrator and mediates all the communication between the client and the database shards.
The client remains unaware of this process and uses the same API used by non-sharded databases to load documents, query, and so on.
The additional communication between the client and the orchestrator and between the orchestrator and the shards does, however, present an overhead over the usage of a non-sharded database.

When Should Sharding Be Used?

While sharding solves many issues related to the storage and management of high-volume databases, the overhead it presents outweighs its benefits when the database size still poses no problem. We can postpone the transit to a sharded database when, for example, the database size is 100 GB, the server is well equipped and would comfortably handle a much larger volume, and no dramatic increase is expected in the number of potential users any time soon.

While each cluster node of a non-sharded database handles a full replica of the entire database, each shard is assigned a subset of the entire database content.

As much as clients are concerned a sharded database is still a single entity: the clients are not required to detect whether the database is sharded or not, and clients of RavenDB versions prior to 6.0, which had no sharding support, can access a sharded database unaltered.

Shard-specific operations are, however, available: a client can, for example, track the shard that a document is stored at and query this shard, and Studio can be used to relocate (reshard) documents from one shard to another.

Studio Document View

Shard Replication

Similarly to non-sharded databases, shards can be replicated by cluster nodes to ensure the continuous availability of all shards in case of a node failure, provide multiple access points, and load-balance the traffic between shard replicas.

The number of nodes a shard is replicated to is determined by the Shard Replication Factor.

Shard Replication

• In the image above, a 3-shards database is hosted by a 5-nodes cluster (where two of the nodes, D and E, are unused by this database).
  The Shard Replication Factor is set to 2, maintaining two replicas of each shard.

Documents are stored in a sharded database within virtual containers named Buckets.
The number of documents and the amount of data stored in each bucket may vary.

Buckets Allocation

The number of buckets allocated for the whole database is fixed, always remaining 1,048,576 (1024 times 1024).
Each shard is assigned a range of buckets from this overall portion, in which documents can be stored.

Buckets Allocation

Buckets Population

Buckets are populated with documents automatically by the cluster.
A hash algorithm is executed over each document ID. The resulting hash code, a number between 0 and 1,048,576, is the number of the bucket in which the document is stored.

Buckets Population

As buckets are spread among different shards, the bucket number allocated for a document also determines which shard the document will reside on.

Document Extensions Storage

Document extensions (i.e. Attachments, Time series, Counters, and Revisions) are stored in the same bucket as the document they belong to.
To make this happen, the bucket number (hash code) they are given is calculated by the ID of the document that owns them.

Anchoring Documents to a Bucket

You can make documents share a bucket (and therefore a shard) by adding their ID a suffix by which RavenDB will calculate their bucket number.
Learn here why and how to do this.

Resharding is the relocation of data placed on one shard, on another shard, to maintain a balanced database in which all shards handle about the same volume of data.

The resharding process moves all the data related to a certain bucket, including documents, document extensions, tombstones, etc., to a different shard, and then associates the bucket with the new shard.

From a user's perspective paging is conducted similarly in sharded and non-sharded databases, using the same API.
Paging is more costly in a sharded database, however, since the orchestrator must load data from each shard and sort the retrieved results before handing the selected page to the user.
Read more about this subject here.

The local IP address of a cluster node can be exposed, so other cluster nodes would prioritize it when they access the node. Using a node's local IP address rather than a public one for inter cluster communications can speed up the service and offer substantial savings over time.

Using this method can be particularly helpful in a sharded cluster, since each client request is handled by an orchestrator, that may communicate the request and its results with all other shards.

Use this configuration option to expose a node's local IP address to other nodes.

• A sharded database can be created via Studio or the API.

• A RavenDB cluster can run sharded and non-sharded databases in parallel.

• When a database is created, The user can choose whether it would be sharded or not. The ability to make this choice is provided by RavenDB (6.0 and on) by default, no further steps are required to enable the feature.