As a database grows very large,
storing and managing it may become too demanding for any single node.
System performance may suffer as resources like RAM, CPU, and storage are
exhausted, routine chores like indexing and backup become massive tasks,
responsiveness to client requests and queries slows down, and the system's
throughput spreads thin serving an ever-growing number of clients.
As the volume of stored data grows, the database can be scaled out by
splitting it into shards, allowing it to be
handled by multiple nodes and presenting practically no limit to its growth.
The size of the overall database, comprised of all shards, can reach in
this fashion dozens of terabytes and more while keeping the resources
of each shard in check and maintaining its high performance and throughput.
As a client connects a sharded database, it is appointed a RavenDB server
that functions as an orchestrator and mediates all the communication
between the client and the database shards.
The client remains unaware of this process and uses the same API used by
non-sharded databases to load documents, query, and so on.
The additional communication between the client and the orchestrator and
between the orchestrator and the shards does, however, present an overhead
over the usage of a non-sharded database.
When Should Sharding Be Used?
While sharding solves many issues related to the storage and management
of high-volume databases, the overhead it presents outweighs its benefits
when the database size still poses no problem. We can postpone the transit
to a sharded database when, for example, the database size is 100 GB, the
server is well equipped and would comfortably handle a much larger volume,
and no dramatic increase is expected in the number of potential users
any time soon.
We recommend that you plan ahead for a transition to a sharded database when
your database size is in the vicinity of 250 GB, so the transition is already well
established when it reaches 500 GB.
RavenDB 6.0 and above can migrate its database to a sharded database
via external replication
or export & import operations.
You cannot, however, upgrade a non-sharded database into a sharded one.
To upgrade RavenDB to 6.0 and migrate the database data you will need
to upgrade the server, create a new, sharded database, and replicate or
export the data into it.