HiLo Algorithm

Generating unique IDs efficiently:

The client creates IDs from a range of unique numbers that it gets from the server.
The HiLo algorithm is efficient because the client can automatically generate unique document IDs without checking with the server or cluster each time a new document is created to ensure that the new ID is unique.
The client receives from the server a range of numbers that are reserved for the client's usage.

Each time a session creates a new document, the client assigns the new document an ID based on the next number from that range. For example, the first client to generate documents on a collection will receive the reserved numbers 1-32. The next one will reserve numbers 33-64, and so on.

The collection name and node-tag are added to the ID.
To further ensure that no two clients generate a document with the same ID, the collection name and the server node-tag are added to the ID. This is an added measure so that if two nodes B and C are working with the same range of numbers, the IDs generated will be orders/54-B and orders/54-C.
This situation is rare because as long as the nodes can communicate when requesting a range of numbers, the clients will receive a different range of numbers. The node-tag is added to ensure unique IDs across the cluster.

Thus, with minimal trips to the server, the client is able to determine to which collection an entity belongs
and automatically assign it a number with a node-tag to ensure that the ID is unique across the cluster.

Using HiLo documents:

HiLo documents are used by the server to provide the next range of numbers.
To ensure that multiple clients can generate identifiers simultaneously without producing duplicates,
a mechanism is needed to avoid duplication.

This is handled by Raven/HiLo/<collection> documents, stored in the @hilo collection in the database.
These documents are created and modified by the server and have a simple structure:

{
    "Max": 32,
    "@metadata": {
        "@collection": "@hilo"
    }
}

The Max property means the maximum possible number that has been used by any client to create the identifier for a given collection. It is used as follows:

1. The client asks the server for a range of numbers that it can use to create a document.
   (32 is the initial capacity, but the range size can dynamically expand based on how frequently the client requests HiLo ranges).
2. Then, the server checks the HiLo file to see what is the last "Max" number it sent to any client for this collection.
3. The client will get the min and the max values it can use from the server (33 - 64 in our case).
4. Then, the client creates a range object using the values received from the server.
   This range object is then used to generate unique document IDs as needed.
5. When the client reaches the max limit, it will repeat the process.

When the document store is disposed, the client sends the server the last value it used to create an identifier and the max value that was previously received from the server.

If the max value on the server-side is equal to the max value of the client and the last used value by the client is smaller or equal to the max of the server-side, the server will update the Max value to the last used value by the client.

var store = new DocumentStore();

using (var session = store.OpenSession())
{
    // Storing the first entity causes the client to receive the initial HiLo range (1-32)
    session.Store(new Employee
    {
        FirstName = "John",
        LastName = "Doe"
    });

    session.SaveChanges(); 
    // The document ID will be: employees/1-A
}

// Release the range when it is no longer relevant
store.Dispose();

store.Dispose() is used in this example to demonstrate that the range is released.
In normal use, the store should only be disposed when the application is closed.

After execution of the code above, the Max value of the Hilo document for the Employees collection in the server will be 1. That's because the client used only one identifier from the range it got before we disposed the store.

The next time that a client asks for a range of numbers from the server for this collection it will get (in our example) the range 2 - 33.

var newStore = new DocumentStore();
using (var session = newStore.OpenSession())
{
    // Storing an entity after disposing the store in the previous example
    // causes the client to receive the next HiLo range (2-33)
    session.Store(new Employee
    {
        FirstName = "Dave",
        LastName = "Brown"
    });

    session.SaveChanges();
    // The document ID will be: employees/2-A
}

• Automatic generation:
  When the session stores a new document with the Id set to null, RavenDB's default HiLo ID generator automatically generates the ID for the document. This document ID includes the collection name, a unique number, and the server node-tag, ensuring the ID is unique across the database.

• Manual generation:
  We provide you with the option of manually retrieving the next ID from the HiLo range currently reserved for the client without having to store the document first. You can retrieve either the next number portion or the full document ID and then use it when storing the document, as explained below:

  • Get next ID - number only
  • Get next ID - full document ID

Get next ID - number only

You can take advantage of the HiLo algorithm and create documents with your own customized ID that is based on the next HiLo ID number provided by the client.

Syntax:

Either one of the following overloads will return the next available ID from the HiLo numbers reserved for the client. The returned ID number can then be used when storing a new document.

Task<long> GenerateNextIdForAsync(string database, object entity);

Task<long> GenerateNextIdForAsync(string database, Type type);

Task<long> GenerateNextIdForAsync(string database, string collectionName);

Example:

The following example shows how to get the next ID number from the HiLo range reserved for the client.
The ID provided is the next unique number without the node tag and the collection.
This ID is then used to create and store a new document.

Calling GenerateNextIdForAsync ensures minimal calls to the server,
as the ID is generated by the client from the reserved range of numbers.

using (var session = store.OpenSession())
{
    // Use any overload to get the next id:
    // (Note how the id increases with each call)
    // ==========================================
    
    var nextId = await store.HiLoIdGenerator.GenerateNextIdForAsync(null, "Products");
    // nextId = 1
    
    nextId = await store.HiLoIdGenerator.GenerateNextIdForAsync(null, new Product());
    // nextId = 2
    
    nextId = await store.HiLoIdGenerator.GenerateNextIdForAsync(null, typeof(Product));
    // nextId = 3

    // Now you can create a new document with the nextId received
    // ==========================================================
    
    var product = new Product
    {
        Id = "MyCustomId/" + nextId.ToString()
    };
    
    // Store the new document
    // The document ID will be: "MyCustomId/3"  
    session.Store(product);
    session.SaveChanges();
}

Get next ID - full document ID

You can request to get the next full document ID from the default HiLo generator without having to store the document first.

Syntax:

Task<string> GenerateDocumentIdAsync(string database, object entity);

Example:

The latest HiLo ID number generated in the example above was 3.
Therefore, when running the following example immediately after, the consecutive number 4 is retrieved and incorporated into the full document ID (products/4-A) that is returned by GenerateDocumentIdAsync.

using (var session = store.OpenSession())
{
    var nextFullId = await store.HiLoIdGenerator.GenerateDocumentIdAsync(null, "Products");
    // nextFullId = "products/4-A"

    // You can now use the nextFullId and customize the document ID as you wish:
    var product = new Product
    {
        Id = "MyCustomId/" + nextFullId
    };
    
    session.Store(product);
    session.SaveChanges();
}

• RavenDB's default HiLo generator is managed by the HiLoIdGenerator property in your DocumentStore object.

• If needed, you can override this default ID generation behavior by setting the AsyncDocumentIdGenerator convention with your own implementation.

• Once you configure your custom behavior through this convention:

  • Your customized ID generation will be applied whenever you store a document without explicitly specifying an Id.

  • Attempting to call GenerateNextIdForAsync or GenerateDocumentIdAsync via the store's HiLoIdGenerator
    will throw an exception.