Indexes: Stale Indexes

RavenDB performs indexing in the background threads. The indexes start processing whenever the new data comes in, and the existing documents are updated or deleted.

Running them in the background allows the server to return query results immediately even if a large number of documents has been just changed.

However, in that case the index is stale until it processes them.

The notion of stale indexes comes from the close observation of the way RavenDB is designed, and the assumption that the user should never suffer from assigning big tasks to the server. As far as RavenDB is concerned, it is better to be stale than offline, and it will return query results even if it knows they may not be up-to-date.

Checking for Stale Results

The querying response has an isStale method indicating whether the results are stale: that is, whether there were any outstanding tasks against the index at the time of querying.

It can be retrieved using the Statistics method:

let stats;

const results = await session.query(Product)
    .statistics($ => stats = $)
    .whereGreaterThan("PricePerUnit", 10)
    .all();

if (stats.stale) {
    // results are known to be stale
}

When isStale is true, some Product has been just added or changed, and the index didn't have enough time to fully update before the query.

Typically, the index updates the records within milliseconds. But there are scenarios where you cannot work with possibly stale data.

Explicitly Waiting for Non-Stale Results

In order to assure that a query won't return stale results, you can use a few approaches. For each of them the default wait timeout will be 15 seconds unless it was specified differently by the user. It is handled by the server, the client won't send any additional requests meanwhile. If it exceeds the timeout then a TimeoutException will be thrown.

Customizing Single Query

You have an option to instruct that a particular query should wait until the index is up-to-date:

const results = session
    .query(Product)
    .waitForNonStaleResults(5000)
    .whereGreaterThan("PricePerUnit", 10)
    .all();

Customizing All Queries

You can also apply such customization at the document store level so all queries will wait for non-stale results:

store.addSessionListener("beforeQuery", event => {
    event.queryCustomization.waitForNonStaleResults();
});

Waiting for Documents Stored in Session

If you need to ensure the indexes process, the documents stored in the current session before saveChanges returns, you can use:

session.advanced.waitForIndexesAfterSaveChanges();

It will wait for the indexes to catch up with the just saved changes. You can control the behavior and specify indexes you want to wait for using the optional parameters in the following example:

session.advanced.waitForIndexesAfterSaveChanges({
    withTimeout: 5000,
    throwOnTimeout: false,
    waitForIndexes: "Products/ByName"
});

The default parameters are:

• timeout - null (will wait 15 seconds),
• throwOnTimeout - false,
• indexes - null (will wait for all indexes impacted by the changes made in the session)

Cutoff Point

If a query sent to the server specifies that it needs to wait for non-stale results then RavenDB sets the cutoff Etag for the staleness check. It is the Etag of the last document (or document tombstone), from the collection(s) processed by the index, as of the query arrived to the server. This way the server won't be waiting forever for the non-stale results even though documents are constantly updated meanwhile.

If the last Etag processed by the index is greater than the cutoff, then the results are considered as non-stale.