Indexes Overview

The index definition tells RavenDB how to index the data.

It specifies which fields to index and defines how they should be indexed,
e.g., configuring a field for full-text search, selecting the analyzer to use, etc.
These fields can be specified explicitly or defined dynamically supporting any document structure.

The index definition is created by the client (Static-Index), or by the server (Auto-Index).

Note: Data from related documents can also be indexed using 'LoadDocument'.
Learn more in Indexing Related Documents.

Indexing is the process of iterating over the raw documents, indexing their data as defined by the index definition, and building a map between the indexed terms and the raw documents that contain them.

Indexing is a background operation, it is scheduled to occur in an async manner upon any document change. Once defined and deployed, an index will initially process the entire dataset.
After that, the index will only process documents that were modified, added or deleted.

A document write operation doesn't wait for the index to complete processing -
the write operation is completed as soon as the transaction is written to disk.
However, a write operation can wait for the indexing process to finish before acknowledging the write by using method WaitForIndexesAfterSaveChanges.

An index is considered Stale if it has not yet processed all the data.
A query can request that results are returned only when the index is up-to-date by using method WaitForNonStaleResults. Learn more in: Understanding Eventual Consistency.

The async indexing process works with hard resets, shutdowns, and the like.
If the database was restarted after a document was modified but before it was indexed,
the indexing process will just pick up from where it left off and complete the work.

Each index is assigned a dedicated thread, thus no indexing process can interfere with any other.
By default, indexing threads start with a lower priority than request-processing threads. The indexing-thread priority can be set higher and RavenDB will update this at the operating system level.

Indexing can be throttled to delay indexing tasks by a pre-set time period.
Throttling is helpful when sufficient server resources need to remain available for users while heavy-duty indexing tasks are due. See: Index Throttling

The resulting output of the indexing process from 'step 2' above is also referred to as an 'Index'.

Index-Entries
During the indexing process, an index-entry is created for each raw document that is processed.
Usually a single index-entry is created per raw document, unless working with a fanout index.

Index-Fields and Terms
Each index-entry contains the index-fields that were defined in the index definition.
Each index-field contains terms that are generated from the data in the raw documents.

The terms generated depend on the analyzer used, and they are the actual indexed values that are stored in the index.
When querying the index, you can retrieve the original documents and filter the results based on these terms.

Stored Data
In addition to the terms, some document fields can be stored directly in the index data.
This allows for query results to be fetched from the index itself instead of loading the original document.

Note: The full document is not stored in the index - only the document ID.
Upon a query match, we load the document itself from the document storage.

Auto Indexes:

• Auto-indexes are created by the server.
• When a query that has some filtering condition doesn't specify a specific index to be used,
  the server Query Optimizer will first analyze the query and search for an already existing Auto-index that can answer the query.
• If there is no such index, the Query Optimizer creates on the fly an Auto-index that can answer this query and all previous queries on that collection.
• When the new Auto-index has caught up, RavenDB cleans up all the old Auto-indexes that are now superseded by the new one.

Static Indexes:

• Static-indexes are created by the user (database admin only) from the Studio or from the Client API.
• A Static-index can be used to make any computation on the document fields.
  These computations run as a background process during indexing time, and not at query time.
  This way the indexed data is ready for queries, providing fast query results when querying the index.
• The index shape (as defined in the index definition) and the shape of the source document don't have to be the same, as the indexed data can be a computed value.

Map Indexes:
Map indexes are simple indexes.
Contain one or more LINQ-based or JavaScript mapping functions indicating what should be indexed from the document, and how it should be indexed, as these functions allow you to compute the indexed value.

Map-Reduce Indexes:
Map-Reduce indexes allow performing complex data aggregation.
The Map stage is similar to a regular Map-Index, defining what data should be indexed.
The Reduce stage operates on the Map results, specifying how the data should be grouped and aggregated.

Single-Collection Indexes:
Index definition contains only one Map function defined on a specific collection.

Multi-Collection Indexes:
Data from several collections can be indexed (each in a different Map) and the results are united in a single index.
The only requirement is that all the Map definitions have the same output shape.