RavenDB uses Indexes to satisfy queries.
Indexes allow for fast query results as the entire dataset is not re-scanned each time like traditional queries.
Also, indexes can perform various calculations, conversions, filtration, and other processes behind the scenes so that queries do not have to.
Indexes keep the processed data in separate storage so that the raw data isn't affected.
Indexes are at the heart of RavenDB's efficiency and should be understood before indexes and queries are defined in production.
Although they remove a lot of the burden from the queries, indexes utilize machine resources behind the scenes. This should be considered when defining indexes and queries.
Indexes can be created:
- From Studio
- From code - see Creating and Deploying Indexes
- Auto indexes are generated by the server - see Auto-Map Index Generation
In this page:
- Indexes - The General Concept
- Indexes - The Moving Parts :
- Index Types :
- Field Configuration Options
- Modifying Index Definition
- Indexes in the Cluster
- Indexing Errors
Indexes - The General Concept
To return query results about your documents without scanning the entire dataset each and every time, RavenDB uses Indexes.
Once defined, the index iterates over the documents, and for every field (document property) that is requested to be indexed, a map is built between the terms derived from these fields and the actual documents that contain them.
- You can also define static indexes that process various calculations, filters, and more so that queries don't have to.
A query operating on these fields ends up with a simple search from the queried terms to the list of documents that contain them.
After the first indexing run, the index will keep that map current without re-processing the entire index data - only update the relevant details when a document update happens in the database.
Indexes are not stored in the 'document store' but have their own separate storage so as not to affect the original data.
Indexes in RavenDB are split across multiple axes (see more below)
- Auto Indexes -vs- Static Indexes
- Map Indexes -vs- Map-Reduce Indexes
- Single-Collection Indexes -vs- Multi-Collection Indexes
Indexes - The Moving Parts
1. Index Definition
The index definition tells RavenDB how to index the data.
It specifies the fields to be indexed and how those fields should be indexed (i.e. allowing a full-text-search option).
These fields can be specified explicitly or defined dynamically supporting any document structure.
The index definition is created by the client (Static-Index), or by the server Query Optimizer(Auto-Index).
Note: Data from related documents can also be indexed using 'LoadDocument'.
Learn more in Indexing Related Documents.
2. Indexing Process
Indexing is the process of indexing the data, iterating over the documents, and creating a map
between the terms indexed and the actual documents that contain them.
Indexing is a background operation, it is scheduled to occur in an async manner upon any document change.
e.g. A document write operation doesn't wait for the index to complete processing -
The write operation is completed as soon as the transaction is written to disk.
An index is considered Stale if it had not yet processed all of the data.
A query can request that results are returned only when the index is up-to-date by using the method
WaitForNonStaleResults(). This way, a write operation can wait for the indexing process to finish before acknowledging the write.
See: Understanding Eventual Consistency
The async indexing process works with hard resets, shutdowns, and the like.
If the database was restarted after a document was modified but before it was indexed,
the indexing process will just pick up from where it left off and complete the work.
Each index is assigned a dedicated thread, thus no indexing process can interfere with any other.
By default, indexing threads start with a lower priority than request-processing threads.
The indexing-thread priority can be set higher and RavenDB will update this at the operating system level.
Indexing can be throttled to delay indexing tasks by a pre-set time period.
Throttling is helpful when sufficient server resources need to remain available for users while heavy-duty indexing tasks are due.
See: Index Throttling
3. Indexed Data
The resulting output of 'step 2' (the indexing process) is also referred to as an Index.
Queries operate on indexed data to get documents result.
Note: The full document is not stored in the index - only the document ID.
Upon a query match, we load the document itself from the document storage.
Index-Entries are all of the document fields that are requested to be indexed, as defined in the index definition.
The index-entries values are broken into Terms according to the specified analyzer used in the index definition.
Term is the actual indexed value that is stored in the index.
In addition to the Terms, some document fields can be stored directly in the index data.
This allows for query results to be fetched from the index itself instead of loading the original document.
Indexes in RavenDB are split across the following multiple axes:
Auto Indexes -vs- Static Indexes
When a Query doesn't specify a specific index to be used, the server Query Optimizer will first analyze the query and search for an already existing Auto-Index that can answer the query. If there is no such index, the Query Optimizer creates on the fly an Auto-Index that can answer this query and all previous queries on that collection.
When the new Auto-Index has caught up, RavenDB cleans up all the old Auto-Indexes that are now superseded by the new one.
Created by the user (database administrator only) from the Studio or from the Client API.
The index shape (as defined in the index definition) and the shape of the source document don't have to be the same, as the indexed data can be a computed value. These computations are run as background indexing processes to provide fast query results when querying the index.
- To query a static index, the index must be specified in the query definition.
Map Indexes -vs- Map-Reduce Indexes
Map indexes are simple indexes.
and how it should be indexed, as these functions allow you to compute the indexed value.
Map-Reduce indexes allow performing complex data aggregation.
The Map stage is similar to a regular Map-Index, defining what data should be indexed.
The Reduce stage operates on the Map results, specifying how the data should be grouped and aggregated.
Single-Collection Indexes -vs- Multi-Collection Indexes
Index definition contains only one Map function defined on a specific collection.
Data from several collections can be indexed (each in a different Map) and the results are united in a single index.
The only requirement is that all the Map definitions have the same output shape.
Field Configuration Options
Additional settings can be specified per field in the index-entry definition, configuring how the terms are indexed inside RavenDB. See Create Map Index to learn how to set these options in the Studio.
The original field data is split and tokenized according to the selected analyzer. Learn more about analyzers here.
- Suggestions - Allow finding similar results to the string in your query. i.e. Martin -> Martine.
- Term Vector - Allow finding similar documents based on shared indexed terms.
Allow geographical querying on longitude and latitude values or WKT values provided from the document.
Customize the spatial indexing strategy.
Learn more in Indexing Spatial Data
Field can be stored within the indexed data.
This allows retrieving the value from the indexed data at query time, instead of loading the original document.
Learn more in Storing Data in Index
Modifying Index Definition
Only an index that is not set as 'Locked' can actually be modified.
When the index definition has changed in a way that invalidates the previous indexing results,
the modification is handled in a side-by-side manner.
e.g. A mapping function change will invalidate previous results, while a change in priority will not.
The original index is retained and is fully operable while the new index (with the new definition) is being built.
Once the new index is up-to-date the original index is removed in favor of the new one.
See the example in Index List View - Side by Side.
- RavenDB keeps a history of index revisions,
allowing you to revert an index to any of its past revisions.
The number of index revisions kept can be configured in the Server Configuration Options, or using the Database Settings view in the Studio.
Indexes in the Cluster
Index & Auto-Index creation is a cluster operation. It goes through the Raft protocol.
Index creation will fail if the majority of the nodes in the cluster are not reachable.
Once an index is created against any node in the Database Group, RavenDB will make sure that its definition is replicated to all the database's nodes. The indexing process will occur separately on each node.
Note: The External Replication ongoing task does NOT replicate indexes.
In a multiple-nodes cluster, indexing can be configured to occur either in Rolling deployment Mode (one node at a time if machine resources are limited) or in Parallel mode (simultaneously on all nodes).
An error in indexing a document means that this particular document is not indexed and you will not see it in the query result.
An index is only allowed a certain failure rate, above which it is marked in an error state.
An index in an error state cannot be queried and will return an immediate error.
See more in Index List View - Errors.