Indexes: Analyzers

Analyzers that remove common "stop words":

Analyzers that do not remove common "stop words"

Analyzers that tokenize according to the defined number of characters

Using the Default Search Analyzer

• When the indexing behavior is set to FieldIndexing.Search and no analyzer is specified for the index-field, RavenDB will use the Default Search Analyzer.
  By default, this analyzer is the RavenStandardAnalyzer (inherits from Lucene's StandardAnalyzer).

• To customize a different analyzer that will serve as your Default Search Analyzer,
  set the Indexing.Analyzers.Search.Default configuration key.

• Using a search analyzer enables full-text search queries against the field.
  Given the same sample text from above, RavenStandardAnalyzer will produce the following tokens:
  [quick] [brown] [fox] [jumped] [over] [lazy] [dogs] [bob@hotmail.com] [123432]

public class BlogPosts_ByContent : AbstractIndexCreationTask<BlogPost>
{
    public BlogPosts_ByContent()
    {
        Map = posts => from post in posts
            select new
            {
                Title = post.Title,
                Content = post.Content
            };

        // Set the Indexing Behavior:
        // ==========================
        
        // Set 'FieldIndexing.Search' on index-field 'Content'
        Indexes.Add(x => x.Content, FieldIndexing.Search);
        
        // => Index-field 'Content' will be processed by the "Default Search Analyzer"
        //    since no other analyzer is set.
    }
}

Using the Default Exact Analyzer

• When the indexing behavior is set to FieldIndexing.Exact, RavenDB will use the Default Exact Analyzer. By default, this analyzer is the KeywordAnalyzer.

• KeywordAnalyzer treats the entire content of the index-field as a single token, preserving the original text's case and ensuring no transformations, such as case normalization or stemming, are applied.
  The field's value is indexed exactly as provided, enabling precise, case-sensitive matching at query time.

• To customize a different analyzer that will serve as your Default Exact Analyzer,
  set the Indexing.Analyzers.Exact.Default configuration key.

• Given the same sample text from above, KeywordAnalyzer will produce a single token:
  [The quick brown fox jumped over the lazy dogs, Bob@hotmail.com 123432.]

public class Employees_ByFirstAndLastName : AbstractIndexCreationTask<Employee>
{
    public Employees_ByFirstAndLastName()
    {
        Map = employees => from employee in employees
                           select new
                           {
                               LastName = employee.LastName,
                               FirstName = employee.FirstName
                           };

        // Set the Indexing Behavior:
        // ==========================
        
        // Set 'FieldIndexing.Exact' on index-field 'FirstName'
        Indexes.Add(x => x.FirstName, FieldIndexing.Exact);
        
        // => Index-field 'FirstName' will be processed by the "Default Exact Analyzer"
    }
}

Using the Default Analyzer

• When no indexing behavior is set and no analyzer is specified for the index-field,
  RavenDB will use the Default Analyzer. By default, this analyzer is the LowerCaseKeywordAnalyzer.

• LowerCaseKeywordAnalyzer behaves like Lucene's KeywordAnalyzer, but additionally performs case normalization, converting all characters to lowercase. The entire content of the field is processed into a single, lowercased token.

• To customize a different analyzer that will serve as your Default Analyzer,
  set the Indexing.Analyzers.Default configuration key.

• Given the same sample text from above, LowerCaseKeywordAnalyzer will produce a single token:
  [the quick brown fox jumped over the lazy dogs, bob@hotmail.com 123432.]

public class Employees_ByFirstName : AbstractIndexCreationTask<Employee>
{
    public Employees_ByFirstName()
    {
        Map = employees => from employee in employees
            select new
            {
                LastName = employee.LastName
            };

        // Index-field 'LastName' will be processed by the "Default Analyzer"
        // since:
        // * No analyzer was specified
        // * No Indexing Behavior was specified (neither Exact nor Search)
    }
}

Add custom analyzer via Studio

Custom analyzers can be added from the Custom Analyzers view in the Studio.
Learn more in this Custom analyzers article.

Add custom analyzer via Client API

First, create a class that inherits from the abstract Lucene.Net.Analysis.Analyzer class.
(you need to reference Lucene.Net.dll, which is included with the RavenDB Server package).
For example:

public class MyAnalyzer : Lucene.Net.Analysis.Analyzer
{
    public override TokenStream TokenStream(string fieldName, TextReader reader)
    {
        // Implement your analyzer's logic
        throw new CodeOmitted();
    }
}

Next, use PutAnalyzersOperation to deploy the analyzer to a specific database.
By default, PutAnalyzersOperation will apply to the default database of the document store you're using.
To target a different database, use the ForDatabase() method.

To make it a server-wide analyzer, use the PutServerWideOperation operation.`

public PutAnalyzersOperation(params AnalyzerDefinition[] analyzersToAdd)

public PutServerWideAnalyzersOperation(params AnalyzerDefinition[] analyzersToAdd)

public class AnalyzerDefinition
{
    // Name of the analyzer
    public string Name { get; set; }
    
    // C# source-code of the analyzer
    public string Code { get; set; }
}

Client API example:

// Define the put analyzer operation:
var putAnalyzerOp = new PutAnalyzersOperation(new AnalyzerDefinition
{
    // The name must be same as the analyzer's class name
    Name = "MyAnalyzer", 
                
    Code = @"
        using System.IO;
        using Lucene.Net.Analysis;
        using Lucene.Net.Analysis.Standard;

        namespace MyAnalyzer
        {
            public class MyAnalyzer : Lucene.Net.Analysis.Analyzer
            {
                public override TokenStream TokenStream(string fieldName, TextReader reader)
                {
                    throw new CodeOmitted();
                }
            }
        }"
});

// Deploy the analyzer:
store.Maintenance.ForDatabase("MyDatabase").Send(putAnalyzerOp);

Add custom analyzer directly to RavenDB's binaries

Another way to add custom analyzers to RavenDB is by placing them next to RavenDB's binaries.

The fully qualified name must be specified for any index-field that will be tokenized by the analyzer.

Note that the analyzer must be compatible with .NET Core 2.0 (e.g., a .NET Standard 2.0 assembly).

This is the only method for adding custom analyzers in RavenDB versions older than 5.2.