The Embeddings Generation Task

• Define the general task settings:

  
  1. Name
     Enter a name for the task.

  2. Identifier
     Enter a unique identifier for the task.
     Each AI task in the database must have a distinct identifier.

     If not specified, or when clicking the "Regenerate" button,
     RavenDB automatically generates the identifier based on the task name. For example:

     • If the task name is: "Generate embeddings from OpenAI"
     • The generated identifier will be: "generate-embeddings-from-openai"

     Allowed characters: only lowercase letters (a-z), numbers (0-9), and hyphens (-).

     This identifier is used:

     • When querying embeddings generated by the task via a dynamic query.
       An example is available in Querying pre-made embeddings.
     • When indexing the embeddings generated by the task.
       An example is available in Indexing pre-made text-embeddings.
     • In documents in the Embeddings collection,
       where the task identifier is used to identify the origin of each embedding.

     See how this identifier is used in the Embeddings collection documents that reference the generated embeddings.

  3. Regenerate
     Click "Regenerate" to automatically create an identifier based on the task name.

  4. Task state
     Enable/Disable the task.
  5. Responsible node
     Select a node from the Database group to be the responsible node for this task.
  6. Connection string
     Select a previously defined AI connection string or create a new one.
  7. Enable document expiration
     This toggle appears only if the Document expiration feature is Not enabled in the database. Enabling document expiration ensures that embeddings in the @embeddings-cache collection are automatically deleted when they expire.
  8. Save
     Click Save to store the task definition or Cancel.

• Define the embeddings source - using PATHS:

  
  1. Collection
     Enter or select the source document collection from the dropdown.
  2. Embeddings source
     Select Paths to define the source content by specifying document properties.
  3. Source text path
     Enter the property name from the document that contains the text for embedding generation.
  4. Chunking method
     Select the method for splitting the source text into chunks.
     Learn more in Chunking methods and tokens.
  5. Max tokens per chunk
     Enter the maximum number of tokens allowed per chunk (this depends on the service provider).
  6. Add path configuration
     Click to add the specified to the list.

• Define the embeddings source - using SCRIPT:

  
  1. Embeddings source
     Select Script to define the source content and chunking methods using a JavaScript script.
  2. Script
     Refer to section Chunking methods and tokens for available JavaScript methods.
  3. Chunking method
     The selected chunking method will be used by default when no method is specified in the script.
     e.g., when the script contains: Name: this.Name.
  4. Max tokens per chunk:
     Enter the default value to use when no specific value is set for the chunking method in the script.

• Define quantization and expiration -
  for the generated embeddings from the source documents:

  
  1. Quantization
     Select the quantization method that RavenDB will apply to embeddings received from the service provider.
     Available options:
     • Single (no quantization)
     • Int8
     • Binary
  2. Embeddings cache expiration
     Set the expiration period for documents stored in the @embeddings-cache collection.
     These documents contain embeddings generated from the source documents, serving as a cache for these embeddings.
     The default initial period is 90 days. This period may be extended when the source documents change.
     Learn more in The embeddings cache collection.
  3. Regenerate embeddings
     This toggle is visible only when editing an existing task.
     Toggle ON to regenerate embeddings for all documents in the collection, as specified by the Paths or Script.

• Define chunking method & expiration -
  for the embedding generated from a search term in a vector search query:

  
  1. Querying
     This label indicates that this section configures parameters only for embeddings
     generated by the task for search terms in vector search queries.
  2. Chunking method
     Select the method for splitting the search term into chunks.
     Learn more in Chunking methods and tokens.
  3. Max tokens per chunk
     Enter the maximum number of tokens allowed per chunk (this depends on the service provider).
  4. Embeddings cache expiration
     Set the expiration period for documents stored in the @embeddings-cache collection.
     These documents contain embeddings generated from the search terms, serving as a cache for these embeddings.
     The default period is 14 days. Learn more in The embeddings cache collection.

Tokens and processing limits:

• A token is the fundamental unit that Large Language Models (LLMs) use to process text.
  AI service providers that generate embeddings from text enforce token limits for each processed text part.
  If a text exceeds the provider’s limit, it may be truncated or rejected.

Using chunking methods:

• To handle lengthy text, you can define chunking strategies in the task definition and specify the desired number of tokens per chunk. Chunking splits large input texts into smaller, manageable chunks, each containing no more than the specified maximum number of tokens.

• The maximum number of tokens per chunk depends on the AI service provider and the specific model defined in the connection string. While RavenDB does not tokenize text, it estimates the number of tokens for chunking purposes by dividing the text length by 4.

• The AI provider generates a single embedding for each chunk.
  Depending on the maximum tokens per chunk setting, a single input text may result in multiple embeddings.

Available chunking methods:
RavenDB offers several chunking methods that can be applied per source type.
These methods determine how input text is split before being sent to the provider.

Chunking method syntax for the JavaScript scripts:

// Available text-splitting methods:
// =================================

// Plain text methods:
text.split(text, maxTokensPerChunk);
text.splitLines(text, maxTokensPerChunk);
text.splitParagraphs(lines, maxTokensPerChunk);

// Markdown methods:
markdown.splitLines(text, maxTokensPerChunk);
markdown.splitParagraphs(lines, maxTokensPerChunk);

// HTML processing:
html.strip(htmlText, maxTokensPerChunk);

The embeddings generation task configuration:

// The 'EmbeddingsGenerationConfiguration' class inherits from 'EtlConfiguration'
// and provides the following specialized configurations for the embeddings generation task:
// =========================================================================================

public class EmbeddingsGenerationConfiguration : EtlConfiguration<AiConnectionString>
{
    public string Identifier { get; set; }
    public string Collection { get; set; }
    public List<EmbeddingPathConfiguration> EmbeddingsPathConfigurations { get; set; }
    public EmbeddingsTransformation EmbeddingsTransformation { get; set; }
    public VectorEmbeddingType Quantization { get; set; }
    public ChunkingOptions ChunkingOptionsForQuerying { get; set; }
    public TimeSpan EmbeddingsCacheExpiration { get; set; } = TimeSpan.FromDays(90);
    public TimeSpan EmbeddingsCacheForQueryingExpiration { get; set; } = TimeSpan.FromDays(14);
}

public class EmbeddingPathConfiguration
{
    public string Path { get; set; }
    public ChunkingOptions ChunkingOptions { get; set; }
}

public class ChunkingOptions
{
    public ChunkingMethod ChunkingMethod { get; set; } // Default is PlainTextSplit
    public int MaxTokensPerChunk { get; set; } = 512;
}

public enum ChunkingMethod
{
    PlainTextSplit,
    PlainTextSplitLines,
    PlainTextSplitParagraphs,
    MarkDownSplitLines,
    MarkDownSplitParagraphs,
    HtmlStrip
}

public class EmbeddingsTransformation
{
    public string Script { get; set; }
    public ChunkingOptions ChunkingOptions {get; set;}
}

public enum VectorEmbeddingType
{
    Single,
    Int8,
    Binary,
    Text
}

Deploying the embeddings generation task:

public AddEmbeddingsGenerationOperation(EmbeddingsGenerationConfiguration configuration);