The Embedding Collections

• RavenDB creates a separate embeddings collection for each source collection from which embeddings are generated. The naming format for these collections is: @embeddings/<source-collection-name>.

• Each document in the embeddings collection references ALL embeddings generated from the content of the corresponding document in the source collection by any defined embeddings generation task.

• The document structure in the embeddings collection is:

{
  "identifier-of-task-1": {
     "@quantization": "<quantization-type>",
     "Property1": [
       "Hash of the embedding vector generated for 1st text chunk of Property1's content",
       "Hash of the embedding vector generated for 2nd text chunk of Property1's content",   
       "Hash of the embedding vector generated for 3rd text chunk of Property1's content", 
       "..."
     ],
     "Property2": [
       "Hash of the embedding vector generated for 1st text chunk of Property2's content",
       "..."
     ]
  },
  "identifier-of-task-2": {
    "Property3": [
      "Hash of the embedding vector generated for 1st text chunk of Property3's content",
      "..."
    ]
  },
  "Other-tasks...": {
     ...
  },
  "@metadata": {
     "@collection": "<source-collection-name>/embeddings",
     "@flags": "HasAttachments"
  }
}

• For example:
  In this task definition, an embeddings generation task is defined on the Categories collection.
  This creates the @embeddings/Categories collection, where a document will look as follows:

  
  1. Collection name
     The unique name of the embeddings collection: @embeddings/Categories.
  2. Document ID
     Each document ID in this collection follows the format: embeddings/<source-document-name>
  3. Task identifier
     The identifier of the task that generated the embeddings for the listed properties.
  4. Quantization type
     The quantization method applied by the task when generating the embeddings.
  5. Source properties & their hashes:
     This section contains properties from the source document whose content was converted into embeddings.
     Each property holds an array of Base64 hashes. Each hash is derived from the content of an embedding vector generated for a text chunk from the property's content:
     <property-name>: [
      <hash-of-the-embedding-vector-for-text-chunk-1>,
      <hash-of-the-embedding-vector-for-text-chunk-2>,
      ...
]
  6. Attachment flag
     Indicates that the document includes attachments, which store the embeddings.
     The next image shows the embedding attachments in the document's properties pane.
  

  • Each attachment contains a single embedding.

  • The attachment name is the Base64 hash derived from the content of the embedding vector stored in the attachment:
    <hash-of-the-embedding-vector-content>

• The @embeddings-cache collection:

  

  1. Collection name
     The name of the embeddings cache collection: @embeddings-cache.

  2. Connection string identifier
     The document ID includes the connection string identifier, which specifies the provider that generated the embedding.

  3. Hash
     The document ID includes a Base64 hash created from a text chunk -
     either from a source document property or from a search term in a vector search query.

• A document in the @embeddings-cache collection:

  

  1. Document ID
     The document ID follows this format:
     embeddings-cache/<connection-string-identifier>/<hash-of-text-chunk-from-property-or-search-term>

     If the embedding was quantized by the task using a type other than Single (e.g., Int8 or Binary),
     the ID format includes the quantization type:
     embeddings-cache/<connection-string-identifier>/<hash-of-text-chunk-from-property-or-search-term>/<quantization-type>

  2. Expiration time
     The document is removed when the expiration time is reached.

• The embedding attachment:

  
  • The name of the attachment is the hash string:
    <hash-of-text-chunk-from-property-or-search-term>