Indexing Nested data

What is a Fanout index:

• A fanout index is an index that outputs multiple index-entries per document.
  A separate index-entry is created for each nested sub-object from the document.

• The fanout index is useful when you need to retrieve documents matching query criteria
  that search for specific sub-objects that comply with some logical conditions.

Fanout index - Map index example:

// A fanout map-index:
// ===================
class Shops_ByTShirt_Fanout extends AbstractJavaScriptIndexCreationTask {
    constructor () {
        super();

        // Creating MULTIPLE index-entries per document,
        // an index-entry for each sub-object in the tShirts list
        this.map("OnlineShops", shop => {
            return shop.tShirts.map(shirt => {
                return {
                    color: shirt.color,
                    size: shirt.size,
                    logo: shirt.logo
                };
            });
        });
    }
}

// Query the fanout index:
// =======================
const shopsThatHaveMediumRedShirts = await session
    .query({ indexName: "Shops/ByTShirt/Fanout" })
     // Query for documents that have a "Medium Red TShirt"
    .whereEquals("color", "red")
    .andAlso()
    .whereEquals("size", "M")
    .all();

from index "Shops/ByTShirt/Fanout" 
where color == "red" and size == "M"

// Query results:
// ==============

// Only the 'Shop1' document will be returned,
// since it is the only document that has the requested combination within the tShirt list.

The index-entries:

Multiple index-entries per document

1. The index-entries content is visible from the Studio Query view.

2. Check option: Show raw index-entries instead of Matching documents.

3. Each row represents an index-entry.
   Each index-entry corresponds to an inner item in the TShirt list.

4. In this example, the total number of index-entries is 12,
   which is the total number of inner items in the TShirt list in all 3 documents in the collection.

Fanout index - Map-Reduce index example:

• The fanout index concept applies to map-reduce indexes as well:

// A fanout map-reduce index:
// ==========================
class Sales_ByTShirtColor_Fanout extends AbstractJavaScriptIndexCreationTask {
    constructor () {
        super();

        this.map("OnlineShops", shop => {
            return shop.tShirts.map(shirt => {
                return {
                    // Define the index-fields:
                    color: shirt.color,
                    itemsSold: shirt.sold,
                    totalSales: shirt.price * shirt.sold
                };
            });
        });

        this.reduce(results => results
            .groupBy(shirt => shirt.color)
            .aggregate(g => {
                return {
                    // Calculate sales per color
                    color: g.key,
                    itemsSold: g.values.reduce((p, c) => p + c.itemsSold, 0),
                    totalSales: g.values.reduce((p, c) => p + c.totalSales, 0),
                }
            }));
    }
}

// Query the fanout index:
// =======================
const queryResult = await session
    .query({ indexName: "Sales/ByTShirtColor/Fanout" })
     // Query for index-entries that contain "black"
    .whereEquals("color", "black")
    .firstOrNull();

// Get total sales for black TShirts
const blackShirtsSales = queryResult?.totalSales ?? 0;

from index "Sales/ByTShirtColor/Fanout"
where color == "black"

// Query results:
// ==============

// With the sample data used in this article,
// The total sales revenue from black TShirts sold (in all shops) is 490

Fanout index - Performance hints:

• Fanout indexes are typically more resource-intensive than other indexes as RavenDB has to index a large number of index-entries. This increased workload can lead to higher CPU and memory utilization, potentially causing a decline in the overall performance of the index.

• When the number of index-entries generated from a single document exceeds a configurable limit,
  RavenDB will issue a High indexing fanout ratio alert in the Studio notification center.

• You can control when this performance hint is created by setting the PerformanceHints.Indexing.MaxIndexOutputsPerDocument configuration key (default is 1024).

• So, for example, adding another OnlineShop document with a tShirt object containing 1025 items
  will trigger the following alert:

  

  High indexing fanout ratio notification

• Clicking the 'Details' button will show the following info:

  

  Fanout index, performance hint details

Fanout index - Paging:

• A fanout index has more index-entries than the number of documents in the collection indexed.
  Multiple index-entries "point" to the same document from which they originated,
  as can be seen in the above index-entries example.

• When making a fanout index query that should return full documents (without projecting results),
  then in this case, the totalResults property (available when calling the query statistics() method)
  will contain the total number of index-entries and Not the total number of resulting documents.

• To overcome this when paging results, you must take into account the number of "duplicate"
  index-entries that are skipped internally by the server when serving the resulting documents.

• Please refer to paging through tampered results for further explanation and examples.