Indexes: Map-Reduce Indexes

When it comes to index creation, the only difference between simple indexes and the map-reduce ones is an additional reduce function defined in index definition. To deploy an index we need to create a definition and deploy it using one of the ways described in the creating and deploying article.

Example I - Count

Let's assume that we want to count the number of products for each category. To do it, we can create the following index using LoadDocument inside:

class Products_ByCategory extends AbstractCsharpIndexCreationTask {

    constructor() {
        super();

        this.map = "docs.Products.Select(product => new { " +
            "    Product = product, " +
            "    CategoryName = (this.LoadDocument(product.Category, \"Categories\")).Name " +
            "}).Select(this0 => new { " +
            "    Category = this0.CategoryName, " +
            "    Count = 1 " +
            "})";

        this.reduce = "results.GroupBy(result => result.category).Select(g => new { " +
            "    category = g.Key, " +
            "    count = Enumerable.Sum(g, x => ((int) x.count)) " +
            "})";
    }
}

and issue the query:

const results = await session
    .query({ indexName: "Products/ByCategory" })
    .whereEquals("Category", "Seafood")
    .all();

from 'Products/ByCategory'
where Category == 'Seafood'

The above query will return one result for Seafood with the appropriate number of products from that category.

Example II - Average

In this example, we will count an average product price for each category. The index definition:

class Products_Average_ByCategory extends AbstractCsharpIndexCreationTask {

    constructor() {
        super();

        this.map = "docs.Products.Select(product => new { " +
            "    Product = product, " +
            "    CategoryName = (this.LoadDocument(product.Category, \"Categories\")).Name " +
            "}).Select(this0 => new { " +
            "    Category = this0.CategoryName, " +
            "    PriceSum = this0.product.PricePerUnit, " +
            "    PriceAverage = 0, " +
            "    ProductCount = 1 " +
            "})";

        this.reduce = "results.GroupBy(result => result.Category).Select(g => new { " +
            "    g = g, " +
            "    ProductCount = Enumerable.Sum(g, x => ((int) x.ProductCount)) " +
            "}).Select(this0 => new { " +
            "    this0 = this0, " +
            "    PriceSum = Enumerable.Sum(this0.g, x0 => ((decimal) x0.PriceSum)) " +
            "}).Select(this1 => new { " +
            "    Category = this1.this0.g.Key, " +
            "    PriceSum = this1.PriceSum, " +
            "    PriceAverage = this1.PriceSum / ((decimal) this1.this0.ProductCount), " +
            "    ProductCount = this1.this0.ProductCount " +
            "})";
    }
}

and the query:

const results = await session
    .query({ indexName: "Products_Average_ByCategory" })
    .whereEquals("Category", "Seafood")
    .all();

from 'Products/Average/ByCategory'
where Category == 'Seafood'

Example III - Calculations

This example illustrates how we can put some calculations inside an index using on one of the indexes available in the sample database (Product/Sales).

We want to know how many times each product was ordered and how much we earned for it. In order to extract that information, we need to define the following index:

class Product_Sales extends AbstractCsharpIndexCreationTask {
    constructor() {
        super();

        this.map = "docs.Orders.SelectMany(order => order.Lines, (order, line) => new { " +
            "    Product = line.Product, " +
            "    Count = 1, " +
            "    Total = (((decimal) line.Quantity) * line.PricePerUnit) * (1M - line.Discount) " +
            "})";


        this.reduce = "results.GroupBy(result => result.Product).Select(g => new { " +
            "    Product = g.Key, " +
            "    Count = Enumerable.Sum(g, x => ((int) x.Count)), " +
            "    Total = Enumerable.Sum(g, x0 => ((decimal) x0.Total)) " +
            "})";
    }
}

and send the query:

const results = await session
    .query({ indexName: "Product/Sales" })
    .all();

from 'Product/Sales'

In addition to storing the aggregation results in the index, the map-reduce index can also output those reduce results as documents to a specified collection. In order to create these documents, called "artificial", you need to define the target collection using the outputReduceToCollection property in the index definition.

Writing map-reduce outputs into documents allows you to define additional indexes on top of them that give you the option to create recursive map-reduce operations. This makes it cheap and easy to, for example, recursively create daily, monthly, and yearly summaries on the same data.

In addition, you can also apply the usual operations on artificial documents (e.g. data subscriptions or ETL).

If the aggregation value for a given reduce key changes, we overwrite the output document. If the given reduce key no longer has a result, the output document will be removed.

Reference Documents

To help organize these output documents, the map-reduce index can also create an additional collection of artificial reference documents. These documents aggregate the output documents and store their document IDs in an array field ReduceOutputs.

The document IDs of reference documents are customized to follow some pattern. The format you give to their document ID also determines how the output documents are grouped.

Because reference documents have well known, predictable IDs, they are easier to plug into indexes and other operations, and can serve as an intermediary for the output documents whose IDs are less predictable. This allows you to chain map-reduce indexes in a recursive fashion, see Example II.

Learn more about how to configure output and reference documents in the Studio: Create Map-Reduce Index.

Artificial Document IDs

The identifiers of artificial documents are generated as:

• <OutputCollectionName>/<hash-of-reduce-key>

For the above sample index, the document ID can be:

• MonthlyProductSales/13770576973199715021

The numeric part is the hash of the reduce key values, in this case: hash(Product, Month).

If the aggregation value for a given reduce key changes then we overwrite the artificial document. It will get removed once there is no result for a given reduce key.

Artificial Document Flags

Documents generated by map-reduce indexes get the following @flags metadata:

"@flags": "Artificial, FromIndex"

Those flags are used internally by the database to filter out artificial documents during replication.

Usage

The map-reduce output documents are configured with these properties of IndexDefinition:

const outputReduceToCollection = indexDefinition.outputReduceToCollection;

const patternReferencesCollectionName = indexDefinition.patternReferencesCollectionName;

const patternForOutputReduceToCollectionReferences = indexDefinition.patternForOutputReduceToCollectionReferences;

Examples

class Product_Sales_ByMonth extends AbstractCsharpIndexCreationTask {
    constructor() {
        super();

        this.map = "docs.Orders.SelectMany(order => order.Lines, (order, line) => new { " +
            "    Product = line.Product, " +
            "    Month = new DateTime(order.OrderedAt.Year, order.OrderedAt.Month, 1), " +
            "    Count = 1, " +
            "    Total = (((decimal) line.Quantity) * line.PricePerUnit) * (1M - line.Discount) " +
            "})";

        this.reduce = "results.GroupBy(result => new { " +
            "    Product = result.Product, " +
            "    Month = result.Month " +
            "}).Select(g => new { " +
            "    Product = g.Key.Product, " +
            "    Month = g.Key.Month, " +
            "    Count = Enumerable.Sum(g, x => ((int) x.Count)), " +
            "    Total = Enumerable.Sum(g, x0 => ((decimal) x0.Total)) " +
            "})";

        this.outputReduceToCollection = "MonthlyProductSales";
    }
}

Important Comments

Saving documents

Artificial documents are stored immediately after the indexing transaction completes.

Recursive indexing loop

It is forbidden to output reduce results to collections such as the following:

• A collection that the current index is already working on.
  E.g., an index on a DailyInvoices collection outputs to DailyInvoices.
• A collection that the current index is loading a document from.
  E.g., an index with LoadDocument(id, "Invoices") outputs to Invoices.
• Two collections, each processed by a map-reduce indexes,
  when each index outputs to the second collection.
  E.g.,
  An index on the Invoices collection outputs to the DailyInvoices collection,
  while an index on DailyInvoices outputs to Invoices.

When an attempt to create such an infinite indexing loop is detected a detailed error is generated.

Output to an Existing collection

Creating a map-reduce index which defines an output collection that already exists and contains documents, will result in an error.
Delete all documents from the target collection before creating the index, or output results to a different collection.

Modification of Artificial Documents

Artificial documents can be loaded and queried just like regular documents.
However, it is not recommended to edit artificial documents manually since any index results update would overwrite all manual modifications made in them.