Indexes: Map-Reduce Indexes

When it comes to index creation, the only difference between simple indexes and the map-reduce ones is an additional reduce function defined in index definition. To deploy an index we need to create a definition and deploy it using one of the ways described in the creating and deploying article.

Example I - Count

Let's assume that we want to count the number of products for each category. To do it, we can create the following index using LoadDocument inside:

class Products_ByCategory extends AbstractIndexCreationTask {

    constructor() {
        super();

        this.map = "docs.Products.Select(product => new { " +
            "    Product = product, " +
            "    CategoryName = (this.LoadDocument(product.Category, \"Categories\")).Name " +
            "}).Select(this0 => new { " +
            "    Category = this0.CategoryName, " +
            "    Count = 1 " +
            "})";

        this.reduce = "results.GroupBy(result => result.category).Select(g => new { " +
            "    category = g.Key, " +
            "    count = Enumerable.Sum(g, x => ((int) x.count)) " +
            "})";
    }
}

and issue the query:

const results = await session
    .query({ indexName: "Products/ByCategory" })
    .whereEquals("Category", "Seafood")
    .all();

from 'Products/ByCategory'
where Category == 'Seafood'

The above query will return one result for Seafood with the appropriate number of products from that category.

Example II - Average

In this example, we will count an average product price for each category. The index definition:

class Products_Average_ByCategory extends AbstractIndexCreationTask {

    constructor() {
        super();

        this.map = "docs.Products.Select(product => new { " +
            "    Product = product, " +
            "    CategoryName = (this.LoadDocument(product.Category, \"Categories\")).Name " +
            "}).Select(this0 => new { " +
            "    Category = this0.CategoryName, " +
            "    PriceSum = this0.product.PricePerUnit, " +
            "    PriceAverage = 0, " +
            "    ProductCount = 1 " +
            "})";

        this.reduce = "results.GroupBy(result => result.Category).Select(g => new { " +
            "    g = g, " +
            "    ProductCount = Enumerable.Sum(g, x => ((int) x.ProductCount)) " +
            "}).Select(this0 => new { " +
            "    this0 = this0, " +
            "    PriceSum = Enumerable.Sum(this0.g, x0 => ((decimal) x0.PriceSum)) " +
            "}).Select(this1 => new { " +
            "    Category = this1.this0.g.Key, " +
            "    PriceSum = this1.PriceSum, " +
            "    PriceAverage = this1.PriceSum / ((decimal) this1.this0.ProductCount), " +
            "    ProductCount = this1.this0.ProductCount " +
            "})";
    }
}

and the query:

const results = await session
    .query({ indexName: "Products_Average_ByCategory" })
    .whereEquals("Category", "Seafood")
    .all();

from 'Products/Average/ByCategory'
where Category == 'Seafood'

Example III - Calculations

This example illustrates how we can put some calculations inside an index using on one of the indexes available in the sample database (Product/Sales).

We want to know how many times each product was ordered and how much we earned for it. In order to extract that information, we need to define the following index:

class Product_Sales extends AbstractIndexCreationTask {
    constructor() {
        super();

        this.map = "docs.Orders.SelectMany(order => order.Lines, (order, line) => new { " +
            "    Product = line.Product, " +
            "    Count = 1, " +
            "    Total = (((decimal) line.Quantity) * line.PricePerUnit) * (1M - line.Discount) " +
            "})";


        this.reduce = "results.GroupBy(result => result.Product).Select(g => new { " +
            "    Product = g.Key, " +
            "    Count = Enumerable.Sum(g, x => ((int) x.Count)), " +
            "    Total = Enumerable.Sum(g, x0 => ((decimal) x0.Total)) " +
            "})";
    }
}

and send the query:

const results = await session
    .query({ indexName: "Product/Sales" })
    .all();

from 'Product/Sales'

In addition to storing the aggregation results in the index, the map-reduce indexes can also output reduce results as documents to a specified collection. In order to create such documents, called artificial, you need to define the target collection using the OutputReduceToCollection property in the index definition.

class Product_Sales_ByMonth extends AbstractIndexCreationTask {
    constructor() {
        super();

        this.map = "docs.Orders.SelectMany(order => order.Lines, (order, line) => new { " +
            "    Product = line.Product, " +
            "    Month = new DateTime(order.OrderedAt.Year, order.OrderedAt.Month, 1), " +
            "    Count = 1, " +
            "    Total = (((decimal) line.Quantity) * line.PricePerUnit) * (1M - line.Discount) " +
            "})";

        this.reduce = "results.GroupBy(result => new { " +
            "    Product = result.Product, " +
            "    Month = result.Month " +
            "}).Select(g => new { " +
            "    Product = g.Key.Product, " +
            "    Month = g.Key.Month, " +
            "    Count = Enumerable.Sum(g, x => ((int) x.Count)), " +
            "    Total = Enumerable.Sum(g, x0 => ((decimal) x0.Total)) " +
            "})";

        this.outputReduceToCollection = "MonthlyProductSales";
    }
}

Writing map-reduce outputs into documents allows you to define additional indexes on top of them that give you the option to create recursive map-reduce operations. This way, you can do daily/monthly/yearly summaries very cheaply and easy.

In addition, you can also apply the usual operations on documents (e.g. data subscriptions or ETL).

Artificial Document IDs

The identifiers of artificial documents are generated as:

• <OutputCollectionName>/<hash-of-reduce-key>

For the above sample index, the document ID can be:

• MonthlyProductSales/13770576973199715021

The numeric part is the hash of the reduce key values, in this case: hash(Product, Month).

If the aggregation value for a given reduce key changes then we overwrite the artificial document. It will get removed once there is no result for a given reduce key.

Artificial Document Flags

Documents generated by map-reduce indexes get the following @flags metadata:

"@flags": "Artificial, FromIndex"

Those flags are used internally by the database to filter out artificial documents during replication.