Indexing Time Series

Auto-Indexes:

• Time series index:
  Dynamic time series indexes are Not created in response to queries.

• Document index:
  Auto-indexes are created in response to dynamic queries.

Data source:

• Time series index:

  • Time series indexes process segments that contain time series entries.
    The entries are indexed through the segment they are stored in.
  • The following items can be indexed per index-entry in a time series index:
    • Values & timestamp of a time series entry
    • The entry tag
    • Content from a document referenced by the tag
    • Properties of the containing segment (see segment properties)

• Document index:

  • The index processes fields from your JSON documents.
    Documents are indexed through the collection they belong to.

Query results:

• Time series index:
  When querying a time series index, each result item corresponds to the type defined by the index-entry in the index definition, (unless results are projected). The documents themselves are not returned.

• Document index:
  The resulting objects are the document entities (unless results are projected).

There are two main ways to create a time series index:

1. Create a class that inherits from the abstract index class AbstractRawJavaScriptTimeSeriesIndexCreationTask.

2. Create a TimeSeriesIndexDefinition and deploy the time series index definition via PutIndexesOperation.

Map index - index single time series from single collection:

• In this index, we index data from the "StockPrices" time series entries in the "Companies" collection (tradeVolume, date).

• In addition, we index the containing document id (documentID), which is obtained from the segment,
  and some content from the document referenced by the entry's Tag (employeeName).

  • Map_index
  • IndexDefinition

  class StockPriceTimeSeriesFromCompanyCollection extends
      AbstractRawJavaScriptTimeSeriesIndexCreationTask {
      
      constructor() {
          super();
  
          this.maps.add(`
              // Call timeSeries.map(), pass:
              // * The collection to index
              // * The time series name
              // * The fuction that defines the index-entries
              // ============================================
              timeSeries.map("Companies", "StockPrices", function (segment) {
                   
                   // Return the index-entries:
                   // =========================
                   return segment.Entries.map(entry => {
                        let employee = load(entry.Tag, "Employees");
                   
                       // Define the index-fields per entry:
                       // ==================================
                       
                       return {
                           // Retrieve content from the time series ENTRY:
                           tradeVolume: entry.Values[4],
                           date: new Date(entry.Timestamp),
                           
                           // Retrieve content from the SEGMENT:
                           companyID: segment.DocumentId,
                           
                           // Retrieve content from the loaded DOCUMENT:
                           employeeName: employee.FirstName + " " + employee.LastName
                       };
                   });
              })
          `);
      }
  }

  const timeSeriesIndexDefinition = new TimeSeriesIndexDefinition();
  
  timeSeriesIndexDefinition.name = "StockPriceTimeSeriesFromCompanyCollection";
  
  timeSeriesIndexDefinition.maps = new Set([`
      from segment in timeSeries.Companies.StockPrices
      from entry in segment.Entries
  
      let employee = LoadDocument(entry.Tag, "Employees")
  
      select new
      {
          tradeVolume = entry.Values[4],
          date = entry.Timestamp.Date,
          companyID = segment.DocumentId,
          employeeName = employee.FirstName + " " + employee.LastName
      }`
  ]);
  
  // Deploy the index to the server via 'PutIndexesOperation'
  await documentStore.maintenance.send(new PutIndexesOperation(timeSeriesIndexDefinition));

• Querying this index, you can retrieve the indexed time series data while filtering by any of the index-fields.

  • Query_example_1
  • RQL_1
  • Query_example_2
  • RQL_2

  const results = await session
       // Retrieve time series data for the specified company:
       // ====================================================
      .query({ indexName: "StockPriceTimeSeriesFromCompanyCollection" })
      .whereEquals("companyID", "Companies/91-A")
      .all();
  
  // Results will include data from all 'StockPrices' entries in document 'Companies/91-A'. 

  from index "StockPriceTimeSeriesFromCompanyCollection"
  where "companyID" == "Comapnies/91-A"

  const results = await session
       // Find what companies had a very high trade volume:
       // ==================================================
      .query({ indexName: "StockPriceTimeSeriesFromCompanyCollection" })
      .whereGreaterThan("tradeVolume", 150_000_000)
      .selectFields(["companyID"])
      .distinct()
      .all();
  
  // Results will contain company "Companies/65-A"
  // since it is the only company with time series entries having such high trade volume.

  from index "StockPriceTimeSeriesFromCompanyCollection"
  where "tradeVolume" > 150_000_000
  select distinct companyID

Map index - index all time series from single collection:

class AllTimeSeriesFromCompanyCollection extends AbstractRawJavaScriptTimeSeriesIndexCreationTask {

    constructor() {
        super();
     
        this.maps.add(`
            // Call timeSeries.map(), pass:
            // * The collection to index and the function that defines the index-entries
            // * No time series is specified - so ALL time series from the collection will be indexed
            // ======================================================================================
            timeSeries.map("Companies", function (segment) {
                 
                 return segment.Entries.map(entry => ({
                     value: entry.Value,
                     date: new Date(entry.Timestamp)
                 }));
            })
        `);
    }
}

Map index - index all time series from all collections:

class AllTimeSeriesFromAllCollections extends AbstractRawJavaScriptTimeSeriesIndexCreationTask {

    constructor() {
        super();

        this.maps.add(`
            // No collection and time series are specified -
            // so ALL time series from ALL collections will be indexed
            // =======================================================
            timeSeries.map(function (segment) {
                 
                 return segment.Entries.map(entry => ({
                     value: entry.Value,
                     date: new Date(entry.Timestamp),
                     documentID: segment.DocumentId,
                 }));
            })
        `);
    }
}

Multi-Map index - index time series from several collections:

class Vehicles_ByLocation  extends AbstractRawJavaScriptTimeSeriesIndexCreationTask {
    
    constructor() {
        super();

        // Call 'timeSeries.map()' for each collection you wish to index
        // =============================================================
        
        this.maps.add(`
            timeSeries.map("Planes", "GPS_Coordinates", function (segment) {
                 
                 return segment.Entries.map(entry => ({
                      latitude: entry.Values[0],
                      longitude: entry.Values[1],
                      date: new Date(entry.Timestamp),
                      documentID: segment.DocumentId
                 }));
            })
        `);

        this.maps.add(`
            timeSeries.map("Ships", "GPS_Coordinates", function (segment) {
                 
                 return segment.Entries.map(entry => ({
                      latitude: entry.Values[0],
                      longitude: entry.Values[1],
                      date: new Date(entry.Timestamp),
                      documentID: segment.DocumentId
                 }));
            })
        `);
    }
}

Map-Reduce index:

class TradeVolume_PerDay_ByCountry extends AbstractRawJavaScriptTimeSeriesIndexCreationTask {

    constructor() {
        super();

        // Define the Map part:
        this.maps.add(`
            timeSeries.map("Companies", "StockPrices", function (segment) {
                 
                 return segment.Entries.map(entry => {
                      let company = load(segment.DocumentId, "Companies");
                     
                     return {
                         date: new Date(entry.Timestamp),
                         country: company.Address.Country,
                         totalTradeVolume: entry.Values[4],
                     };
                 });
            })
        `);

        // Define the Reduce part:
        this.reduce = `
            groupBy(x => ({date: x.date, country: x.country}))
                .aggregate(g => {
                    return {
                        date: g.key.date,
                        country: g.key.country,
                        totalTradeVolume: g.values.reduce((sum, x) => x.totalTradeVolume + sum, 0)
                    };
                })
        `;
    }
}

AbstractRawJavaScriptTimeSeriesIndexCreationTask

// To define a raw JavaScript index extend the following class:
// ============================================================
abstract class AbstractRawJavaScriptTimeSeriesIndexCreationTask
{    
    // The set of JavaScript map functions for this index
    maps; // Set<string>

    // The JavaScript reduce function
    reduce; // string
}

TimeSeriesIndexDefinition

class TimeSeriesIndexDefinition extends IndexDefinition

While TimeSeriesIndexDefinition is currently functionally equivalent to the regular IndexDefinition class from which it inherits, it is recommended to use TimeSeriesIndexDefinition when creating a time series index definition in case additional functionality is added in future versions of RavenDB.

Segment properties

• Segment properties include the entries data and aggregated values that RavenDB automatically updates in the segment's header.

• Unlike the C# client, class TimeSeriesSegment is Not defined in the Node.js client.
  However, the following are the segment properties that can be indexed from your raw javascript index definition which the server recognizes:

  // The ID of the document this time series belongs to
  DocumentId; // string
  
  // The name of the time series this segment belongs to
  Name; // string
  
  // The smallest values from all entries in the segment
  // The first array item is the Min of all first values, etc.
  Min; // number[]
  
  // The largest values from all entries in the segment
  // The first array item is the Max of all first values, etc.
  Max; // number[]
  
  // The sum of all values from all entries in the segment
  // The first array item is the Sum of all first values, etc.
  Sum; // number[]
  
  // The number of entries in the segment
  Count; // number
  
  // The timestamp of the first entry in the segment
  Start; // Date
  
  // The timestamp of the last entry in the segment
  End; // Date
  
  // The segment's entries themselves
  Entries; // TimeSeriesEntry[]

• These are the properties of a TimeSeriesEntry which can be indexed:

  class TimeSeriesEntry
  {
      timestamp; // Date
      tag;       // string
      values;    // number[]
  
      // This is equivalent to values[0]
      value;     // number
  }