Indexing Time Series

Auto-Indexes:

• Time series index:
  Dynamic time series indexes are Not created in response to queries.

• Document index:
  Auto-indexes are created in response to dynamic queries.

Data source:

• Time series index:

  • Time series indexes process segments that contain time series entries.
    The entries are indexed through the segment they are stored in, for example, using a LINQ syntax that resembles this one:

  from segment in timeseries
  from entry in segment
  ...

  • The following items can be indexed per index-entry in a time series index:
    • Values & timestamp of a time series entry
    • The entry tag
    • Content from a document referenced by the tag
    • Properties of the containing segment (see TimeSeriesSegment)

• Document index:

  • The index processes fields from your JSON documents.
    Documents are indexed through the collection they belong to, for example, using this LINQ syntax:

  from employee in employees
  ...

Query results:

• Time series index:
  When querying a time series index, each result item corresponds to the type defined by the index-entry in the index definition, (unless results are projected). The documents themselves are not returned.

• Document index:
  The resulting objects are the document entities (unless results are projected).

There are two main ways to create a time series index:

1. Create a class that inherits from one of the following abstract index creation task classes:

   • AbstractTimeSeriesIndexCreationTask for map and map-reduce time series indexes.
   • AbstractMultiMapTimeSeriesIndexCreationTask for multi-map time series indexes.
   • AbstractJavaScriptTimeSeriesIndexCreationTask for static javascript indexes.

2. Deploy a time series index definition via PutIndexesOperation:

   • Create a TimeSeriesIndexDefinition directly.
   • Create a strongly typed index definition using TimeSeriesIndexDefinitionBuilder.

Map index - index single time series from single collection:

• In this index, we index data from the "StockPrices" time series entries in the "Companies" collection (TradeVolume, Date).

• In addition, we index the containing document id (DocumentID), which is obtained from the segment,
  and some content from the document referenced by the entry's Tag (EmployeeName).

• Each tab below presents one of the different ways the index can be defined.

  • Map_index
  • NonTyped_index
  • JS_index
  • IndexDefinition
  • IndexDefinition_builder

  public class StockPriceTimeSeriesFromCompanyCollection : AbstractTimeSeriesIndexCreationTask<Company>
  {
      // The index-entry:
      // ================
      public class IndexEntry
      {
          // The index-fields:
          // =================
          public double TradeVolume { get; set; }
          public DateTime Date { get; set; }
          public string CompanyID { get; set; }
          public string EmployeeName { get; set; }
      }
      
      public StockPriceTimeSeriesFromCompanyCollection()
      { 
          // Call 'AddMap', specify the time series name to be indexed 
          AddMap("StockPrices", timeseries =>
                  from segment in timeseries
                  from entry in segment.Entries
                  
                  // Can load the document referenced in the TAG:
                  let employee = LoadDocument<Employee>(entry.Tag)
                  
                  // Define the content of the index-fields:
                  // =======================================
                  select new IndexEntry()
                  {
                      // Retrieve content from the time series ENTRY:
                      TradeVolume = entry.Values[4],
                      Date = entry.Timestamp.Date,
                      
                      // Retrieve content from the SEGMENT:
                      CompanyID = segment.DocumentId,
                      
                      // Retrieve content from the loaded DOCUMENT:
                      EmployeeName = employee.FirstName + " " + employee.LastName 
                  });
      }
  }

  public class StockPriceTimeSeriesFromCompanyCollection_NonTyped : AbstractTimeSeriesIndexCreationTask 
  {
      public override TimeSeriesIndexDefinition CreateIndexDefinition()
      {
          return new TimeSeriesIndexDefinition
          {
              Name = "StockPriceTimeSeriesFromCompanyCollection_NonTyped",
              Maps =
              {
                  @"
                  from segment in timeSeries.Companies.StockPrices 
                  from entry in segment.Entries
    
                  let employee = LoadDocument(entry.Tag, ""Employees"")
  
                  select new 
                  { 
                      TradeVolume = entry.Values[4], 
                      Date = entry.Timestamp.Date,
                      CompanyID = segment.DocumentId,
                      EmployeeName = employee.FirstName + ' ' + employee.LastName 
                  }"
              }
          };
      }
  }

  public class StockPriceTimeSeriesFromCompanyCollection_JS : 
      AbstractJavaScriptTimeSeriesIndexCreationTask
  {
      public StockPriceTimeSeriesFromCompanyCollection_JS()
      {
          Maps = new HashSet<string> { @"
              timeSeries.map('Companies', 'StockPrices', function (segment) {
  
                  return segment.Entries.map(entry => {
                      let employee = load(entry.Tag, 'Employees');
  
                      return {
                          TradeVolume: entry.Values[4],
                          Date: new Date(entry.Timestamp.getFullYear(),
                                         entry.Timestamp.getMonth(),
                                         entry.Timestamp.getDate()),
                          CompanyID: segment.DocumentId,
                          EmployeeName: employee.FirstName + ' ' + employee.LastName
                      };
                  });
              })"
          };
      }
  }

  // Define the 'index definition'
  var indexDefinition = new TimeSeriesIndexDefinition
      {
          Name = "StockPriceTimeSeriesFromCompanyCollection ",
          Maps =
          {
              @"
              from segment in timeSeries.Companies.StockPrices 
              from entry in segment.Entries 
  
              let employee = LoadDocument(entry.Tag, ""Employees"")
  
              select new 
              { 
                  TradeVolume = entry.Values[4], 
                  Date = entry.Timestamp.Date,
                  CompanyID = segment.DocumentId,
                  EmployeeName = employee.FirstName + ' ' + employee.LastName 
              }"
          }
      };
  
  // Deploy the index to the server via 'PutIndexesOperation'
  documentStore.Maintenance.Send(new PutIndexesOperation(indexDefinition));

  // Create the index builder
  var TSIndexDefBuilder =
      new TimeSeriesIndexDefinitionBuilder<Company>("StockPriceTimeSeriesFromCompanyCollection ");
  
  TSIndexDefBuilder.AddMap("StockPrices", timeseries => 
      from segment in timeseries
      from entry in segment.Entries
      
      // Note:
      // Class TimeSeriesIndexDefinitionBuilder does not support the 'LoadDocument' API method.
      // Use one of the other index creation methods if needed.
      
      select new
      {
          TradeVolume = entry.Values[4],
          Date = entry.Timestamp.Date,
          ComapnyID = segment.DocumentId
      });
  
  // Build the index definition
  var indexDefinitionFromBuilder = TSIndexDefBuilder.ToIndexDefinition(documentStore.Conventions);
  
  // Deploy the index to the server via 'PutIndexesOperation'
  documentStore.Maintenance.Send(new PutIndexesOperation(indexDefinitionFromBuilder));

• Querying this index, you can retrieve the indexed time series data while filtering by any of the index-fields.

  • Query_example_1
  • RQL_1
  • Query_example_2
  • RQL_2

  using (var session = documentStore.OpenSession())
  {
      // Retrieve time series data for the specified company:
      // ====================================================
      List<StockPriceTimeSeriesFromCompanyCollection.IndexEntry> results = session
         .Query<StockPriceTimeSeriesFromCompanyCollection.IndexEntry,
             StockPriceTimeSeriesFromCompanyCollection>()
         .Where(x => x.CompanyID == "Companies/91-A")
         .ToList();
  }
  
  // Results will include data from all 'StockPrices' entries in document 'Companies/91-A'. 

  from index "StockPriceTimeSeriesFromCompanyCollection"
  where "CompanyID" == "Comapnies/91-A"

  using (var session = documentStore.OpenSession())
  {
      // Find what companies had a very high trade volume:
      // ==================================================
      List<string> results = session
          .Query<StockPriceTimeSeriesFromCompanyCollection.IndexEntry,
              StockPriceTimeSeriesFromCompanyCollection>()
          .Where(x => x.TradeVolume >  150_000_000)
          .Select(x => x.CompanyID)
          .Distinct()
          .ToList();
  }
  
  // Results will contain company "Companies/65-A"
  // since it is the only company with time series entries having such high trade volume.

  from index "StockPriceTimeSeriesFromCompanyCollection"
  where "TradeVolume" > 150_000_000
  select distinct CompanyID

Map index - index all time series from single collection:

public class AllTimeSeriesFromCompanyCollection : AbstractTimeSeriesIndexCreationTask<Company>
{
    public class IndexEntry
    {
        public double Value { get; set; }
        public DateTime Date { get; set; }
    }
    
    public AllTimeSeriesFromCompanyCollection()
    {
        // Call 'AddMapForAll' to index ALL the time series in the 'Companies' collection 
        // ==============================================================================
        AddMapForAll(timeseries =>
            from segment in timeseries
            from entry in segment.Entries
                
            select new IndexEntry()
            {
                Value = entry.Value,
                Date = entry.Timestamp.Date
            });
    }
}

Map index - index all time series from all collections:

// Inherit from AbstractTimeSeriesIndexCreationTask<object>
// Specify <object> as the type to index from ALL collections
// ==========================================================

public class AllTimeSeriesFromAllCollections : AbstractTimeSeriesIndexCreationTask<object>
{
    public class IndexEntry
    {
        public double Value { get; set; }
        public DateTime Date { get; set; }
        public string DocumentID { get; set; }
    }
    
    public AllTimeSeriesFromAllCollections()
    {
        AddMapForAll(timeseries =>
            from segment in timeseries
            from entry in segment.Entries
                
            select new IndexEntry()
            {
                Value = entry.Value,
                Date = entry.Timestamp.Date,
                DocumentID = segment.DocumentId
            });
    }
}

Multi-Map index - index time series from several collections:

public class Vehicles_ByLocation : AbstractMultiMapTimeSeriesIndexCreationTask
{
    public class IndexEntry
    {
        public double Latitude { get; set; }
        public double Longitude { get; set; }
        public DateTime Date { get; set; }
        public string DocumentID { get; set; }
    }
    
    public Vehicles_ByLocation()
    {
        // Call 'AddMap' for each collection you wish to index
        // ===================================================
        
        AddMap<Plane>(
            "GPS_Coordinates",timeSeries =>
                from segment in timeSeries
                from entry in segment.Entries
                select new IndexEntry()
                {
                    Latitude = entry.Values[0],
                    Longitude = entry.Values[1],
                    Date = entry.Timestamp.Date,
                    DocumentID = segment.DocumentId
                });

        AddMap<Ship>(
            "GPS_Coordinates",timeSeries =>
                from segment in timeSeries
                from entry in segment.Entries
                select new IndexEntry()
                {
                    Latitude = entry.Values[0],
                    Longitude = entry.Values[1],
                    Date = entry.Timestamp.Date,
                    DocumentID = segment.DocumentId
                });
    }
}

Map-Reduce index:

public class TradeVolume_PerDay_ByCountry : 
    AbstractTimeSeriesIndexCreationTask<Company, TradeVolume_PerDay_ByCountry.Result>
{
    public class Result
    {
        public double TotalTradeVolume { get; set; }
        public DateTime Date { get; set; }
        public string Country { get; set; }
    }

    public TradeVolume_PerDay_ByCountry()
    {
        // Define the Map part:
        AddMap("StockPrices", timeSeries =>
            from segment in timeSeries
            from entry in segment.Entries
            
            let company = LoadDocument<Company>(segment.DocumentId)
            
            select new Result
            {
                Date = entry.Timestamp.Date,
                Country = company.Address.Country,
                TotalTradeVolume = entry.Values[4]
            });

        // Define the Reduce part:
        Reduce = results =>
            from r in results
            group r by new {r.Date, r.Country}
            into g
            select new Result
            {
                Date = g.Key.Date,
                Country = g.Key.Country,
                TotalTradeVolume = g.Sum(x => x.TotalTradeVolume)
            };
    }
}

AbstractTimeSeriesIndexCreationTask

// To define a Map index inherit from:
// ===================================
public abstract class AbstractTimeSeriesIndexCreationTask<TDocument> { }
// Time series that belong to documents of the specified `TDocument` type will be indexed. 

// To define a Map-Reduce index inherit from:
// ==========================================
public abstract class AbstractTimeSeriesIndexCreationTask<TDocument, TReduceResult> { }
// Specify both the document type and the reduce type

// Methods available in AbstractTimeSeriesIndexCreationTask class:
// ===============================================================

// Set a map function for the specified time series
protected void AddMap(string timeSeries,
    Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);

// Set a map function for all time series 
protected void AddMapForAll(
    Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);

AbstractMultiMapTimeSeriesIndexCreationTask

// To define a Multi-Map index inherit from:
// =========================================
public abstract class AbstractMultiMapTimeSeriesIndexCreationTask { }

// Methods available in AbstractMultiMapTimeSeriesIndexCreationTask class:
// =======================================================================

// Set a map function for all time series with the specified name
// that belong to documents of type `TSource`
protected void AddMap<TSource>(string timeSeries,
    Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);

// Set a map function for all time series that belong to documents of type `TBase`
// or any type that inherits from `TBase`
protected void AddMapForAll<TBase>(
    Expression<Func<IEnumerable<TimeSeriesSegment>,IEnumerable>> map);

AbstractJavaScriptTimeSeriesIndexCreationTask

// To define a JavaScript index inherit from:
// ==========================================
public abstract class AbstractJavaScriptTimeSeriesIndexCreationTask
{    
    public HashSet<string> Maps; // The set of JavaScript map functions for this index
    protected string Reduce;     // The JavaScript reduce function
}

Learn more about JavaScript indexes in JavaScript Indexes.

TimeSeriesIndexDefinition

public class TimeSeriesIndexDefinition : IndexDefinition

While TimeSeriesIndexDefinition is currently functionally equivalent to the regular IndexDefinition class from which it inherits, it is recommended to use TimeSeriesIndexDefinition when creating a time series index definition in case additional functionality is added in future versions of RavenDB.

TimeSeriesIndexDefinitionBuilder

public class TimeSeriesIndexDefinitionBuilder<TDocument>
{ 
    public TimeSeriesIndexDefinitionBuilder(string indexName = null)  
}

TimeSeriesSegment

• Segment properties include the entries data and aggregated values that RavenDB automatically updates in the segment's header.

• The following segment properties can be indexed:

  public sealed class TimeSeriesSegment
  {
      // The ID of the document this time series belongs to
      public string DocumentId { get; set; }
   
      // The name of the time series this segment belongs to
      public string Name { get; set; }
    
      // The smallest values from all entries in the segment
      // The first array item is the Min of all first values, etc.
      public double[] Min { get; set; }
  
      // The largest values from all entries in the segment
      // The first array item is the Max of all first values, etc.
      public double[] Max { get; set; }
    
      // The sum of all values from all entries in the segment 
      // The first array item is the Sum of all first values, etc.
      public double[] Sum { get; set; }
    
      // The number of entries in the segment
      public int Count { get; set; }
    
      // The timestamp of the first entry in the segment
      public DateTime Start { get; set; }
    
      // The timestamp of the last entry in the segment
      public DateTime End { get; set; }
    
      // The segment's entries themselves
      public TimeSeriesEntry[] Entries { get; set; }
  }

• These are the properties of a TimeSeriesEntry which can be indexed:

  public class TimeSeriesEntry
  {
      public DateTime Timestamp;
      public string Tag;
      public double[] Values;
  
      // This is exactly equivalent to Values[0]
      public double Value;
  }