Indexing Time Series



Time series indexes vs Document indexes

Auto-Indexes:

  • Time series index:
    Dynamic time series indexes are Not created in response to queries.

  • Document index:
    Auto-indexes are created in response to dynamic queries.


Data source:

  • Time series index:

    • Time series indexes process segments that contain time series entries.
      The entries are indexed through the segment they are stored in, for example, using a LINQ syntax that resembles this one:

    from segment in timeseries
    from entry in segment
    ...
    • The following items can be indexed per index-entry in a time series index:
      • Values & timestamp of a time series entry
      • The entry tag
      • Content from a document referenced by the tag
      • Properties of the containing segment (see TimeSeriesSegment)
  • Document index:

    • The index processes fields from your JSON documents.
      Documents are indexed through the collection they belong to, for example, using this LINQ syntax:

    from employee in employees
    ...

Query results:

  • Time series index:
    When querying a time series index, each result item corresponds to the type defined by the index-entry in the index definition, (unless results are projected). The documents themselves are not returned.

  • Document index:
    The resulting objects are the document entities (unless results are projected).

Ways to create a time series index

There are two main ways to create a time series index:

  1. Create a class that inherits from one of the following abstract index creation task classes:

  2. Deploy a time series index definition via PutIndexesOperation:

Examples of time series indexes

Map index - index single time series from single collection:

  • In this index, we index data from the "StockPrices" time series entries in the "Companies" collection (TradeVolume, Date).

  • In addition, we index the containing document id (DocumentID), which is obtained from the segment,
    and some content from the document referenced by the entry's Tag (EmployeeName).

  • Each tab below presents one of the different ways the index can be defined.

    public class StockPriceTimeSeriesFromCompanyCollection : AbstractTimeSeriesIndexCreationTask<Company>
    {
        // The index-entry:
        // ================
        public class IndexEntry
        {
            // The index-fields:
            // =================
            public double TradeVolume { get; set; }
            public DateTime Date { get; set; }
            public string CompanyID { get; set; }
            public string EmployeeName { get; set; }
        }
        
        public StockPriceTimeSeriesFromCompanyCollection()
        { 
            // Call 'AddMap', specify the time series name to be indexed 
            AddMap("StockPrices", timeseries =>
                    from segment in timeseries
                    from entry in segment.Entries
                    
                    // Can load the document referenced in the TAG:
                    let employee = LoadDocument<Employee>(entry.Tag)
                    
                    // Define the content of the index-fields:
                    // =======================================
                    select new IndexEntry()
                    {
                        // Retrieve content from the time series ENTRY:
                        TradeVolume = entry.Values[4],
                        Date = entry.Timestamp.Date,
                        
                        // Retrieve content from the SEGMENT:
                        CompanyID = segment.DocumentId,
                        
                        // Retrieve content from the loaded DOCUMENT:
                        EmployeeName = employee.FirstName + " " + employee.LastName 
                    });
        }
    }
    public class StockPriceTimeSeriesFromCompanyCollection_NonTyped : AbstractTimeSeriesIndexCreationTask 
    {
        public override TimeSeriesIndexDefinition CreateIndexDefinition()
        {
            return new TimeSeriesIndexDefinition
            {
                Name = "StockPriceTimeSeriesFromCompanyCollection_NonTyped",
                Maps =
                {
                    @"
                    from segment in timeSeries.Companies.StockPrices 
                    from entry in segment.Entries
      
                    let employee = LoadDocument(entry.Tag, ""Employees"")
    
                    select new 
                    { 
                        TradeVolume = entry.Values[4], 
                        Date = entry.Timestamp.Date,
                        CompanyID = segment.DocumentId,
                        EmployeeName = employee.FirstName + ' ' + employee.LastName 
                    }"
                }
            };
        }
    }
    public class StockPriceTimeSeriesFromCompanyCollection_JS : 
        AbstractJavaScriptTimeSeriesIndexCreationTask
    {
        public StockPriceTimeSeriesFromCompanyCollection_JS()
        {
            Maps = new HashSet<string> { @"
                timeSeries.map('Companies', 'StockPrices', function (segment) {
    
                    return segment.Entries.map(entry => {
                        let employee = load(entry.Tag, 'Employees');
    
                        return {
                            TradeVolume: entry.Values[4],
                            Date: new Date(entry.Timestamp.getFullYear(),
                                           entry.Timestamp.getMonth(),
                                           entry.Timestamp.getDate()),
                            CompanyID: segment.DocumentId,
                            EmployeeName: employee.FirstName + ' ' + employee.LastName
                        };
                    });
                })"
            };
        }
    }
    // Define the 'index definition'
    var indexDefinition = new TimeSeriesIndexDefinition
        {
            Name = "StockPriceTimeSeriesFromCompanyCollection ",
            Maps =
            {
                @"
                from segment in timeSeries.Companies.StockPrices 
                from entry in segment.Entries 
    
                let employee = LoadDocument(entry.Tag, ""Employees"")
    
                select new 
                { 
                    TradeVolume = entry.Values[4], 
                    Date = entry.Timestamp.Date,
                    CompanyID = segment.DocumentId,
                    EmployeeName = employee.FirstName + ' ' + employee.LastName 
                }"
            }
        };
    
    // Deploy the index to the server via 'PutIndexesOperation'
    documentStore.Maintenance.Send(new PutIndexesOperation(indexDefinition));
    // Create the index builder
    var TSIndexDefBuilder =
        new TimeSeriesIndexDefinitionBuilder<Company>("StockPriceTimeSeriesFromCompanyCollection ");
    
    TSIndexDefBuilder.AddMap("StockPrices", timeseries => 
        from segment in timeseries
        from entry in segment.Entries
        
        // Note:
        // Class TimeSeriesIndexDefinitionBuilder does not support the 'LoadDocument' API method.
        // Use one of the other index creation methods if needed.
        
        select new
        {
            TradeVolume = entry.Values[4],
            Date = entry.Timestamp.Date,
            ComapnyID = segment.DocumentId
        });
    
    // Build the index definition
    var indexDefinitionFromBuilder = TSIndexDefBuilder.ToIndexDefinition(documentStore.Conventions);
    
    // Deploy the index to the server via 'PutIndexesOperation'
    documentStore.Maintenance.Send(new PutIndexesOperation(indexDefinitionFromBuilder));
  • Querying this index, you can retrieve the indexed time series data while filtering by any of the index-fields.

    using (var session = documentStore.OpenSession())
    {
        // Retrieve time series data for the specified company:
        // ====================================================
        List<StockPriceTimeSeriesFromCompanyCollection.IndexEntry> results = session
           .Query<StockPriceTimeSeriesFromCompanyCollection.IndexEntry,
               StockPriceTimeSeriesFromCompanyCollection>()
           .Where(x => x.CompanyID == "Companies/91-A")
           .ToList();
    }
    
    // Results will include data from all 'StockPrices' entries in document 'Companies/91-A'. 
    from index "StockPriceTimeSeriesFromCompanyCollection"
    where "CompanyID" == "Comapnies/91-A"
    using (var session = documentStore.OpenSession())
    {
        // Find what companies had a very high trade volume:
        // ==================================================
        List<string> results = session
            .Query<StockPriceTimeSeriesFromCompanyCollection.IndexEntry,
                StockPriceTimeSeriesFromCompanyCollection>()
            .Where(x => x.TradeVolume >  150_000_000)
            .Select(x => x.CompanyID)
            .Distinct()
            .ToList();
    }
    
    // Results will contain company "Companies/65-A"
    // since it is the only company with time series entries having such high trade volume.
    from index "StockPriceTimeSeriesFromCompanyCollection"
    where "TradeVolume" > 150_000_000
    select distinct CompanyID

Map index - index all time series from single collection:

public class AllTimeSeriesFromCompanyCollection : AbstractTimeSeriesIndexCreationTask<Company>
{
    public class IndexEntry
    {
        public double Value { get; set; }
        public DateTime Date { get; set; }
    }
    
    public AllTimeSeriesFromCompanyCollection()
    {
        // Call 'AddMapForAll' to index ALL the time series in the 'Companies' collection 
        // ==============================================================================
        AddMapForAll(timeseries =>
            from segment in timeseries
            from entry in segment.Entries
                
            select new IndexEntry()
            {
                Value = entry.Value,
                Date = entry.Timestamp.Date
            });
    }
}

Map index - index all time series from all collections:

// Inherit from AbstractTimeSeriesIndexCreationTask<object>
// Specify <object> as the type to index from ALL collections
// ==========================================================

public class AllTimeSeriesFromAllCollections : AbstractTimeSeriesIndexCreationTask<object>
{
    public class IndexEntry
    {
        public double Value { get; set; }
        public DateTime Date { get; set; }
        public string DocumentID { get; set; }
    }
    
    public AllTimeSeriesFromAllCollections()
    {
        AddMapForAll(timeseries =>
            from segment in timeseries
            from entry in segment.Entries
                
            select new IndexEntry()
            {
                Value = entry.Value,
                Date = entry.Timestamp.Date,
                DocumentID = segment.DocumentId
            });
    }
}

Multi-Map index - index time series from several collections:

public class Vehicles_ByLocation : AbstractMultiMapTimeSeriesIndexCreationTask
{
    public class IndexEntry
    {
        public double Latitude { get; set; }
        public double Longitude { get; set; }
        public DateTime Date { get; set; }
        public string DocumentID { get; set; }
    }
    
    public Vehicles_ByLocation()
    {
        // Call 'AddMap' for each collection you wish to index
        // ===================================================
        
        AddMap<Plane>(
            "GPS_Coordinates",timeSeries =>
                from segment in timeSeries
                from entry in segment.Entries
                select new IndexEntry()
                {
                    Latitude = entry.Values[0],
                    Longitude = entry.Values[1],
                    Date = entry.Timestamp.Date,
                    DocumentID = segment.DocumentId
                });

        AddMap<Ship>(
            "GPS_Coordinates",timeSeries =>
                from segment in timeSeries
                from entry in segment.Entries
                select new IndexEntry()
                {
                    Latitude = entry.Values[0],
                    Longitude = entry.Values[1],
                    Date = entry.Timestamp.Date,
                    DocumentID = segment.DocumentId
                });
    }
}

Map-Reduce index:

public class TradeVolume_PerDay_ByCountry : 
    AbstractTimeSeriesIndexCreationTask<Company, TradeVolume_PerDay_ByCountry.Result>
{
    public class Result
    {
        public double TotalTradeVolume { get; set; }
        public DateTime Date { get; set; }
        public string Country { get; set; }
    }

    public TradeVolume_PerDay_ByCountry()
    {
        // Define the Map part:
        AddMap("StockPrices", timeSeries =>
            from segment in timeSeries
            from entry in segment.Entries
            
            let company = LoadDocument<Company>(segment.DocumentId)
            
            select new Result
            {
                Date = entry.Timestamp.Date,
                Country = company.Address.Country,
                TotalTradeVolume = entry.Values[4]
            });

        // Define the Reduce part:
        Reduce = results =>
            from r in results
            group r by new {r.Date, r.Country}
            into g
            select new Result
            {
                Date = g.Key.Date,
                Country = g.Key.Country,
                TotalTradeVolume = g.Sum(x => x.TotalTradeVolume)
            };
    }
}

Syntax


AbstractTimeSeriesIndexCreationTask

// To define a Map index inherit from:
// ===================================
public abstract class AbstractTimeSeriesIndexCreationTask<TDocument> { }
// Time series that belong to documents of the specified `TDocument` type will be indexed. 

// To define a Map-Reduce index inherit from:
// ==========================================
public abstract class AbstractTimeSeriesIndexCreationTask<TDocument, TReduceResult> { }
// Specify both the document type and the reduce type

// Methods available in AbstractTimeSeriesIndexCreationTask class:
// ===============================================================

// Set a map function for the specified time series
protected void AddMap(string timeSeries,
    Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);

// Set a map function for all time series 
protected void AddMapForAll(
    Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);

AbstractMultiMapTimeSeriesIndexCreationTask

// To define a Multi-Map index inherit from:
// =========================================
public abstract class AbstractMultiMapTimeSeriesIndexCreationTask { }

// Methods available in AbstractMultiMapTimeSeriesIndexCreationTask class:
// =======================================================================

// Set a map function for all time series with the specified name
// that belong to documents of type `TSource`
protected void AddMap<TSource>(string timeSeries,
    Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);

// Set a map function for all time series that belong to documents of type `TBase`
// or any type that inherits from `TBase`
protected void AddMapForAll<TBase>(
    Expression<Func<IEnumerable<TimeSeriesSegment>,IEnumerable>> map);

AbstractJavaScriptTimeSeriesIndexCreationTask

// To define a JavaScript index inherit from:
// ==========================================
public abstract class AbstractJavaScriptTimeSeriesIndexCreationTask
{    
    public HashSet<string> Maps; // The set of JavaScript map functions for this index
    protected string Reduce;     // The JavaScript reduce function
}

Learn more about JavaScript indexes in JavaScript Indexes.


TimeSeriesIndexDefinition

public class TimeSeriesIndexDefinition : IndexDefinition

While TimeSeriesIndexDefinition is currently functionally equivalent to the regular IndexDefinition class from which it inherits, it is recommended to use TimeSeriesIndexDefinition when creating a time series index definition in case additional functionality is added in future versions of RavenDB.


TimeSeriesIndexDefinitionBuilder

public class TimeSeriesIndexDefinitionBuilder<TDocument>
{ 
    public TimeSeriesIndexDefinitionBuilder(string indexName = null)  
}

Note:

  • Currently, class TimeSeriesIndexDefinitionBuilder does Not support API methods from abstract class AbstractCommonApiForIndexes, such as LoadDocument or Recurse.

  • Use one of the other index creation methods if needed.


TimeSeriesSegment

  • Segment properties include the entries data and aggregated values that RavenDB automatically updates in the segment's header.

  • The following segment properties can be indexed:

    public sealed class TimeSeriesSegment
    {
        // The ID of the document this time series belongs to
        public string DocumentId { get; set; }
     
        // The name of the time series this segment belongs to
        public string Name { get; set; }
      
        // The smallest values from all entries in the segment
        // The first array item is the Min of all first values, etc.
        public double[] Min { get; set; }
    
        // The largest values from all entries in the segment
        // The first array item is the Max of all first values, etc.
        public double[] Max { get; set; }
      
        // The sum of all values from all entries in the segment 
        // The first array item is the Sum of all first values, etc.
        public double[] Sum { get; set; }
      
        // The number of entries in the segment
        public int Count { get; set; }
      
        // The timestamp of the first entry in the segment
        public DateTime Start { get; set; }
      
        // The timestamp of the last entry in the segment
        public DateTime End { get; set; }
      
        // The segment's entries themselves
        public TimeSeriesEntry[] Entries { get; set; }
    }
  • These are the properties of a TimeSeriesEntry which can be indexed:

    public class TimeSeriesEntry
    {
        public DateTime Timestamp;
        public string Tag;
        public double[] Values;
    
        // This is exactly equivalent to Values[0]
        public double Value;
    }