Indexing Time Series
-
Static time series indexes can be created from your client application or from the Studio.
-
Indexing allows for fast retrieval of the indexed time series data when querying a time series.
-
In this page:
Time series indexes vs Document indexes
Auto-Indexes:
-
Time series index:
Dynamic time series indexes are Not created in response to queries. -
Document index:
Auto-indexes are created in response to dynamic queries.
Data source:
-
Time series index:
- Time series indexes process segments that contain time series entries.
The entries are indexed through the segment they are stored in, for example, using a LINQ syntax that resembles this one:
from segment in timeseries from entry in segment ...
-
The following items can be indexed per index-entry in a time series index:
- Values & timestamp of a time series entry
- The entry tag
- Content from a document referenced by the tag
- Properties of the containing segment (see
TimeSeriesSegment
)
- Time series indexes process segments that contain time series entries.
-
Document index:
- The index processes fields from your JSON documents.
Documents are indexed through the collection they belong to, for example, using this LINQ syntax:
from employee in employees ...
- The index processes fields from your JSON documents.
Query results:
-
Time series index:
When querying a time series index, each result item corresponds to the type defined by the index-entry in the index definition, (unless results are projected). The documents themselves are not returned. -
Document index:
The resulting objects are the document entities (unless results are projected).
Ways to create a time series index
There are two main ways to create a time series index:
-
Create a class that inherits from one of the following abstract index creation task classes:
AbstractTimeSeriesIndexCreationTask
for map and map-reduce time series indexes.AbstractMultiMapTimeSeriesIndexCreationTask
for multi-map time series indexes.AbstractJavaScriptTimeSeriesIndexCreationTask
for static javascript indexes.
-
Deploy a time series index definition via PutIndexesOperation:
- Create a
TimeSeriesIndexDefinition
directly. - Create a strongly typed index definition using
TimeSeriesIndexDefinitionBuilder
.
- Create a
Examples of time series indexes
Map index - index single time series from single collection:
-
In this index, we index data from the "StockPrices" time series entries in the "Companies" collection (
TradeVolume
,Date
). -
In addition, we index the containing document id (
DocumentID
), which is obtained from the segment,
and some content from the document referenced by the entry's Tag (EmployeeName
). -
Each tab below presents one of the different ways the index can be defined.
public class StockPriceTimeSeriesFromCompanyCollection : AbstractTimeSeriesIndexCreationTask<Company> { // The index-entry: // ================ public class IndexEntry { // The index-fields: // ================= public double TradeVolume { get; set; } public DateTime Date { get; set; } public string CompanyID { get; set; } public string EmployeeName { get; set; } } public StockPriceTimeSeriesFromCompanyCollection() { // Call 'AddMap', specify the time series name to be indexed AddMap("StockPrices", timeseries => from segment in timeseries from entry in segment.Entries // Can load the document referenced in the TAG: let employee = LoadDocument<Employee>(entry.Tag) // Define the content of the index-fields: // ======================================= select new IndexEntry() { // Retrieve content from the time series ENTRY: TradeVolume = entry.Values[4], Date = entry.Timestamp.Date, // Retrieve content from the SEGMENT: CompanyID = segment.DocumentId, // Retrieve content from the loaded DOCUMENT: EmployeeName = employee.FirstName + " " + employee.LastName }); } }
public class StockPriceTimeSeriesFromCompanyCollection_NonTyped : AbstractTimeSeriesIndexCreationTask { public override TimeSeriesIndexDefinition CreateIndexDefinition() { return new TimeSeriesIndexDefinition { Name = "StockPriceTimeSeriesFromCompanyCollection_NonTyped", Maps = { @" from segment in timeSeries.Companies.StockPrices from entry in segment.Entries let employee = LoadDocument(entry.Tag, ""Employees"") select new { TradeVolume = entry.Values[4], Date = entry.Timestamp.Date, CompanyID = segment.DocumentId, EmployeeName = employee.FirstName + ' ' + employee.LastName }" } }; } }
public class StockPriceTimeSeriesFromCompanyCollection_JS : AbstractJavaScriptTimeSeriesIndexCreationTask { public StockPriceTimeSeriesFromCompanyCollection_JS() { Maps = new HashSet<string> { @" timeSeries.map('Companies', 'StockPrices', function (segment) { return segment.Entries.map(entry => { let employee = load(entry.Tag, 'Employees'); return { TradeVolume: entry.Values[4], Date: new Date(entry.Timestamp.getFullYear(), entry.Timestamp.getMonth(), entry.Timestamp.getDate()), CompanyID: segment.DocumentId, EmployeeName: employee.FirstName + ' ' + employee.LastName }; }); })" }; } }
// Define the 'index definition' var indexDefinition = new TimeSeriesIndexDefinition { Name = "StockPriceTimeSeriesFromCompanyCollection ", Maps = { @" from segment in timeSeries.Companies.StockPrices from entry in segment.Entries let employee = LoadDocument(entry.Tag, ""Employees"") select new { TradeVolume = entry.Values[4], Date = entry.Timestamp.Date, CompanyID = segment.DocumentId, EmployeeName = employee.FirstName + ' ' + employee.LastName }" } }; // Deploy the index to the server via 'PutIndexesOperation' documentStore.Maintenance.Send(new PutIndexesOperation(indexDefinition));
// Create the index builder var TSIndexDefBuilder = new TimeSeriesIndexDefinitionBuilder<Company>("StockPriceTimeSeriesFromCompanyCollection "); TSIndexDefBuilder.AddMap("StockPrices", timeseries => from segment in timeseries from entry in segment.Entries // Note: // Class TimeSeriesIndexDefinitionBuilder does not support the 'LoadDocument' API method. // Use one of the other index creation methods if needed. select new { TradeVolume = entry.Values[4], Date = entry.Timestamp.Date, ComapnyID = segment.DocumentId }); // Build the index definition var indexDefinitionFromBuilder = TSIndexDefBuilder.ToIndexDefinition(documentStore.Conventions); // Deploy the index to the server via 'PutIndexesOperation' documentStore.Maintenance.Send(new PutIndexesOperation(indexDefinitionFromBuilder));
-
Querying this index, you can retrieve the indexed time series data while filtering by any of the index-fields.
using (var session = documentStore.OpenSession()) { // Retrieve time series data for the specified company: // ==================================================== List<StockPriceTimeSeriesFromCompanyCollection.IndexEntry> results = session .Query<StockPriceTimeSeriesFromCompanyCollection.IndexEntry, StockPriceTimeSeriesFromCompanyCollection>() .Where(x => x.CompanyID == "Companies/91-A") .ToList(); } // Results will include data from all 'StockPrices' entries in document 'Companies/91-A'.
from index "StockPriceTimeSeriesFromCompanyCollection" where "CompanyID" == "Comapnies/91-A"
using (var session = documentStore.OpenSession()) { // Find what companies had a very high trade volume: // ================================================== List<string> results = session .Query<StockPriceTimeSeriesFromCompanyCollection.IndexEntry, StockPriceTimeSeriesFromCompanyCollection>() .Where(x => x.TradeVolume > 150_000_000) .Select(x => x.CompanyID) .Distinct() .ToList(); } // Results will contain company "Companies/65-A" // since it is the only company with time series entries having such high trade volume.
from index "StockPriceTimeSeriesFromCompanyCollection" where "TradeVolume" > 150_000_000 select distinct CompanyID
Map index - index all time series from single collection:
public class AllTimeSeriesFromCompanyCollection : AbstractTimeSeriesIndexCreationTask<Company>
{
public class IndexEntry
{
public double Value { get; set; }
public DateTime Date { get; set; }
}
public AllTimeSeriesFromCompanyCollection()
{
// Call 'AddMapForAll' to index ALL the time series in the 'Companies' collection
// ==============================================================================
AddMapForAll(timeseries =>
from segment in timeseries
from entry in segment.Entries
select new IndexEntry()
{
Value = entry.Value,
Date = entry.Timestamp.Date
});
}
}
Map index - index all time series from all collections:
// Inherit from AbstractTimeSeriesIndexCreationTask<object>
// Specify <object> as the type to index from ALL collections
// ==========================================================
public class AllTimeSeriesFromAllCollections : AbstractTimeSeriesIndexCreationTask<object>
{
public class IndexEntry
{
public double Value { get; set; }
public DateTime Date { get; set; }
public string DocumentID { get; set; }
}
public AllTimeSeriesFromAllCollections()
{
AddMapForAll(timeseries =>
from segment in timeseries
from entry in segment.Entries
select new IndexEntry()
{
Value = entry.Value,
Date = entry.Timestamp.Date,
DocumentID = segment.DocumentId
});
}
}
Multi-Map index - index time series from several collections:
public class Vehicles_ByLocation : AbstractMultiMapTimeSeriesIndexCreationTask
{
public class IndexEntry
{
public double Latitude { get; set; }
public double Longitude { get; set; }
public DateTime Date { get; set; }
public string DocumentID { get; set; }
}
public Vehicles_ByLocation()
{
// Call 'AddMap' for each collection you wish to index
// ===================================================
AddMap<Plane>(
"GPS_Coordinates",timeSeries =>
from segment in timeSeries
from entry in segment.Entries
select new IndexEntry()
{
Latitude = entry.Values[0],
Longitude = entry.Values[1],
Date = entry.Timestamp.Date,
DocumentID = segment.DocumentId
});
AddMap<Ship>(
"GPS_Coordinates",timeSeries =>
from segment in timeSeries
from entry in segment.Entries
select new IndexEntry()
{
Latitude = entry.Values[0],
Longitude = entry.Values[1],
Date = entry.Timestamp.Date,
DocumentID = segment.DocumentId
});
}
}
Map-Reduce index:
public class TradeVolume_PerDay_ByCountry :
AbstractTimeSeriesIndexCreationTask<Company, TradeVolume_PerDay_ByCountry.Result>
{
public class Result
{
public double TotalTradeVolume { get; set; }
public DateTime Date { get; set; }
public string Country { get; set; }
}
public TradeVolume_PerDay_ByCountry()
{
// Define the Map part:
AddMap("StockPrices", timeSeries =>
from segment in timeSeries
from entry in segment.Entries
let company = LoadDocument<Company>(segment.DocumentId)
select new Result
{
Date = entry.Timestamp.Date,
Country = company.Address.Country,
TotalTradeVolume = entry.Values[4]
});
// Define the Reduce part:
Reduce = results =>
from r in results
group r by new {r.Date, r.Country}
into g
select new Result
{
Date = g.Key.Date,
Country = g.Key.Country,
TotalTradeVolume = g.Sum(x => x.TotalTradeVolume)
};
}
}
Syntax
AbstractTimeSeriesIndexCreationTask
// To define a Map index inherit from:
// ===================================
public abstract class AbstractTimeSeriesIndexCreationTask<TDocument> { }
// Time series that belong to documents of the specified `TDocument` type will be indexed.
// To define a Map-Reduce index inherit from:
// ==========================================
public abstract class AbstractTimeSeriesIndexCreationTask<TDocument, TReduceResult> { }
// Specify both the document type and the reduce type
// Methods available in AbstractTimeSeriesIndexCreationTask class:
// ===============================================================
// Set a map function for the specified time series
protected void AddMap(string timeSeries,
Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);
// Set a map function for all time series
protected void AddMapForAll(
Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);
AbstractMultiMapTimeSeriesIndexCreationTask
// To define a Multi-Map index inherit from:
// =========================================
public abstract class AbstractMultiMapTimeSeriesIndexCreationTask { }
// Methods available in AbstractMultiMapTimeSeriesIndexCreationTask class:
// =======================================================================
// Set a map function for all time series with the specified name
// that belong to documents of type `TSource`
protected void AddMap<TSource>(string timeSeries,
Expression<Func<IEnumerable<TimeSeriesSegment>, IEnumerable>> map);
// Set a map function for all time series that belong to documents of type `TBase`
// or any type that inherits from `TBase`
protected void AddMapForAll<TBase>(
Expression<Func<IEnumerable<TimeSeriesSegment>,IEnumerable>> map);
AbstractJavaScriptTimeSeriesIndexCreationTask
// To define a JavaScript index inherit from:
// ==========================================
public abstract class AbstractJavaScriptTimeSeriesIndexCreationTask
{
public HashSet<string> Maps; // The set of JavaScript map functions for this index
protected string Reduce; // The JavaScript reduce function
}
Learn more about JavaScript indexes in JavaScript Indexes.
TimeSeriesIndexDefinition
public class TimeSeriesIndexDefinition : IndexDefinition
While TimeSeriesIndexDefinition
is currently functionally equivalent to the regular IndexDefinition
class from which it inherits,
it is recommended to use TimeSeriesIndexDefinition
when creating a time series index definition in case additional functionality is added in future versions of RavenDB.
TimeSeriesIndexDefinitionBuilder
public class TimeSeriesIndexDefinitionBuilder<TDocument>
{
public TimeSeriesIndexDefinitionBuilder(string indexName = null)
}
Note:
-
Currently, class
TimeSeriesIndexDefinitionBuilder
does Not support API methods from abstract classAbstractCommonApiForIndexes
, such asLoadDocument
orRecurse
. -
Use one of the other index creation methods if needed.
TimeSeriesSegment
-
Segment properties include the entries data and aggregated values that RavenDB automatically updates in the segment's header.
-
The following segment properties can be indexed:
public sealed class TimeSeriesSegment { // The ID of the document this time series belongs to public string DocumentId { get; set; } // The name of the time series this segment belongs to public string Name { get; set; } // The smallest values from all entries in the segment // The first array item is the Min of all first values, etc. public double[] Min { get; set; } // The largest values from all entries in the segment // The first array item is the Max of all first values, etc. public double[] Max { get; set; } // The sum of all values from all entries in the segment // The first array item is the Sum of all first values, etc. public double[] Sum { get; set; } // The number of entries in the segment public int Count { get; set; } // The timestamp of the first entry in the segment public DateTime Start { get; set; } // The timestamp of the last entry in the segment public DateTime End { get; set; } // The segment's entries themselves public TimeSeriesEntry[] Entries { get; set; } }
-
These are the properties of a
TimeSeriesEntry
which can be indexed:public class TimeSeriesEntry { public DateTime Timestamp; public string Tag; public double[] Values; // This is exactly equivalent to Values[0] public double Value; }