Time Series Rollups and Retention

What are rollups?

A rollup is a time series that summarizes the data from another time series, with each rollup entry representing a specific time frame in the original time series.
Each rollup entry contains 6 values that aggregate the data from all the entries in the original time frame:

• First - the value of the first entry in the frame.
• Last - the value of the last entry.
• Min - the smallest value.
• Max - the largest value.
• Sum - the sum of all the values in the frame.
• Count - the total number of entries in the frame.

This results in a much more compact time series that still contains useful information about the original time series (also called "raw" time series).

Rollup policies:

Rollup time series are created automatically according to rollup policies that can be defined from Studio or client code.

• A rollup policy applies to all time series of every document in the given collection.

• Each collection can be configured to have multiple policies which are applied sequentially:

  • The raw time series is first rolled up using the policy with the shortest aggregation frame.
  • Subsequently, the resulting rollup time series is further aggregated using the policy with the next shortest aggregation frame,
    and so on.

Querying with group-by will transparently traverse over the rollups to retrieve the relevant results.

Let's look at an example of rollup data:

A rollup time series' entries

1) Name:
The name of a rollup time series has this format: <name of raw time series>@<name of time series policy>
It is a combination of the name of the raw time series and the name of the time series policy separated by @.
In the image above these are "HeartRates" and "byHour" respectively.
For this reason, neither a time series name nor a policy name can have the character @ in it.

2) Timestamp:
The aggregation frame always begins at a round number of one of these time units: a second, minute, hour, day, week, month, or year. So the frame includes all entries starting at a round number of time units, and ending at a round number minus one millisecond (since milliseconds are the minimal resolution in RavenDB time series). The timestamp for a rollup entry is the beginning of the frame it represents.

For example, if the aggregation frame is three days, a frame will start and end at a time stamps like:
2020-01-01 00:00:00 - 2020-01-03 23:59:59.999.

3) Values:
Each group of six values represents one value from the original entries. If the raw time series has n values per entry, the rollup time series will have 6 * n per entry: the first six summarize the first raw value, the next six summarize the next raw value, and so on. The aggregated values have the names: "First (<name of raw value>)", "Last (<name of raw value>)", ... respectively.

Because time series entries are limited to 32 values, rollups are limited to the first five values of an original time series entry, or 30 aggregate values.

Create time series policies:

var oneWeek = TimeValue.FromDays(7);
var fiveYears = TimeValue.FromYears(5);

// Define a policy on the RAW time series data:
// ============================================
var rawPolicy = new RawTimeSeriesPolicy(fiveYears); // Retain entries for five years

// Define a ROLLUP policy: 
// =======================
var rollupPolicy = new TimeSeriesPolicy(
        "By1WeekFor1Year", // Name of policy
        oneWeek, // Aggregation time, roll-up the data for each week
        fiveYears); // Retention time, keep data for five years

// Define the time series configuration for collection "Companies" (use above policies):
// =====================================================================================
var timeSeriesConfig = new TimeSeriesConfiguration();
timeSeriesConfig.Collections["Companies"] = new TimeSeriesCollectionConfiguration
{
    Policies = new List<TimeSeriesPolicy> { rollupPolicy },
    RawPolicy = rawPolicy
};
               
// Deploy the time series configuration to the server
// by sending the 'ConfigureTimeSeriesOperation' operation:
// ========================================================
store.Maintenance.Send(new ConfigureTimeSeriesOperation(timeSeriesConfig));

// NOTE:
// The time series entries in the RavenDB sample data are dated up to the year 2020.
// To ensure that you see the rollup time series created when running this example,
// the retention time should be set to exceed that year.

Retrieve rollup data:

• Retrieving entries from a rollup time series is similar to getting the raw time series data.

• Learn more about using TimeSeriesFor.Get in Get time series entries.

// Get all data from the RAW time series:
// ======================================

var rawData = session
    .TimeSeriesFor("companies/91-A", "StockPrices")
    .Get(DateTime.MinValue, DateTime.MaxValue);

// Get all data from the ROLLUP time series:
// =========================================

// Either - pass the rollup name explicitly to 'TimeSeriesFor':
var rollupData = session
    .TimeSeriesFor("companies/91-A", "StockPrices@By1WeekFor1Year")
    .Get(DateTime.MinValue, DateTime.MaxValue);

// Or - get the rollup name by calling 'GetTimeSeriesName':
rollupData = session
    .TimeSeriesFor("companies/91-A", rollupPolicy.GetTimeSeriesName("StockPrices"))
    .Get(DateTime.MinValue, DateTime.MaxValue);

// The raw time series has 100 entries
Assert.Equal(rawData.Length, 100);
Assert.Equal(rawData[0].IsRollup, false);

// The rollup time series has only 22 entries
// as each entry aggregates 1 week's data from the raw time series
Assert.Equal(rollupData.Length, 22);
Assert.Equal(rollupData[0].IsRollup, true);

The time series policies

• Raw policy:

  • Used to define the retention time of the raw time series.
  • Only one such policy per collection can be defined.
  • Does not perform aggregation.

• Rollup policy:

  • Used to define the aggregation time frame and retention time for the rollup time series.
  • Multiple policies can be defined per collection.

public class RawTimeSeriesPolicy : TimeSeriesPolicy
{
    public TimeValue RetentionTime;
}

public class TimeSeriesPolicy
{
    public string Name;
    public TimeValue RetentionTime; { get; protected set; }
    public TimeValue AggregationTime; { get; private set; }
}

public struct TimeValue
{
    public static TimeValue FromSeconds(int seconds);
    public static TimeValue FromMinutes(int minutes);
    public static TimeValue FromHours(int hours);
    public static TimeValue FromDays(int days);
    public static TimeValue FromMonths(int months);
    public static TimeValue FromYears(int years);
}

Each of the above TimeValue methods returns a TimeValue object representing a whole number of the specified time units. These methods are used to define the aggregation and retention spans om time series policies.

The time series configuration object

public class TimeSeriesConfiguration
{
    public Dictionary<string, TimeSeriesCollectionConfiguration> Collections;
}

public class TimeSeriesCollectionConfiguration
{
    public bool Disabled;
    public List<TimeSeriesPolicy> Policies;
    public RawTimeSeriesPolicy RawPolicy;
}

The time series configuration operation

public ConfigureTimeSeriesOperation(TimeSeriesConfiguration configuration);

Learn more about operations in: What are operations.

Casting time series entries

Time series entries are of one of the following classes:

public class TimeSeriesEntry {   }
public class TimeSeriesEntry<T> : TimeSeriesEntry {   }
public class TimeSeriesRollupEntry<TValues> : TimeSeriesEntry {   }

If you have an existing rollup entry of type TimeSeriesEntry,
you can cast it to a TimeSeriesRollupEntry using AsRollupEntry().

public static TimeSeriesRollupEntry<T> AsRollupEntry<T>(this TimeSeriesEntry<T> entry);

You can cast a TimeSeriesRollupEntry to a TimeSeriesEntry directly.
Its values will consist of all the First values of the rollup entry.

var rollupEntry = new TimeSeriesRollupEntry<int>(new DateTime(2020,1,1));
TimeSeriesEntry<int> TSEntry = (TimeSeriesEntry<int>)rollupEntry;

Read more about time series with generic types here.