Time Series Rollups and Retention

What are rollups?

A rollup is a time series that summarizes the data from another time series, with each rollup entry representing a specific time frame in the original time series.
Each rollup entry contains 6 values that aggregate the data from all the entries in the original time frame:

• First - the value of the first entry in the frame.
• Last - the value of the last entry.
• Min - the smallest value.
• Max - the largest value.
• Sum - the sum of all the values in the frame.
• Count - the total number of entries in the frame.

This results in a much more compact time series that still contains useful information about the original time series (also called "raw" time series).

Rollup policies:

Rollup time series are created automatically according to rollup policies that can be defined from the tudio or client code.

• A rollup policy applies to all time series of every document in the given collection.

• Each collection can be configured to have multiple policies which are applied sequentially:

  • The raw time series is first rolled up using the policy with the shortest aggregation frame.
  • Subsequently, the resulting rollup time series is further aggregated using the policy with the next shortest aggregation frame,
    and so on.

Querying with group-by will transparently traverse over the rollups to retrieve the relevant results.

Let's look at an example of rollup data:

A rollup time series' entries

1) Name:
The name of a rollup time series has this format: <name of raw time series>@<name of time series policy>
It is a combination of the name of the raw time series and the name of the time series policy separated by @.
In the image above these are "HeartRates" and "byHour" respectively.
For this reason, neither a time series name nor a policy name can have the character @ in it.

2) Timestamp:
The aggregation frame always begins at a round number of one of these time units: a second, minute, hour, day, week, month, or year. So the frame includes all entries starting at a round number of time units, and ending at a round number minus one millisecond (since milliseconds are the minimal resolution in RavenDB time series). The timestamp for a rollup entry is the beginning of the frame it represents.

For example, if the aggregation frame is three days, a frame will start and end at a time stamps like:
2020-01-01 00:00:00 - 2020-01-03 23:59:59.999.

3) Values:
Each group of six values represents one value from the original entries. If the raw time series has n values per entry, the rollup time series will have 6 * n per entry: the first six summarize the first raw value, the next six summarize the next raw value, and so on. The aggregated values have the names: "First (<name of raw value>)", "Last (<name of raw value>)", ... respectively.

Because time series entries are limited to 32 values, rollups are limited to the first five values of an original time series entry, or 30 aggregate values.

Create time series policies:

// Define a policy on the RAW time series data:
// ============================================
const rawPolicy = new RawTimeSeriesPolicy(TimeValue.ofYears(5)); // Retain data for five years

// Define a ROLLUP policy: 
// =======================
const rollupPolicy = new TimeSeriesPolicy(
    "By1WeekFor1Year",      // Name of policy
    TimeValue.ofDays(7),    // Aggregation time, roll-up the data for each week
    TimeValue.ofYears(5));  // Retention time, keep data for five years

// Define the time series configuration for collection "Companies" (use above policies):
// =====================================================================================
const collectionConfig = new TimeSeriesCollectionConfiguration();        
collectionConfig.rawPolicy = rawPolicy;
collectionConfig.policies = [rollupPolicy];

const timeSeriesConfig = new TimeSeriesConfiguration();
timeSeriesConfig.collections.set("Companies", collectionConfig);

// Deploy the time series configuration to the server
// by sending the 'ConfigureTimeSeriesOperation' operation:
// ========================================================
await documentStore.maintenance.send(new ConfigureTimeSeriesOperation(timeSeriesConfig));

// NOTE:
// The time series entries in the RavenDB sample data are dated up to the year 2020.
// To ensure that you see the rollup time series created when running this example,
// the retention time should be set to exceed that year.

Retrieve rollup data:

• Retrieving entries from a rollup time series is similar to getting the raw time series data.

• Learn more about using timeSeriesFor.get in Get time series entries.

// Get all data from the RAW time series:
// ======================================

const rawData = await session
    .timeSeriesFor("companies/91-A", "StockPrices")
    .get();

// Get all data from the ROLLUP time series:
// =========================================

// Either - pass the rollup name explicitly to 'TimeSeriesFor':
let rollupData = await session
    .timeSeriesFor("companies/91-A", "StockPrices@By1WeekFor1Year")
    .get();

// Or - get the rollup name by calling 'GetTimeSeriesName':
rollupData = await session
    .timeSeriesFor("companies/91-A", rollupPolicy.getTimeSeriesName("StockPrices"))
    .get();

// The raw time series has 100 entries
assert.equal(rawData.length, 100);
assert.equal(rawData[0].isRollup, false);

// The rollup time series has only 22 entries
// as each entry aggregates 1 week's data from the raw time series
assert.equal(rollupData.length, 22);
assert.equal(rollupData[0].isRollup, true);

The time series policies

• Raw policy:

  • Used to define the retention time of the raw time series.
  • Only one such policy per collection can be defined.
  • Does not perform aggregation.

• Rollup policy:

  • Used to define the aggregation time frame and retention time for the rollup time series.
  • Multiple policies can be defined per collection.

class RawTimeSeriesPolicy extends TimeSeriesPolicy {
    retentionTime;  // TimeValue
}

class TimeSeriesPolicy {
    name;           // string;
    retentionTime   // TimeValue
    aggregationTime // TimeValue
}

class TimeValue {
    static ofSeconds(seconds);
    static ofMinutes(minutes);
    static ofHours(hours);
    static ofDays(days); 
    static ofMonths(months);
    static ofYears(years);
}

The time series configuration object

class TimeSeriesConfiguration {
    collections; // Map<string, TimeSeriesCollectionConfiguration>
}

class TimeSeriesCollectionConfiguration {
    disabled;  // boolean
    policies;  // TimeSeriesPolicy[]
    rawPolicy; // RawTimeSeriesPolicy
}

The time series configuration operation

ConfigureTimeSeriesOperation(configuration);

Learn more about operations in: What are operations.