RavenDB Features

Features

• A fluent and predictable hand-tailored API
• Full LINQ to RQL support
• Safe by default
• Extensible and configurable with the ability to inject behavior directly from server
• DDD compatible
• Unit of Work

                // 'DocumentStore' is a main-entry point for client API.
// It is responsible for managing and establishing connections
// between your application and RavenDB server/cluster
// and is capable of working with multiple databases at once.
// Due to it's nature, it is recommended to have only one
// singleton instance per application
public static class DocumentStoreHolder
{
    private static readonly Lazy<IDocumentStore> LazyStore =
        new Lazy<IDocumentStore>(() =>
        {
            var store = new DocumentStore
            {
                Urls = new[] { "http://localhost:8080" },
                Database = "Northwind"
            };

            return store.Initialize();
        });

    public static IDocumentStore Store => LazyStore.Value;
}

            

                // 'DocumentSession' or 'Session' is a short-living and lightweight object
// used as a main and recommended interaction point with the database.
// With 'Session' you are able to load documents, modify them, save them back
// issue queries, patches, and much more.
// It implements Unit Of Work concept and is batching requests
// to reduce the expensive network traffic
using (IDocumentSession session = DocumentStoreHolder.Store.OpenSession())
{
    // code here
}

            

                var category = new Category
{
    Name = "Electronics"
};

// Store the category in 'Session'
// and automatically assign Id using HiLo algorithm
session.Store(category); 

var product = new Product
{
    Name = "Laptop 2000",
    Category = category.Id      // use the previously assigned Id
};

// Store the product in 'Session'
// and automatically assign Id using HiLo algorithm
session.Store(product);

// Synchronize changes with the server.
// All changes will be send in one batch
// that will be processed as _one_ ACID transaction
session.SaveChanges();

            

                // Load the 'Employee' and start tracking its changes
Employee employee = session.Load<Employee>("employees/1-A");

// Apply modifications
employee.FirstName = "Juliette";

// Synchronize changes with the server.
// All changes will be send in one batch
// that will be processed as _one_ ACID transaction
session.SaveChanges();

            

                // return all entities from 'Employees' collection
// where FirstName equals 'Robert'
List<Employee> employees = session
    .Query<Employee>()
    .Where(x => x.FirstName == "Robert")
    .ToList();

// return all 'LastNames' from 'Employees' collection
// that were born after '1960-01-01'
List<string> lastNames = session
    .Query<Employee>()
    .Where(x => x.Birthday > new DateTime(1960, 1, 1))
    .Select(x => x.LastName)
    .ToList();

            

Features

• A fluent and predictable hand-tailored API
• Safe by default
• Extensible and configurable
• DDD compatible
• Unit of Work

                // 'DocumentStore' is a main-entry point for client API.
// It is responsible for managing and establishing connections
// between your application and RavenDB server/cluster
// and is capable of working with multiple databases at once.
// Due to it's nature, it is recommended to have only one
// singleton instance per application
public class DocumentStoreHolder {

    private static class DocumentStoreContainer {
        public static final IDocumentStore store =
            new DocumentStore("http://localhost:8080", "Northwind");

        static {
            store.initialize();
        }
    }

    public static IDocumentStore getStore() {
        return DocumentStoreContainer.store;
    }
}

            

                // 'DocumentSession' or 'Session' is a short-living and lightweight object
// used as a main and recommended interaction point with the database.
// With 'Session' you are able to load documents, modify them, save them back
// issue queries, patches, and much more.
// It implements Unit Of Work concept and is batching requests
// to reduce the expensive network traffic
try (IDocumentSession session = DocumentStoreHolder.getStore().openSession()) {
{
    // code here
}

            

                Category category = new Category();
category.setName("Electronics");

// Store the category in 'Session'
// and automatically assign Id using HiLo algorithm
session.store(category);

Product product = new Product();
product.setName("Laptop 2000");
product.setCategory(category.getId()); // use the previously assigned Id

// Store the product in 'Session'
// and automatically assign Id using HiLo algorithm
session.store(product);

// Synchronize changes with the server.
// All changes will be send in one batch
// that will be processed as _one_ ACID transaction
session.saveChanges();

            

                // Load the 'Employee' and start tracking its changes
Employee employee = session.load(Employee.class, "employees/1-A");

// Apply modifications
employee.setFirstName("Juliette");

// Synchronize changes with the server.
// All changes will be send in one batch
// that will be processed as _one_ ACID transaction
session.saveChanges();

            

                // return all entities from 'Employees' collection
// where firstName equals 'Robert'
List<Employee> employees = session.query(Employee.class)
                .whereEquals("firstName", "Robert")
                .toList();

// return all 'lastNames' from 'Employees' collection
// that are over 18
List<String> lastNames = session.query(Employee.class)
                .whereGreaterThanOrEqual("age", 18)
                .selectFields(String.class, "lastName")
                .toList();

            

Features

• A fluent and predictable hand-tailored API
• Safe by default
• Extensible and configurable
• DDD compatible
• Unit of Work

                // 'DocumentStore' is a main-entry point for client API.
// It is responsible for managing and establishing connections
// between your application and RavenDB server/cluster
// and is capable of working with multiple databases at once.
// Due to it's nature, it is recommended to have only one
// singleton instance per application
import { DocumentStore } from "ravendb";

const documentStore = new DocumentStore("http://localhost:8080", "Northwind");
documentStore.initialize();

            

                // 'DocumentSession' or 'Session' is a short-living and lightweight object
// used as a main and recommended interaction point with the database.
// With 'Session' you are able to load documents, modify them, save them back
// issue queries, patches, and much more.
// It implements Unit Of Work concept and is batching requests
// to reduce the expensive network traffic
const session = documentStore.openSession();

// code here

            

                const category = new Category();
category.name = "Electronics";

// Store the category in 'Session'
// and automatically assign Id using HiLo algorithm
await session.store(category);

const product = new Product();
product.name = "Laptop 2000";
product.category = category.id; // use the previously assigned Id

// Store the product in 'Session'
// and automatically assign Id using HiLo algorithm
await session.store(product);

// Synchronize changes with the server.
// All changes will be send in one batch
// that will be processed as _one_ ACID transaction
await session.saveChanges();

            

                // Load the 'Employee' and start tracking its changes
const employee = await session.load("employees/1-A");

// Apply modifications
employee.firstName = "Juliette";

// Synchronize changes with the server.
// All changes will be send in one batch
// that will be processed as _one_ ACID transaction
await session.saveChanges();

            

                // return all entities from 'Employees' collection
// where firstName equals 'Robert'
const employees = await session.query({ collection: "Employees" })
                .whereEquals("firstName", "Robert")
                .all();

// return all 'lastNames' from 'Employees' collection
// that are over 18
const lastNames = await session.query({ collection: "Employees" })
                .whereGreaterThanOrEqual("age", 18)
                .selectFields("lastName")
                .all();

            

Countless systems, including an ever-growing number of IoT (Internet of Things) devices, produce streams of values that can show the behavior and development of a process over time. The barometer measurements produced by a weather station, the coordinates produced by a delivery truck’s GPS, share prices reported by a stock exchange, and the heartrate of a runner, reported by a wearable pulse tracker, are values that can be collected and put to very good use. A time-series is a sequence of data points in which collected values are ordered by time, to ease their management and usage.

Cross-System Support

RavenDB 5.0’s time-series support includes a highly-efficient time-series storage, a comfortable and thorough set of API methods, the support of core features like indexing and querying and of dedicated time-series features like rollup and retention, and the full support of a management Studio that lets you conveniently watch and manage time-series, play with them, query and plot them in graphs.

Features:

• Indexing and Querying
  Time-series are indexed to speed up their performance, and can be queried by their indexes. Queries can be performed using LINQ expressions, over raw RQL, and using the Studio.
• Rollup and Retention
  You can set an automatic rollup and retention policy to regularly create custom time-series from your raw data and delete the data you don’t need anymore.
  Rollup is the process of aggregating a time-series’ data to create a smaller dataset that can be used and queried. You can take a 100,000-entries heartrate time-series for example, and aggregate it by the maximal heartrate taken every 10 minutes.
  Retention is the process of removing outdated time-series entries to keep storage usage at a minimum and management light.
• Including time-series data
  This feature shows the efficiency gained by adding time-series functionality to the document model. A document’s time-series data can be included while loading the document, retrieved into the client’s cache, and provided to the user the instant it is needed.
• ETL
  Time Series data can be queried and transformed by an ETL process, and even added to new documents.

Time Series in the Document Model

RavenDB is a distributed document database, and its time-series management reflects it and is empowered by it.

• A Time-Series is a part of a document’s data
  a time-series is created and handled as an integral part of a specific document, to provide it with a clear context and with the same simple, solid management provided to other data types. Time-series data can be backed up, for example, like the rest of a document’s data. Since time-series are expected to grow large and be updated often, their data is kept separately from their documents’ data and can change rapidly without invoking document-change events and degrading performance.
• Concurrent updates
  Different cluster nodes and different clients of the same node can update the same time-series simultaneously without conflict.

                // Open a session
using (var session = store.OpenSession())
{
    // Use the session to create a document
    session.Store(new { Name = "John" }, "users/john");

    // Append a Heartrate entry at the first-minute timestamp
    session.TimeSeriesFor("users/john", "Hearthrate")
        .Append(baseline.AddMinutes(1), 70d, "Watches/fibit");
    session.SaveChanges();
}

            

                // Retrieve all entries of the "Hearthatea" time-series
using (var session = store.OpenSession())
{
    IEnumerable<TimeSeriesEntry> val = session.TimeSeriesFor("users/john", "Heartrate")
        .Get(DateTime.MinValue, DateTime.MaxValue);
}

            

Time-series are segmented, to speed up load and query time by handling only the segments that contain relevant data.

A time-series entry is composed of:

• A UTC date & time Timestamp, with 1ms precision.
• An optional Tag, that can provide information about the values’ source and point to a document that provides additional information.
• Up to 32 Values. Some entries can become meaningful only when populated with multiple values, like in the case of a GPS tracking device whose measurements include both a latitude and a longitude.

Our RQL (Raven Query Language) allows you to execute all available types of queries and is a part of our JavaScript patching API. Based on SQL, it can be easily assimilated with few additions of our own.

                from Orders                                             // select
where Lines.Count > 4                                   // filter
select Lines[].ProductName as ProductNames,             // project
OrderedAt, ShipTo.City                                  

            

                from Orders as o                                        // select
load o.Company as c                                     // load related data
select {                                                // project using JavaScript
    Name: c.Name.toLowerCase(),                         
    Country: c.Address.Country,
    LinesCount: o.Lines.length
}

            

                from Orders                                             // select
group by Company                                        // group
where count() > 5                                       // filter
order by count() desc                                   // order
select count() as Count, key() as Company               // project
include Company                                         // side-load related data

            

Perform advanced full-text search operations over one or more fields at once with the ability to supply your own custom analyzer.

                       // return all of the companies
// which one of the words in 'Name' is 'store'
from Companies
where search(Name, 'store')

                   

                       // return all of the companies
// which one of the words in 'Name' is 'store'
var companies = session
    .Query<Company>
    .Search(x => x.Name, "store")
    .ToList();

                   

                       // return all of the companies
// which one of the words in 'name' is 'store'
List<Company> companies = session
                .query(Company.class)
                .search("name", "store")
                .toList();

                   

                       // return all of the companies
// which one of the words in 'name' is 'store'
const companies = await session
    .query({ collection: "companies" })
    .search("name", "store")
    .all();

                   

Perform calculations such as counts, sums, averages, min and max on aggregated data and split it into defined ranges without a fuss.

In order to execute faceted search a static index is required. It can be as simple as follows:

                private class Cameras_ByManufacturerModelCostDateOfListingAndMegapixels :
    AbstractIndexCreationTask<Camera>
{
    public Cameras_ByManufacturerModelCostDateOfListingAndMegapixels()
    {
        Map = cameras => from camera in cameras
            select new
            {
                camera.Manufacturer,
                camera.Model,
                camera.Cost,
                camera.DateOfListing,
                camera.Megapixels
            };
    }
}

            

Those are all the requirements for faceted search.
Now queries can be executed against all index fields as follows:

                       // return 3 facet results for cameras 
// using 'Cameras/ByManufacturerModelCostDateOfListingAndMegapixels' index
// - first result will return aggregations (Count) based on 'Manufacturer'
// - second result will return aggregations (Count, Sum) based on 'Cost' 
//   and divided into supplied 'Cost' ranges
// - third result will return aggregations (Count, Min, Max, Average) based on 'Cost'
//   and divided into supplied 'Megapixels' ranges
from index 'Cameras/ByManufacturerModelCostDateOfListingAndMegapixels' 
where Cost between 100 and 300
select 	facet(Manufacturer), 
		facet(Cost < 200, Cost >= 200 and Cost < 400, Cost >= 400 and Cost < 600, Cost >= 600 and Cost < 800, Cost >= 800, sum(Cost)), 
		facet(Megapixels < 3, Megapixels >= 3 and Megapixels < 7, Megapixels >= 7 and Megapixels < 10, Megapixels >= 10, min(Cost), max(Cost), average(Cost))

                   

                       // return 3 facet results for cameras 
// using 'Cameras/ByManufacturerModelCostDateOfListingAndMegapixels' index
// - first result will return aggregations (Count) based on 'Manufacturer'
// - second result will return aggregations (Count, Sum) based on 'Cost' 
//   and divided into supplied 'Cost' ranges
// - third result will return aggregations (Count, Min, Max, Average) based on 'Cost'
//   and divided into supplied 'Megapixels' ranges
Dictionary<string, FacetResult> facetResults = session
    .Query<Camera, Cameras_ByManufacturerModelCostDateOfListingAndMegapixels>()
    .Where(x => x.Cost >= 100 && x.Cost <= 300)
    .AggregateBy(builder => builder.ByField(x => x.Manufacturer))
    .AndAggregateBy(builder => builder
        .ByRanges(
            camera => camera.Cost < 200m,
            camera => camera.Cost >= 200m && camera.Cost < 400m,
            camera => camera.Cost >= 400m && camera.Cost < 600m,
            camera => camera.Cost >= 600m && camera.Cost < 800m,
            camera => camera.Cost >= 800m)
        .SumOn(x => x.Cost))
    .AndAggregateBy(builder => builder
        .ByRanges(
            camera => camera.Megapixels < 3.0,
            camera => camera.Megapixels >= 3.0 && camera.Megapixels < 7.0,
            camera => camera.Megapixels >= 7.0 && camera.Megapixels < 10.0,
            camera => camera.Megapixels >= 10.0)
        .MinOn(x => x.Cost)
        .MaxOn(x => x.Cost)
        .AverageOn(x => x.Cost))
    .Execute();

                   

                       // return 3 facet results for cameras 
// using 'Cameras/ByManufacturerModelCostDateOfListingAndMegapixels' index
// - first result will return aggregations (Count) based on 'Manufacturer'
// - second result will return aggregations (Count, Sum) based on 'Cost' 
//   and divided into supplied 'Cost' ranges
// - third result will return aggregations (Count, Min, Max, Average) based on 'Cost'
//   and divided into supplied 'Megapixels' ranges
Dictionary<string, FacetResult> facetResults = session
    .Query<Camera, Cameras_ByManufacturerModelCostDateOfListingAndMegapixels>()
    .Where(x => x.Cost >= 100 && x.Cost <= 300)
    .AggregateBy(builder => builder.ByField(x => x.Manufacturer))
    .AndAggregateBy(builder => builder
        .ByRanges(
            camera => camera.Cost < 200m,
            camera => camera.Cost >= 200m && camera.Cost < 400m,
            camera => camera.Cost >= 400m && camera.Cost < 600m,
            camera => camera.Cost >= 600m && camera.Cost < 800m,
            camera => camera.Cost >= 800m)
        .SumOn(x => x.Cost))
    .AndAggregateBy(builder => builder
        .ByRanges(
            camera => camera.Megapixels < 3.0,
            camera => camera.Megapixels >= 3.0 && camera.Megapixels < 7.0,
            camera => camera.Megapixels >= 7.0 && camera.Megapixels < 10.0,
            camera => camera.Megapixels >= 10.0)
        .MinOn(x => x.Cost)
        .MaxOn(x => x.Cost)
        .AverageOn(x => x.Cost))
    .Execute();

                   

Finding similar documents has never been easier! Use our built-in MoreLikeThis feature to retrieve them.

Let’s start by defining an index, where we will index contents of the articles (ArticleBody) using ‘StandardAnalyzer’ from Lucene and storing the results inside the Lucene index.

                public class Articles_ByArticleBody : AbstractIndexCreationTask<Article>
{
    public Articles_ByArticleBody()
    {
        Map = docs => from doc in docs
                      select new
                      {
                          doc.ArticleBody
                      };

        Stores.Add(x => x.ArticleBody, FieldStorage.Yes);
        Analyzers.Add(x => x.ArticleBody, "StandardAnalyzer");
    }
}

            

At this point everything needed to use MoreLikeThis is done and we can search for our similar articles:

                       // return all articles
// similar to 'articles/1'
// using 'Articles/ByArticleBody' index
from index 'Articles/ByArticleBody' 
where morelikethis(id() = 'articles/1')

                   

                       // return all articles
// similar to 'articles/1'
// using 'Articles/ByArticleBody' index
List<Article> articles = session
    .Query<Article, Articles_ByArticleBody>()
    .MoreLikeThis(builder => builder
        .UsingDocument(x => x.Id == "articles/1"))
    .ToList();

                   

Spatial field can be created from Coordinates (latitude and longitude), or directly from WKT shape using following strategies:

• with BoundingBox or QuadPrefixTree
• Geographical with BoundingBox, GeohashPrefixTree (default) or QuadPrefixTree

RQL itself and our Client API contain built-in methods to help you find the documents with the search areas that can be defined using:

• within (and within radius)
• contains
• disjoin
• intersect
• custom relation to WKT shape

                       // return all the companies 
// within radius of 300km 
// from 45 latitude and 4 longitude coordinate
from Companies
where spatial.within(
    spatial.point(Address.Location.Latitude, Address.Location.Longitude), 
    spatial.circle(300, 45, 4))

                   

                       // return all the companies 
// within radius of 300km 
// from 45 latitude and 4 longitude coordinate
List<Company> companies = session
    .Query<Company>()
    .Spatial(
        factory => factory
            .Point(    
                x => x.Address.Location.Latitude,
                x => x.Address.Location.Longitude),
        criteria => criteria.WithinRadius(300, 45, 4))
    .ToList();

                   

                       // return all the companies 
// within radius of 300km 
// from 45 latitude and 4 longitude coordinate
List<Company> companies = session
                .query(Company.class)
                .spatial(
                        new PointField("address.location.latitude", "address.location.longitude"),
                        factory -> factory.withinRadius(300, 45, 4))
                .toList();

                   

                       // return all the companies 
// within radius of 300km 
// from 45 latitude and 4 longitude coordinate
const companies = await session
    .query({ collection: "companies" })
    .spatial(
        new PointField("address.location.latitude", "address.location.longitude"),
        factory => factory.withinRadius(300, 45, 4))
    .all();

                   

Update documents directly on the server-side using our integrated JavaScript-based patching API, a part of RQL syntax.

                from Orders 
update { 
    this.Lines = this.Lines.filter(l => l.Product != 'products/1-A');
}

            

                from Orders as o
load o.Company as c
update { 
    o.CompanyName = c.Name;
}

            

                from index 'Orders/Totals' as i
where i.Total > 10000
load i.Company as c
update { 
    i.LowerName = c.Name.toLowerCase();
}

            

Auto Indexes are designed when there is no need to customize the capabilities extensively.

With Auto Indexes you will be able to perform queries, including advanced ones like full-text search.

RavenDB supports following auto index types:

• Auto Map indexes
• Auto Map-Reduce indexes

RavenDB learns the behavior of your application to merge auto indexes to reduce your overhead. It will schedule unused auto indexes for deletion.

With Static Indexes, your indexing capabilities are almost endless. Use C# LINQ or JavaScript functions to shape the indexation to address your needs by defining single or multiple mapping functions and reduce (aggregate) the results if necessary.

RavenDB supports the following index types:

• Map indexes
• Multi-Map indexes
• Map-Reduce indexes

Each Static Index allows you to configure the behavior of each indexed field with the following options:

• Store the data directly in index, allowing you to fetch data directly from it without the need to fetch the document from disk
• Turn on a full-text search with an additional option to use your own custom analyzer
• Turn on suggestions
• Index document extensions as well as their contents – including attachments, time series, and counters.
• Many more

You can override server or database settings regarding indexing on a per index basis or add your own custom C# or JavaScript code to be used during indexing.

Attachment content and metadata can now be indexed using static indexes.

An attachment is a kind of data that is associated with some document, but either it can’t be expressed as JSON, or we’d prefer to load and modify it separately from the document itself. Examples of attachments might be images, audio, or just pure binary data.

The attachment content can be loaded to the index as a Stream or as a string. Besides that, the following metadata can be accessed: the attachment name, its size, its hash, and its content type.

The new Additional Assemblies feature allows you to import libraries from NuGet and other sources into your indexes so that they can be used within the index logic itself. This makes it possible to integrate a variety of existing technologies such as:

• Machine Learning (ML) – see this blog post by Oren Eini for a step-by-step example of using image recognition to index pictures with descriptive tags.
• Optical Character Recognition (OCR) Image Processing – scan flat images to see if text can be extracted.
• File-type Conversion – see this blog post for an example of how to scan Word and Excel files (.docx and .xlsx) and extract text which can then be indexed.

Besides NuGet, Additional Assemblies also allows you to import libraries from runtime or from a local folder.

As a rule, RavenDB gives you as much valuable debugging information as possible.

We created an indexing performance graph, to see indexing internals, particular steps in the process, their timings, and the amount of input and output documents. RavenDB makes indexing as transparent as possible to help you make the best analysis and take the optimal next steps.

Map-Reduce visualizer will take you into deepest abyss of the map-reduction itself. You can understand what mapping results your documents are yielding and follow reduction steps that are taking place before you reach the final result.

Indexing text requires an analyzer that splits the text into tokens which can then be searched.

For example, some analyzers tokenize words separated by whitespace characters, while the NGram analyzer divides the text into tokens of the same number of characters.

RavenDB comes with several Lucene analyzers, and you can also add your own custom analyzers. Version 5.2 has a new interface in the Studio for creating custom analyzers and using them in indexes.

RavenDB’s fully automated indexing is one of the keys to its extraordinary performance, allowing the acceleration of data retrieval by executing queries over pre-prepared indexes rather than ever-repeating cumbersome full scans.

But heavy-duty indexing operations may become a burden for the cluster if all nodes are busy indexing at the same time.

• On Site, the toll concurrent indexing takes on nodes’ storage and CPU usage may decrease the cluster’s performance.
• On the Cloud, concurrent indexing may exhaust all nodes’ credits at the same time and leave too few or even none of the nodes available for service.

Starting with RavenDB 5.2 you can use Rolling Index Deployment to deploy indexing gradually, one node at a time. While one of the nodes surrenders its resources to indexing, the cluster may reassign its tasks to other nodes if needed.

Keeping indexing as efficient as ever, rolling index deployment leaves the cluster at its peak performance and availability at all times.

Distributed Counters are numeric data variables that can be added to your documents. They are designed for scenarios like:

• High-throughput counting e.g. vote counting or any type of survey
• Continuously record the number of visitors on an event page
• Avoid having to update your entire document for a single numeric value change
• Keep track of the number of times your document has been viewed or rated

                using (var session = store.OpenSession())
{
    // increment counter 'likes' of document 'users/1-A' by 10

    session.CountersFor("users/1-A").Increment("likes", 10);
    session.SaveChanges();
}

            

Track the history of changes for a document. Compare each revision, or revert to a specific one.

You can revise your entire database to a specific point in time in a single click. You can also revise a specific collection or collections.

Perform persistent live queries with high availability in mind.
In order to use a DataSubscription, it should be defined first:

                store.Subscriptions.Create(new Raven.Client.Documents.Subscriptions.SubscriptionCreationOptions
                {
                    Name = "Heavy Orders Tracking",
                    Query = @"
                            from Orders as o
                            where o.Freight > 50
                            select 
                            {
                                Freight: o.Freight,
                                ShipTo: o.ShipTo,
                                Employee: o.Employee,
                                Products: o.Lines.map(x=>x.Product)                                
                            }"

                });

            

The subscription definition is being sent to the server and stored in the cluster.

The SubscriptionWorker allows processing data on demand, and continue from the point it last stopped:
The worker will continue running until stopped by user or processing error.

                var workerTask = worker.Run(async x => {
                        foreach (var orderSummary in x.Items)
                        {
                            await NotifyLogisticsDepartmentAboutHeavyShipment(orderSummary.Result);
                        }
                    });

            

With the massive amounts of information that modern applications are expected to handle, it’s important to be able to compress data into smaller sizes to save storage space. In most databases, this presents a tradeoff – the more efficiently data is compressed, the slower it becomes to store, retrieve, or modify it. RavenDB 5.0 introduces a new way of compressing documents more efficiently, without sacrificing the high performance and speed you’ve come to expect.

In many use cases, the data stored in RavenDB consists of many documents with similar structures or other commonalities. A lot of space could be saved by compressing collections of similar documents as a single unit. But this would mean that to load a single document, the entire collection would have to be decompressed. To add, delete, or change a single document, the entire collection would have to be compressed again in a different way. Past versions of RavenDB offered ways of compressing the data within documents, but we found that compressing multiple documents simply wasn’t worth the performance cost.

In RavenDB 5.0 we have introduced a solution to this dilemma by integrating the Zstd compression algorithm (Zstandard) first developed for Facebook. This algorithm is able to ‘train’ on a batch of documents and learn the commonalities between them, and then compress new documents individually, which makes it possible to decompress them individually as well.

This new Documents Compression feature can be toggled for each document collection, as well as document revisions. By training on the first few documents in the collection, the Zstd algorithm generates a dictionary that it then applies to every new document. The algorithm monitors the compression ratio achieved for each document, and if it is not satisfactory, the algorithm goes back and re-trains on the most recently modified documents. After this, if the algorithm is able to compress the new document more efficiently, the dictionary is updated. In this way, the algorithm continuously adapts to the data you feed it.

Using this new feature, we often see compression ratios of over 50%, with negligible costs to performance. In fact, because of the reduced I/O usage, we often see an overall increase in the speed the system is able to process and handle operations.

As your database grows, you will inevitably reach a point in time when a dataset you operate on exceeds the capacity of a single machine. Database Sharding is one possible solution in this situation – splitting the database into several pieces and placing each one of those pieces on a separate machine.

There are many subtle but important details you need to consider, even though this sounds like an easy solution to an unpleasant challenge.First, you are introducing change at the lowest level of your application, and now you may need to propagate these changes all the way up to the Data Layer of your application; this often requires a complete redesign of the data access code.

Second, careful planning is needed – not all workloads are created equal, so you must carefully analyze and plan the sizing and positioning of the shards you will create. You do not want to overwhelm some of the servers while having others sitting idle most of the time.

Finally, your new sharded setup must be not only an appropriate setup to support further growth in data volume and traffic – it also must not get harder to operate over time. With so many fine details and every project being unique, it is easy to make mistakes that would turn out to be hugely expensive in the future.

When designing and implementing Sharding in RavenDB v6, we considered all these challenges. As a result, RavenDB can scale infinitely without sacrificing performance. Furthermore, we concentrated on the challenging part – developer experience and operational ease of use.

Transitioning from your existing setup to a sharded one is as seamless as it can be – once you create a sharded database, aside from the fact that your data is split across multiple machines, you will notice no change in behavior. You can move your existing RavenDB-based system to a sharded environment without a single change. We made an extreme effort to keep Sharding implementation self-contained, preventing leakage of any implementation details into your data access layer.

With RavenDB clustering is easy!

In a matter of minutes, you can configure it without any expert knowledge using our GUI management studio.

The RavenDB clustering is build on top of two layers:

• First, the cluster layer is managed by a consensus protocol called Raft. In CAP theorem it is CP (consistent and partition tolerant).
• The second layer, the database layer, is AP (it is always available, even if there is a partition, and it’s eventually consistent) and is handled by a gossip protocol between the databases on different nodes, forming multi-master mesh and replicating data between each other.

RavenDB utilizes the different layers for different purposes. At the cluster layer, the consensus protocol ensures that operators have the peace of mind of knowing that their commands are accepted and followed. At the database layer you know that RavenDB will never lose writes and will always keep your data safe.

Scheduling a backup, an ETL job, or any other ongoing process on just one node is not acceptable. If that node goes down for any reason, the show must go on. That’s is why RavenDB introduced cluster-wide tasks, assigned to a node, but owned by the cluster.

In the case of a failure of the assigned node, the RavenDB cluster will re-assign the work to another node, ensuring continuity of operations.

The following tasks can be created:

1. External Replication
2. RavenDB ETL and SQL ETL
3. Backup
4. Subscription

With cluster-wide tasks or as we are calling them Ongoing Tasks, you have a guarantee that only one cluster node is processing that job. With the high availability in place, you have assurance that the task will automatically switch to a new node if the old one is down.

A cluster can assign a database to all the nodes in the cluster, or to just some of them. In a five node cluster, most databases will only reside on two or three of the nodes, since duplicating all information times five is usually excessive.

Instead, you’ll typically spread the databases with a replication factor of two or three on the various nodes. If a node goes down (hardware failure, for example), the cluster will note that and if it fails to come back up quickly enough, will take steps to ensure that the number of live active replicas of the database is maintained.

The cluster will do that by adding another node in the cluster for the database, resulting in another copy of the data and ensuring that the configured number of replicas is maintained. When the failed node is brought up again, the cluster will determine whatever to keep the data on the old node or to use the new topology. Your operations team doesn’t have to be on their toes at all times, RavenDB is constantly monitoring and acting on your behalf within the boundaries set by your administrators.

Running a production database cluster can be a daunting task. There are a lot of knobs to turn, especially when you have co-dependent settings and options.

Far from being a opaque system, RavenDB works very hard to externalize and make visible all the details about the system that your operations team needs, and is often deployed in a self managing option.

Features such as automatic failover, dynamic distribution of tasks and databases, multi master writes means that your operations team can sleep in peace, knowing that if anything happens, they can get to it in the morning.

RavenDB also continuously runs diagnostics and self checks. Each member of the cluster is verifying each other to ensure the health of the entire system. If anything troubling shows up, RavenDB will alert the administrator, and usually include the suggested fix.

Tested in production for the past decade, RavenDB is a mature product requiring minimal guidance on day to day issues.

Combine documents with Compare Exchange values in a single transaction to favor consistency over availability and ensure that changes are going to be applied in an identical manner across the cluster even in the presence of failures and network partitions.

No major code change is required, simply open the session with the TransactionMode set to ClusterWide and you have a cluster-wide transaction.

                using (var session = store.OpenSession(new SessionOptions
{
    TransactionMode = TransactionMode.ClusterWide
}))
{
    var user = new User
    {
        FirstName = "John",
        LastName = "Doe",
        Email "johndoe@ravendb.net"
    };

    // create cluster wide unique users per email.
    session.Store(user);
    session.Advanced.ClusterTransaction.CreateCompareExchangeValue(user.Email, user.Id);
    session.SaveChanges();
}

            

Do you need to have offsite replica of your data?
Have you decided to connect geo-distributed clusters together?
Need to run resource intensive calculations on a separated node?

External Replication is a master-slave replication which continuously push the data to destination node in asynchronous manner whenever the data changes on the source node.

Does your solution involves many instances requiring data synchronization with the main server?

Pull Replication will provide a secured master-slave replication initiated by the slave side to reduce the networking configuration fatigue needed.

For this purpose, in order to deploy slave nodes, only a single definition on the master server is required. Allowing you to connect numerous of nodes in matter of minutes without the additional overhead.

Want to replicate documents, but not an entire database?
Do you have sensitive data that you need to limit access to?

Replication now allows you to configure both hub servers and sink servers to send and accept only certain documents. In other words, you can control which databases have read and write access to which documents. These documents can be specified not just by their document ID, but also all document IDs that begin with some prefix.

Imagine your application stores information for a hospital, and it has a central database and many other databases with limited access. Filtered Replication allows you to create a database for Dr. Alice Smith that has read access to this document in the central database:
doctors/Smith_A

And all documents in the central database with IDs that have this prefix:
pharmacy/medicines/

In addition, Dr. Smith’s database might have write access for all documents with this prefix:
prescriptions/Smith_A/

This feature allows you to plan far in advance which data will be read, replicated, and modified by whom.

The safety of your data is always our top priority. We strive to make sure that anything your users share with you on the assumption of privacy will remain private. This does not only mean protecting your data on the disk drive (read more about this in Encryption feature), but also in transferring it securely over the wire. To achieve this, we introduced the ability to utilize the HTTPS with TLS 1.2 protocols using X.509 certificates which grants you enterprise-level security for your data during transfers over the network.

The certificates are also used to grant specific privileges to certificate holders allowing them to access only subsets of all databases on the server, or to execute operations that are allowed to one of the predefined roles (security clearances). All of this is configurable in a convenient way using our Management Studio GUI.

Client certificates can have different levels of access for different databases: Admin, Read/Write, and Read-Only. Admin is the highest level and grants access to everything. Read/Write grants access to most things that relate to a database, but not to those that relate to the server. The next section deals with the Read-Only level.

Our encryption techniques are among the best on the market.

RavenDB comes with built-in encryption support that utilizes the modern XChaCha20-Poly1305 algorithm from a well-known and battle-tested encryption library called libsodium giving you the best of both worlds: security and performance in one go.

When you open the Studio interface of a freshly installed RavenDB, your first view is that of the Server Dashboard, which conveniently gathers and presents the server’s configuration and live statistics. You can always return to this page to display, among other statistics, the server’s current CPU and memory usage, its indexing activities, and its traffic volume.

But RavenDB is a distributed database and is often deployed as a multi-node cluster. How then can you efficiently monitor all of its nodes at once and assess its operability as a whole? Our state-of-the-art solution to this puzzle is an adjustable Cluster Dashboard. You can not only display data related to all cluster nodes in this view but also fully personalize it.

Widgets containing cluster info and live statistics can be freely added, removed, relocated, and resized, so the dashboard would display just the information you want to see in just the way you want to see it.

Since each widget displays statistics for all nodes, you can easily compare the operability of your nodes in various aspects. The traffic and CPU widgets, for example, provide you with an instant comparison of the traffic volume and CPU usage of the different nodes and clarify whether you need to take steps like distributing your workload differently or upgrading a node.

When you have dozens of servers to maintain, you can hook-up one tool to monitor them all. RavenDB comes with built-in support for SNMPv2 and SNMPv3, and exposes over 50 unique OIDs for you to take advantage of.

RavenDB has its own plugin for Telegraf, a server agent that collects data and statistics from your database. The RavenDB plugin collects dozens of useful metrics about server behavior, clusters, indexing, and more. These include things like your server’s CPU usage, the progress of your indexes, or the chatter between the servers in your cluster.

In addition, we’ve created a Grafana template for RavenDB. Grafana is a dashboard that displays data from Telegraf as graphs in real-time, giving you insight into the inner workings of your database in a convenient visual format.

Monitoring and evaluating RavenDB’s performance is made surprisingly easy with its Studio’s statistics pages. These graphical views allow you to draw and examine events in as many details as you like, from a general outline of tasks behavior over a given time period to the specifics of a chosen incident.

• The Indexing Performance view lets you examine chosen indexing operations.
  This, for example, is a look at all indexing activities during a chosen period of time:

While the screenshot below captures a more detailed look at the specific index Orders/Totals during a part of the time period chosen in the previous example:

Each bar represents a certain aspect of the indexing, e.g. Indexing, Mapping or Storage, and hovering over a selected bar reveals additional information.

• The Ongoing Tasks Stats view offers a graphic display of task-related operations over a continuous timeline. You can choose the tasks you want to observe, scroll the timeline for a history of their activity, zoom in to magnify a specific incident, or zoom out for a broader overview.
  You can use the Stats View to monitor the conduct of Replication and ETL tasks, and starting with RavenDB 5.2, of Data Subscription tasks as well.

Let’s take a look at the monitoring of a data subscription task to see how helpful this view can be.

A data subscription task is an ongoing operation in which the cluster sends document batches to a client in an orderly manner and waits for the consumption of each batch before sending the next. Documents transmitted this way may be, for example, orders sent to an accountant-client, making an otherwise tedious operation extremely easy to manage.

The graphical representation of all communications between a subscription task and its clients provides you with a clear understanding of the task’s conduct at a glance and makes further inquiries much simpler. You can instantly spot activity peaks and lows on the timeline, magnify chosen time segments, and find whether a problem lies with the server or the client (e.g. by checking if the client had acknowledged batch reception before processing has halted).

RavenDB enables you to schedule full and/or incremental automatic backups, choose if those backups should be binary (slower creation, quicker restore) or JSON (quicker creation, slower restore), and choose at least one of the following destinations:

• Local disk drive
• Amazon AWS S3
• Amazon AWS Glacier
• Microsoft Azure
• FTP

RavenDB can read and write events from one or more Kafka Topics and store them in collections. Once you set up Sink in a low-code manner, Kafka Topics will be streamed directly into your database. Such synchronization is automated, resilient, and reliable. This way, you can provide permanent storage for Kafka Events or leverage a powerful map/reduce indexing mechanism to aggregate events and provide new insights.

Besides sending messages to other subsystems via RabbitMQ, your database can also consume them. With a low-code setup, RavenDB can dequeue messages from one or more RabbitMQ queues in a reliable and resilient way. These messages are transformed into documents and stored in a database. From there, you can use RavenDB features like Subscriptions and Indexing to react to these new documents.

ETL (Extract, Transform, Load) is a robust automatic process that reads data from a RavenDB database, changes or rearranges it using JavaScript, and stores it in another database. RavenDB runs ETL operations as Ongoing Tasks that continuously react to updates in selected data, process the data using a user-configurable transformation script, and deliver it to a chosen destination.

We offer multiple types of ETL processes: RavenDB ETL, SQL ETL, and OLAP ETL (provided starting with version 5.2).

RavenDB ETL allows you to load (write) selected data to a collection outside the database group. You can filter and modify the data using a transformation script, and gain for example:

• The construction of whatever data format you want to create.
• Sharing of just a subset of your data for security or volume-reduction purposes.
• Data distribution to various destinations without requiring a continuous replication process.
• Aggregation of data sent from multiple sources.

The process includes three main stages:

• Extraction of chosen documents from your database.
  You can extract documents from a single collection or from multiple collections, and even all documents.
• Transformation of the extracted documents by user-defined JavaScript.
  A RavenDB ETL task can even use multiple transformation scripts. The extraction and transformation phases are performed in an orderly and highly efficient manner using batch processing.
• Loading (writing) the transformed data to your chosen destination.

A RavenDB ETL task can be defined from your code or using Studio.

You can use SQL ETL to extract selected documents from your database, freely transform them using JavaScript, prepare the data for storage in a relational format and deliver it to one of these popular databases:

• Microsoft SQL Server
• MySQL
• PostgreSQL
• Oracle

Being able to easily deliver documents to relational databases means that enterprises do not need to replace operative systems they grew to know and trust, nor to keep using them on the expanse of improved services.

An accounting or a reporting system, for example, that uses an SQL Server and functions perfectly well, can keep on using it while importing data from RavenDB via ETL. The company will preserve its familiar interface and operability, and still enjoy RavenDB’s power and be introduced with an abundance of new services.

You need to create the tables that would collect the transformed data in the destination database and define them for RavenDB so the SQL ETL task would be able to format the data appropriately.

An SQL ETL task can be defined from your code or using Studio.

OLAP (OnLine Analytical Processing) is a part of the ‘Business Intelligence’ practice of analyzing your data and system performance to find how they serve the needs of your enterprise, which frequently involves complex and heavy queries.

With OLAP ETL, RavenDB gives you tools to structure and store your data so the analysis you need to perform over it is as fast and efficient as possible. OLAP ETL translates your data to the Parquet format, a column based storage format ideal for Big Data analysis.

Your data can be delivered to destinations that are ideal for analytical purposes, including:

• An Amazon S3 “data bucket”, where it can be queried using Amazon Athena. This is very convenient because Athena outputs its results to another S3 bucket.
• An Azure Blob Storage, where it can be queried using Azure Data Lake.

Before adopting a new piece of technology you need to know that it will be a team player in your existing ecosystem. If you’re using Elasticsearch as a solution for full-text search, you can now add RavenDB 5.3 to the list of applications populating your organization’s Elasticsearch database. Through the powerful ETL (Extract, Transform, Load) mechanism, you can reshape RavenDB data before pushing it to Elasticsearch and avoid building time-expensive custom solutions.

From version 5.3, RavenDB is supporting integration with Power BI: a powerful business intelligence service by Microsoft. This enables you to push your data to Power BI, where you can analyze, process, transform and remodel it. Using this functionality you can create intelligent reports and gain new insights into your data and business. Additionally, you can now run queries from your BI client to fetch any data from RavenDB that you want to process, analyze and report on.

Kafka is one of the industry standards for systems integration. Its event stream is a backbone of distributed systems, providing a fast and reliable solution for building decoupled communication between heterogeneous components. RavenDB 5.4 introduces a powerful ETL that is able to reshape data before pushing it into Kafka’s topics.

Queues are a powerful mechanism for implementing high-capacity asynchronous processing. With RabbitMQ you can reliably store a high volume of commands arriving at your system and then process them within the capacity of your application. This way, you will be able to sustain traffic spikes without losing any of the received data. Starting with version 5.4, RavenDB can transform documents and publish them as messages directly to RabbitMQ for further processing.

From version 5.4, RavenDB is supporting integration with Grafana, a leading open-source solution for data visualization. Expose your database as a Grafana data source and create widgets and reports increasing the observability of your data.