Client API: How to Handle Document Relationships
One of the design principles that RavenDB adheres to is the idea that documents are independent, meaning all data required to process a document is stored within the document itself. However, this doesn't mean there should not be relations between objects.
There are valid scenarios where we need to define relationships between objects. By doing so, we expose ourselves to one major problem: whenever we load the containing entity, we are going to need to load data from the referenced entities as well (unless we are not interested in them). While the alternative of storing the whole entity in every object graph it is referenced in seems cheaper at first, this proves to be quite costly in terms of database resources and network traffic.
RavenDB offers three elegant approaches to solve this problem. Each scenario will need to use one or more of them. When applied correctly, they can drastically improve performance, reduce network bandwidth, and speed up development.
Denormalization
The easiest solution is to denormalize the data within the containing entity, forcing it to contain the actual value of the referenced entity in addition to (or instead of) the foreign key.
Take this JSON document for example:
// Order document with ID: orders/1-A
{
"Customer": {
"Name": "Itamar",
"Id": "customers/1-A"
},
Items: [
{
"Product": {
"Id": "products/1-A",
"Name": "Milk",
"Cost": 2.3
},
"Quantity": 3
}
]
}
As you can see, the Order
document now contains denormalized data from both the Customer
and the Product
documents which are saved elsewhere in full. Note we won't have copied all the customer properties into the order; instead we just clone the ones that we care about when displaying or processing an order. This approach is called denormalized reference.
The denormalization approach avoids many cross document lookups and results in only the necessary data being transmitted over the network, but it makes other scenarios more difficult. For example, consider the following entity structure as our start point:
public class Order
{
public string CustomerId { get; set; }
public string[] SupplierIds { get; set; }
public Referral Referral { get; set; }
public LineItem[] LineItems { get; set; }
public double TotalPrice { get; set; }
}
public class Customer
{
public string Id { get; set; }
public string Name { get; set; }
}
If we know that whenever we load an Order
from the database we will need to know the customer's name and address, we could decide to create a denormalized Order.Customer
field and store those details directly in the Order
object. Obviously, the password and other irrelevant details will not be denormalized:
public class DenormalizedCustomer
{
public string Id { get; set; }
public string Name { get; set; }
public string Address { get; set; }
}
There wouldn't be a direct reference between the Order
and the Customer
. Instead, Order
holds a DenormalizedCustomer
, which contains the interesting bits from Customer
that we need whenever we process Order
objects.
But what happens when the user's address is changed? We will have to perform an aggregate operation to update all orders this customer has made. What if the customer has a lot of orders or changes their address frequently? Keeping these details in sync could become very demanding on the server. What if another process that works with orders needs a different set of customer properties? The DenormalizedCustomer
will need to be expanded, possibly to the point that the majority of the customer record is cloned.
Tip
Denormalization is a viable solution for rarely changing data or for data that must remain the same despite the underlying referenced data changing over time.
Includes
The Includes feature addresses the limitations of denormalization. Instead of one object containing copies of the properties from another object, it is only necessary to hold a reference to the second object. This object can be a document, a counter, a time series, or a compare exchange value. Then the server can be instructed to pre-load the referenced object at the same time that the root object is retrieved. We do this using:
Order order = session
.Include<Order>(x => x.CustomerId)
.Load("orders/1-A");
// this will not require querying the server!
Customer customer = session
.Load<Customer>(order.CustomerId);
Above we are asking RavenDB to retrieve the Order
orders/1-A
, and at the same time "include" the Customer
referenced by the Order.CustomerId
property. The second call to Load()
is resolved completely client side (i.e. without a second request to the RavenDB server) because the relevant Customer
object has already been retrieved (this is the full Customer
object not a denormalized version).
There is also a possibility to load multiple documents:
Dictionary<string, Order> orders = session
.Include<Order>(x => x.CustomerId)
.Load("orders/1-A", "orders/2-A");
foreach (Order order in orders.Values)
{
// this will not require querying the server!
Customer customer = session.Load<Customer>(order.CustomerId);
}
You can also use Includes with queries:
IList<Order> orders = session
.Query<Order>()
.Include(o => o.CustomerId)
.Where(x => x.TotalPrice > 100)
.ToList();
foreach (Order order in orders)
{
// this will not require querying the server!
Customer customer = session
.Load<Customer>(order.CustomerId);
}
IList<Order> orders = session
.Query<Order>()
.Include(i => i
.IncludeDocuments<Customer>(x => x.CustomerId) //single document
.IncludeCounter("OrderUpdateCount")) //fluent builder can include counters as well
.Where(x => x.TotalPrice > 100)
.ToList();
foreach (Order order in orders)
{
// this will not require querying the server!
Customer customer = session
.Load<Customer>(order.CustomerId);
}
IList<Order> orders = session
.Advanced
.DocumentQuery<Order>()
.Include(x => x.CustomerId)
.WhereGreaterThan(x => x.TotalPrice, 100)
.ToList();
foreach (Order order in orders)
{
// this will not require querying the server!
Customer customer = session
.Load<Customer>(order.CustomerId);
}
from Orders
where TotalPrice > 100
include CustomerId
from Orders as o
where TotalPrice > 100
include CustomerId,counters(o,'OrderUpdateCount')
This works because RavenDB has two channels through which it can return information in response to a load request. The first is the Results channel, through which the root object retrieved by the Load()
method call is returned. The second is the Includes channel, through which any included documents are sent back to the client. Client side, those included documents are not returned from the Load()
method call, but they are added to the session unit of work, and subsequent requests to load them are served directly from the session cache, without requiring any additional queries to the server.
Note
Embedded and builder variants of Include clause are essentially syntax sugar and are equivalent at the server side.
{INFO Streaming query results does not support the includes feature. Learn more in How to Stream Query Results. /}
One to Many Includes
Include can be used with a many to one relationship. In the above classes, an Order
has a property SupplierIds
which contains an array of references to Supplier
documents. The following code will cause the suppliers to be pre-loaded:
Order order = session
.Include<Order>(x => x.SupplierIds)
.Load("orders/1-A");
foreach (string supplierId in order.SupplierIds)
{
// this will not require querying the server!
Supplier supplier = session.Load<Supplier>(supplierId);
}
Alternatively, it is possible to use the fluent builder syntax.
var order = session.Load<Order>(
"orders/1-A",
i => i.IncludeDocuments(x => x.SupplierIds));
foreach (string supplierId in order.SupplierIds)
{
// this will not require querying the server!
var supplier = session.Load<Supplier>(supplierId);
}
The calls to Load()
within the foreach
loop will not require a call to the server as the Supplier
objects will already be loaded into the session cache.
Multi-loads are also possible:
Dictionary<string, Order> orders = session
.Include<Order>(x => x.SupplierIds)
.Load("orders/1-A", "orders/2-A");
foreach (Order order in orders.Values)
{
foreach (string supplierId in order.SupplierIds)
{
// this will not require querying the server!
Supplier supplier = session.Load<Supplier>(supplierId);
}
}
Secondary Level Includes
An Include does not need to work only on the value of a top level property within a document. It can be used to load a value from a secondary level. In the classes above, the Order
contains a Referral
property which is of the type:
public class Referral
{
public string CustomerId { get; set; }
public double CommissionPercentage { get; set; }
}
This class contains an identifier for a Customer
. The following code will include the document referenced by that secondary level identifier:
Order order = session
.Include<Order>(x => x.Referral.CustomerId)
.Load("orders/1-A");
// this will not require querying the server!
Customer customer = session.Load<Customer>(order.Referral.CustomerId);
It is possible to execute the same code with the fluent builder syntax:
var order = session.Load<Order>(
"orders/1-A",
i => i.IncludeDocuments(x => x.Referral.CustomerId));
// this will not require querying the server!
Customer customer = session.Load<Customer>(order.Referral.CustomerId);
The alternative way is to provide a string-based path:
Order order = session.Include("Referral.CustomerId")
.Load<Order>("orders/1-A");
// this will not require querying the server!
Customer customer = session.Load<Customer>(order.Referral.CustomerId);
With the fluent builder syntax, it is also possible to use a string-based path:
var order = session.Load<Order>(
"orders/1-A",
i => i.IncludeDocuments("Referral.CustomerId"));
// this will not require querying the server!
Customer customer = session.Load<Customer>(order.Referral.CustomerId);
This secondary level include will also work with collections. The Order.LineItems
property holds a collection of LineItem
objects which each contain a reference to a Product
:
public class LineItem
{
public string ProductId { get; set; }
public string Name { get; set; }
public int Quantity { get; set; }
}
The Product
documents can be included using the following syntax:
Order order = session
.Include<Order>(x => x.LineItems.Select(l => l.ProductId))
.Load("orders/1-A");
foreach (LineItem lineItem in order.LineItems)
{
// this will not require querying the server!
Product product = session.Load<Product>(lineItem.ProductId);
}
The fluent builder syntax works here too.
var order = session.Load<Order>(
"orders/1-A",
i => i.IncludeDocuments(x => x.LineItems.Select(l => l.ProductId)));
foreach (LineItem lineItem in order.LineItems)
{
// this will not require querying the server!
Product product = session.Load<Product>(lineItem.ProductId);
}
when you want to load multiple documents.
The Select()
within the Include tells RavenDB which property of secondary level objects to use as a reference.
string
Path Conventions
When using string-based includes like:
Order order = session
.Include<Order>(x => x.Referral.CustomerId)
.Load("orders/1-A");
// this will not require querying the server!
Customer customer = session.Load<Customer>(order.Referral.CustomerId);
you must remember to follow certain rules that must apply to the provided string path:
- Dots are used to separate properties e.g.
"Referral.CustomerId"
in the example above means that ourOrder
contains propertyReferral
and that property contains another property calledCustomerId
. - Indexer operator is used to indicate that property is a collection type. So if our
Order
has a list of LineItems and eachLineItem
contains aProductId
property, then we can create string path as follows:"LineItems[].ProductId"
. - Prefixes can be used to indicate the prefix of the identifier of the document that is going to be included. It can be useful when working with custom or semantic identifiers. For example, if you have a customer stored under
customers/login@domain.com
then you can include it using"Referral.CustomerEmail(customers/)"
(customers/
is the prefix here).
Learning string path rules may be useful when you will want to query database using HTTP API.
curl -X GET "http://localhost:8080/databases/Northwind/docs?id=orders/1-A&include=Lines[].Product"
Dictionary Includes
Dictionary keys and values can also be used when doing includes. Consider following scenario:
public class Person
{
public string Id { get; set; }
public string Name { get; set; }
public Dictionary<string, string> Attributes { get; set; }
}
session.Store(
new Person
{
Id = "people/1-A",
Name = "John Doe",
Attributes = new Dictionary<string, string>
{
{ "Mother", "people/2" },
{ "Father", "people/3" }
}
});
session.Store(
new Person
{
Id = "people/2",
Name = "Helen Doe",
Attributes = new Dictionary<string, string>()
});
session.Store(
new Person
{
Id = "people/3",
Name = "George Doe",
Attributes = new Dictionary<string, string>()
});
Now we want to include all documents that are under dictionary values:
var person = session
.Include<Person>(x => x.Attributes.Values)
.Load("people/1-A");
var mother = session
.Load<Person>(person.Attributes["Mother"]);
var father = session
.Load<Person>(person.Attributes["Father"]);
Assert.Equal(1, session.Advanced.NumberOfRequests);
The code above can be also rewritten with fluent builder syntax:
var person = session.Load<Person>(
"people/1-A",
i => i.IncludeDocuments<Person>(x => x.Attributes.Values));
var mother = session
.Load<Person>(person.Attributes["Mother"]);
var father = session
.Load<Person>(person.Attributes["Father"]);
Assert.Equal(1, session.Advanced.NumberOfRequests);
You can also include values from dictionary keys:
var person = session
.Include<Person>(x => x.Attributes.Keys)
.Load("people/1-A");
Here, as well, this can be written with fluent builder syntax:
var person = session
.Load<Person>("people/1-A",
i => i.IncludeDocuments<Person>(x => x.Attributes.Keys));
Complex Types
If values in dictionary are more complex e.g.
public class PersonWithAttribute
{
public string Id { get; set; }
public string Name { get; set; }
public Dictionary<string, Attribute> Attributes { get; set; }
}
public class Attribute
{
public string Ref { get; set; }
}
session.Store(
new PersonWithAttribute
{
Id = "people/1-A",
Name = "John Doe",
Attributes = new Dictionary<string, Attribute>
{
{ "Mother", new Attribute { Ref = "people/2" } },
{ "Father", new Attribute { Ref = "people/3" } }
}
});
session.Store(
new Person
{
Id = "people/2",
Name = "Helen Doe",
Attributes = new Dictionary<string, string>()
});
session.Store(
new Person
{
Id = "people/3",
Name = "George Doe",
Attributes = new Dictionary<string, string>()
});
We can also do includes on specific properties:
var person = session
.Include<PersonWithAttribute>(x => x.Attributes.Values.Select(v => v.Ref))
.Load("people/1-A");
var mother = session
.Load<Person>(person.Attributes["Mother"].Ref);
var father = session
.Load<Person>(person.Attributes["Father"].Ref);
Assert.Equal(1, session.Advanced.NumberOfRequests);
Combining Approaches
It is possible to combine the above techniques. Using the DenormalizedCustomer
from above and creating an order that uses it:
public class Order3
{
public DenormalizedCustomer Customer { get; set; }
public string[] SupplierIds { get; set; }
public Referral Referral { get; set; }
public LineItem[] LineItems { get; set; }
public double TotalPrice { get; set; }
}
We have the advantages of a denormalization, a quick and simple load of an Order
, and the fairly static Customer
details that are required for most processing. But we also have the ability to easily and efficiently load the full Customer
object when necessary using:
Order3 order = session
.Include<Order3, Customer>(x => x.Customer.Id)
.Load("orders/1-A");
// this will not require querying the server!
Customer customer = session.Load<Customer>(order.Customer.Id);
This combining of denormalization and Includes could also be used with a list of denormalized objects.
It is possible to use Include on a query being a projection. Includes are evaluated after the projection has been evaluated. This opens up the possibility of implementing Tertiary Includes (i.e. retrieving documents that are referenced by documents that are referenced by the root document).
RavenDB can support Tertiary Includes, but before resorting to them you should re-evaluate your document model. Needing Tertiary Includes can be an indication that you are designing your documents along "Relational" lines.
Summary
There are no strict rules as to when to use which approach, but the general idea is to give it a lot of thought and consider the implications each approach has.
As an example, in an e-commerce application it might be better to denormalize product names and prices into an order line object since you want to make sure the customer sees the same price and product title in the order history. But the customer name and addresses should probably be references rather than denormalized into the order entity.
For most cases where denormalization is not an option, Includes are probably the answer.