Querying: Projections

When performing a query, we usually pull the full document back from the server.

However, we often need to display the data to the user. Instead of pulling the whole document back and picking just what we'll show, we can ask the server to send us just the details we want to show the user and thus reduce the amount of traffic on the network.

The savings can be very significant if we need to show just a bit of information on a large document.

A good example in the sample data set would be the order document. If we ask for all the Orders where Company is "companies/65-A", the size of the result that we get back from the server is 19KB.

However, if we perform the same query and ask to get back only the Employee and OrderedAt fields, the size of the result is only 5KB.

Aside from allowing you to pick only a portion of the data, projection functions give you the ability to rename some fields, load external documents, and perform transformations on the results.

Projections are Applied as the Last Stage in the Query

It is important to understand that projections are applied after the query has been processed, filtered, sorted, and paged. The projection doesn't apply to all the documents in the database, only to the results that are actually returned.
This reduces the load on the server significantly, since we can avoid doing work only to throw it immediately after. It also means that we cannot do any filtering work as part of the projection. You can filter what will be returned, but not which documents will be returned. That has already been determined earlier in the query pipeline.

The Cost of Running a Projection

Another consideration to take into account is the cost of running the projection. It is possible to make the projection query expensive to run. RavenDB has limits on the amount of time it will spend in evaluating the projection, and exceeding these (quite generous) limits will fail the query.

Projections and Stored Fields

If a projection function only requires fields that are stored, then the document will not be loaded from storage and all data will come from the index directly. This can increase query performance (by the cost of disk space used) in many situations when whole document is not needed. You can read more about field storing here.

Projections can be applied using the select_fields and select_fields_query_data methods.

The projection fields can be specified as a str array of field names,
and the projection type can be passed as a generic parameter.

fields = ["Name", "Phone"]
results = list(
    session.advanced.document_query_from_index_type(Companies_ByContact, Company).select_fields(
        ContactDetails, fields
    )
)

class Companies_ByContact(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = "companies.Select(x => new {name = x.Contact.Name, phone = x.Phone})"
        self._store_all_fields(FieldStorage.YES)  # Name and Phone fields can be retrieved directly from index

class ContactDetails:
    def __init__(self, name: str = None, phone: str = None):
        self.name = name
        self.phone = phone

from index 'Companies/ByContact'
select Name, Phone

The projection can also be defined by simply passing the projection type as a generic parameter.

results = list(
    session.advanced.document_query_from_index_type(Companies_ByContact, Company).select_fields(
        ContactDetails
    )
)

class Companies_ByContact(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = "companies.Select(x => new {name = x.Contact.Name, phone = x.Phone})"
        self._store_all_fields(FieldStorage.YES)  # Name and Phone fields can be retrieved directly from index

class ContactDetails:
    def __init__(self, name: str = None, phone: str = None):
        self.name = name
        self.phone = phone

from index 'Companies/ByContact'
select Name, Phone

Example I - Projecting Individual Fields of the Document

results = list(
    session.query_index_type(Employees_ByFirstAndLastName, Employee).select_fields(
        Employee, "FirstName", "LastName"
    )
)

class Employees_ByFirstAndLastName(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from employee in docs.Employees "
            "select new "
            "{"
            " FirstName = employee.FirstName,"
            " LastName = employee.LastName"
            "}"
        )

from index 'Employees/ByFirstAndLastName'
select FirstName, LastName

This will issue a query to a database, requesting only FirstName and LastName from all documents that index entries match query predicate from Employees/ByFirstAndLastName index. What does it mean? If an index entry matches our query predicate, then we will try to extract all requested fields from that particular entry. If all requested fields are available in there, then we do not download it from storage. The index Employees/ByFirstAndLastName used in the above query is not storing any fields, so the documents will be fetched from storage.

Example II - Projecting Stored Fields

If we create an index that stores FirstName and LastName and it requests only those fields in query, then the data will come from the index directly.

results = list(
    session.query_index_type(Employees_ByFirstAndLastNameWithStoredFields, Employee).select_fields(
        Employee, "FirstName", "LastName"
    )
)

class Employees_ByFirstAndLastNameWithStoredFields(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from employee in docs.Employees "
            "select new"
            "{"
            " FirstName = employee.FirstName,"
            " LastName = employee.LastName"
            "}"
        )
        self._store_all_fields(FieldStorage.YES)

from index 'Employees/ByFirstAndLastNameWithStoredFields'
select FirstName, LastName

Example III - Projecting Arrays and Objects

query_data = QueryData(["ShipTo", "Lines[].ProductName"], ["ShipTo", "Products"])
results = list(session.query(object_type=Order).select_fields_query_data(ShipToAndProducts, query_data))

class Orders_ByShipToAndLines(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = "from order in docs.Orders select new { ShipTo = order.ShipTo, Lines = order.Lines}"

from index 'Orders/ByShipToAndLines' as o
select 
{ 
    ShipTo: o.ShipTo, 
    Products : o.Lines.map(function(y){return y.ProductName;}) 
}

Example IV - Projection with Expression

results = list(
    session.advanced.raw_query(
        'from Employees as e select { FullName: e.FirstName + " " + e.LastName }', FullName
    )
)

class Employees_ByFirstAndLastName(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from employee in docs.Employees "
            "select new "
            "{"
            " FirstName = employee.FirstName,"
            " LastName = employee.LastName"
            "}"
        )

from index 'Employees/ByFirstAndLastName' as e
select 
{ 
    FullName : e.FirstName + " " + e.LastName 
}

Example V - Projection with let

results = list(
    session.advanced.raw_query(
        "declare function output (e) { "
        '    var format = function(p){ return p.FirstName + " " + p.LastName; };'
        "    return { FullName : format(e) }; "
        "} "
        "from Employees as e select output(e)",
        Employee,
    )
)

class Employees_ByFirstAndLastName(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from employee in docs.Employees "
            "select new "
            "{"
            " FirstName = employee.FirstName,"
            " LastName = employee.LastName"
            "}"
        )

declare function output(e) {
	var format = function(p){ return p.FirstName + " " + p.LastName; };
	return { FullName : format(e) };
}
from index 'Employees/ByFirstAndLastName' as e select output(e)

Example VI - Projection with Calculation

results = session.advanced.raw_query(
    "from Orders as o "
    "select { "
    "    Total : o.Lines.reduce( "
    "        (acc, 1) => acc += l.PricePerUnit * l.Quantity, 0) "
    "}",
    Total,
)

class Orders_ByShipToAndLines(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = "from order in docs.Orders select new { ShipTo = order.ShipTo, Lines = order.Lines}"

from index 'Orders/ByShipToAndLines' as o
select {
    Total : o.Lines.reduce(
        (acc , l) => acc += l.PricePerUnit * l.Quantity, 0)
}

Example VII - Projection With a Count() Predicate

class Orders_ByShippedAtAndCompany(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from order in docs.Orders "
            "select new "
            "{"
            " ShippedAt = order.ShippedAt,"
            " Company = order.Company"
            "}"
        )

from Orders as o 
load o.Company as c 
select 
{ 
    CompanyName : c.Name, 
    ShippedAt : o.ShippedAt, 
    TotalProducts : o.Lines.length, 
    TotalDiscountedProducts : o.Lines.filter(x => x.Discount > 0 ).length 
}

Example VIII - Projection Using a Loaded Document

results = list(
    session.advanced.raw_query(
        "from Orders as o "
        "load o.company as c "
        "select { "
        "    CompanyName: c.Name,"
        "    ShippedAt: o.ShippedAt"
        "}",
        OrderProjection,
    )
)

class Orders_ByShippedAtAndCompany(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from order in docs.Orders "
            "select new "
            "{"
            " ShippedAt = order.ShippedAt,"
            " Company = order.Company"
            "}"
        )

from index 'Orders/ByShippedAtAndCompany' as o
load o.Company as c
select {
	CompanyName: c.Name,
	ShippedAt: o.ShippedAt
}

Example IX - Projection with Dates

results = list(
    session.advanced.raw_query(
        "from Employees as e "
        "select { "
        "    DayOfBirth : new Date(Date.parse(e.Birthday)).getDate(), "
        "    MonthOfBirth : new Date(Date.parse(e.Birthday)).getMonth() + 1, "
        "    Age : new Date().getFullYear() - new Date(Date.parse(e.Birthday)).getFullYear() "
        "}"
    )
)

class Employees_ByFirstNameAndBirthday(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from employee in docs.Employees "
            "select new "
            "{"
            " FirstName = employee.FirstName,"
            " Birthday = employee.Birthday"
            "}"
        )

from index 'Employees/ByFirstNameAndBirthday' as e 
select { 
    DayOfBirth : new Date(Date.parse(e.Birthday)).getDate(), 
    MonthOfBirth : new Date(Date.parse(e.Birthday)).getMonth() + 1,
    Age : new Date().getFullYear() - new Date(Date.parse(e.Birthday)).getFullYear() 
}

Example X - Projection with Raw JavaScript Code

results = list(
    session.advanced.raw_query(
        "from Employees as e "
        "select { "
        "    Date : new Date(Date.parse(e.Birthday)), "
        "    Name : e.FirstName.substr(0,3) "
        "}",
        EmployeeProjection,
    )
)

class Employees_ByFirstNameAndBirthday(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from employee in docs.Employees "
            "select new "
            "{"
            " FirstName = employee.FirstName,"
            " Birthday = employee.Birthday"
            "}"
        )

from index 'Employees/ByFirstNameAndBirthday' as e 
select {
    Date : new Date(Date.parse(e.Birthday)), 
    Name : e.FirstName.substr(0,3)
}

Example XI - Projection with Metadata

results = list(
    session.advanced.raw_query(
        "from Employee as e " "select {" "    Name : e.FirstName, " "    Metadata : getMetadata(e)" "}",
        Employee,
    )
)

class Employees_ByFirstAndLastName(AbstractIndexCreationTask):
    def __init__(self):
        super().__init__()
        self.map = (
            "from employee in docs.Employees "
            "select new "
            "{"
            " FirstName = employee.FirstName,"
            " LastName = employee.LastName"
            "}"
        )

from index 'Employees/ByFirstAndLastName' as e 
select {
     Name : e.FirstName, 
     Metadata : getMetadata(e)
}

def select_fields(
    self,
    projection_class: Type[_TProjection],
    *fields: str,
    projection_behavior: Optional[ProjectionBehavior] = ProjectionBehavior.DEFAULT,
) -> DocumentQuery[_TProjection]: ...

def select_fields_query_data(
    self, projection_class: Type[_TProjection], query_data: QueryData
) -> DocumentQuery[_TProjection]: ...

class QueryData:
    def __init__(
        self,
        fields: List[str],
        projections: List[str],
        from_alias: Optional[str] = None,
        declare_tokens: Optional[List[DeclareToken]] = None,
        load_tokens: Optional[List[LoadToken]] = None,
        is_custom_function: Optional[bool] = None,
    ):
        self.fields = fields
        self.projections = projections
        self.from_alias = from_alias
        self.declare_tokens = declare_tokens
        self.load_tokens = load_tokens
        self.is_custom_function = is_custom_function

        self.map_reduce: Union[None, bool] = None
        self.project_into: Union[None, bool] = None
        self.projection_behavior: Union[None, ProjectionBehavior] = None

ProjectionBehavior Syntax:

class ProjectionBehavior(Enum):
    DEFAULT = "Default"
    FROM_INDEX = "FromIndex"
    FROM_INDEX_OR_THROW = "FromIndexOrThrow"
    FROM_DOCUMENT = "FromDocument"
    FROM_DOCUMENT_OR_THROW = "FromDocumentOrThrow"

• Default
  Retrieve values from the stored index fields when available.
  If fields are not stored then get values from the document,
  a field that is not found in the document is skipped.

• FromIndex
  Retrieve values from the stored index fields when available.
  A field that is not stored in the index is skipped.

• FromIndexOrThrow
  Retrieve values from the stored index fields when available.
  An exception is thrown if the index does not store the requested field.

• FromDocument
  Retrieve values directly from the documents store.
  A field that is not found in the document is skipped.

• FromDocumentOrThrow
  Retrieve values directly from the documents store.
  An exception is thrown if the document does not contain the requested field.