Session: Querying:

How to Filter by Non-Existing Field



Query a Static Index

You can search for documents with missing fields by using a static index if it indexes the field which is suspected to be missing in some of the documents.

The index definition must also index a field that exists in every document (such as Id) so that all documents will be indexed.

  • For example, if you want to find documents that are missing the field Freight in the Orders collection,
    query an index that indexes the fields Freight and Id.
  • If your static index does not contain the desired field, either
    • Modify your index definition to index the specific field. (This will trigger re-indexing.)
    • Create an auto-index.

Example: Query a Static Index

In our example, we are looking for documents that are missing the field Freight from the collection Orders.

First we need an index that includes Freight and a field that exists in every document

We index the missing field Freight and the field Id, which exists in every document.
This way, the index includes all of the documents in the collection, including those that are missing the specified field.

// Create or modify a static index called Orders_ByFreight
public class Orders_ByFreight : AbstractIndexCreationTask<Order>
{
    public Orders_ByFreight()
    {
        // Specify collection name
        Map = orders => from doc in orders
                           select new
                           {
                               // Field that is missing in some documents
                               doc.Freight,
                               // Field that exists in all documents
                               doc.Id
                           };
    }
}

Then we query the index to find documents where the field does not exist

SAMPLE QUERY:

Query the index Orders_ByFreight and filter documents where freight does not exist.

List<Order> results = session
    .Advanced
    // Query the static index 
    .DocumentQuery<Order, Orders_ByFreight>()
    // Verify that the index is not stale (optional)
    .WaitForNonStaleResults(TimeSpan.MaxValue)
    // Negate the next method
    .Not
    // Specify the field that is suspected to be missing
    .WhereExists(x => x.Freight)
    .ToList();
    // `results` will contain the list of incomplete documents.
from index "Orders/ByFreight"
where true and not exists("Freight")
// `not` cannot come immediately after `where`, thus we use `where true`.

LINQ SYNTAX:

List<T> results = session
    .Advanced
    .DocumentQuery<T, TIndexCreator>()
    .Not
    .WhereExists("missingFieldName")
    .ToList();
Parameters Type Description
T string An object in a collection (singular of the collection name - e.g. Order from the collection Orders).
TIndexCreator string The name of the index that you want to use.
missingFieldName string The field that is missing in some of the documents.

Query the Collection to Create an Auto-Index

Another option is to query the collection for the missing field.
This will either create a new auto-index or add the new field to an existing auto-index if it indexes the same collection.

See the example and query syntax descriptions below:

Example: A query that creates an Auto-Index

The following query will create an auto-index on the "Freight" field that is missing in some documents in the Orders collection.
The query result will contain all documents that do not have this field.

List<Order> results = session
    .Advanced
    .DocumentQuery<Order>()
    .Not
    .WhereExists("Freight")
    .ToList();
from "Orders" 
where true and not exists("Freight")
// `not` cannot be used immediately after `where`, thus we use `where true`.

LINQ Query Syntax

List<T> results = session
    .Advanced
    .DocumentQuery<T>()
    .Not
    .WhereExists("missingFieldName")
    .ToList();
Parameters Type Description
T string An object in a collection (singular of the collection name - e.g. Order from the collection Orders).
missingFieldName string The field that is missing in some of the documents.

Use Studio to filter by non-existing field

You can also use Studio to find missing fields with an RQL query such as:

from "Orders"  
where exists("Company") and not exists("Freight")

In Studio we always use RQL syntax.
Like the LINQ syntax examples above, we must first call a field that exists in every document in the collection and then the field that does not exist in some of them.

List Documents Without a Specified Field

List Documents Without a Specified Field

  1. Indexes
    Click to see the Indexes menu items.
  2. Query
    Select to open the Query view.
  3. Query editor
    Write the query according to the RQL example described above.
  4. Run Query
    Click or press ctrl+enter to run the query.
  5. Index used
    This is the name of the auto-index created for this query.
    You can click it to see the available Studio options for this index.
  6. Results
    This is the list of documents that do not have the specified field.
    (The field "Freight" was explicitly removed from these Northwind documents for this example.)