Ongoing Tasks: RavenDB ETL

Loading Documents

• In order to load data to the destination database you must call the loadTo<CollectionName>() method and pass a JS object.

• Indicating the collection name in the load method is a convention.

• The objects passed to the loadTo method will be sent to the target database in the last Load stage.

• All results created in a single ETL run will be sent in a single batch and processed transactionally on the destination.

• For example, if you want to write data to Employees collection you need to call the following method in the script body:

loadToEmployees({ ... });

• The method parameter must be a JS object. You can create it as follows:

loadToEmployees({
    Name: this.FirstName + " " + this.LastName
});

• Or simply transform the current document object and pass it:

this.Name = this.FirstName + " " + this.LastName;

delete this.Address;
delete this.FirstName;
delete this.LastName;

loadToEmployees(this);

Alternative Syntax

The target collection name can be passed to the loadTo command separately, as a string argument, using this syntax: loadTo('Target', obj)

• Example:
  The following two calls to loadTo are equivalent.
  loadToEmployees(this);
loadTo('Employees', this);

Documents Identifiers

• The documents generated in the destination database are given an id according to the collection name specified in the loadTo method.

• If the specified collection is the same as the original one then the document is loaded to the same collection and the original identifier is preserved.

• For example, the following ETL script defined on Employees collection will keep the same idenifiers in the target database:

// original identifier will be preserved
loadToEmployees({ ... });

• If the 'loadTo' method indicates a different target collection, e.g. People,
  then the employee documents will get new identifiers that combine the original ID and the new collection name in the destination database.

// new identifier will be generated
loadToPeople({ ... });

• In addition, ETL appends the symbol / to the requested id so that the target database will generate identifiers on its side.
  As a result, documents in the People collection in the target database will have identifiers such as: employees/1-A/people/0000000000000000001-A.

Filtering

• Documents can be filtered from the ETL by calling the loadTo method only for documents that match some condition:

if (this.Active) {
    // load only active users
    loadToEmployees({ ... });
}

Loading Data from Other Documents

• The load method loads a document with a specified ID during script execution.

// this.ReportsTo has some document ID
var manager = load(this.ReportsTo);

Accessing Metadata

• The metadata can be accessed in the following way:

var value = this['@metadata']['custom-metadata-key'];

Creating Multiple Documents from a Single Document

• The loadTo method can be called multiple times in a single script.
  That allows you to split a single source document into multiple documents on the destination database:

// documents will be creaed in `Addresses` collection
loadToAddresses({
    City: this.Address.City ,
    Country: this.Address.Country ,
    Address: this.Address.Line1
});

delete this.Address;

// documents will be created in the `Employees` collection
loadToEmployees(this);