Administration: Exporting and Importing data

To Export data, use this command:

Raven.Smuggler out http://localhost:8080/ dump.ravendump --database=Northwind

Raven.Smuggler out http://localhost:8080/databases/Northwind dump.ravendump

This command will export all indexes, transformers, documents and attachments from Northwind database to a file named dump.ravendump.

The dump file will also include documents that were added during the export process, so you can make changes while the export is being executed.

Exporting and Replication

If Replication is turned on in your source database, it is recommended that you filter out the Raven/Replication/Destinations document, using the following command:

Raven.Smuggler out http://localhost:8080 dump.ravendump --databases=Northwind --negative-metadata-filter:@id=Raven/Replication/Destinations

Raven.Smuggler in http://localhost:8080 dump.ravendump --database=NewNorthwind

This command will import all the indexes, transformers, documents, and attachments from the file to the NewNorthwind database.

You can continue using this RavenDB instance while data is being imported to it.

Importing and Replication

Note that if you have either the replication bundle or the periodic export bundle active on a database, it is recommended that you'll filter out the following documents when doing an import: Raven/Replication/Destinations, Raven/Replication/VersionHilo, Raven/Backup/Periodic/Setup, Raven/Backup/Periodic/Status.
This can be done using the following command:

Raven.Smuggler in http://localhost:8080 dump.ravendump --database=NewNorthwind --negative-metadata-filter:@id=Raven/Replication/Destinations --negative-metadata-filter:@id=Raven/Backup/Periodic/Setup --negative-metadata-filter:@id=Raven/Backup/Periodic/Status --negative-metadata-filter:@id=Raven/Replication/VersionHilo

With the incremental export operation we can use in order to backup the database incrementally, on each export, we will get to export only the documents created or updated since the last export.

To export data incrementally we can use two options:

• If it is the first run and the folder does not exist yet, use:

Raven.Smuggler out http://localhost:8080 folder_location --database=Northwind --incremental

Note that this cammand can be used every time.

• If you ran the command before or you created the folder earlier, you can use:

Raven.Smuggler out http://localhost:8080 folder_location --database=Northwind

To import data that was exported incrementally, you can use either of the following:

Raven.Smuggler in http://localhost:8080  folder_location --database=NewNorthwind --incremental

or

Raven.Smuggler in http://localhost:8080 folder_location --database=NewNorthwind

Incremental exports and deletions

To move data directly between two instances (or different databases in the same instance) use the between option in following manner:

Raven.Smuggler between http://localhost:8080 http://localhost:8080 folder_location --database=Northwind --database2=NewNorthwind

You can tweak the export/import process with the following parameters:

• database: The database to operate on. If not specified, the operations will be performed on a default database.
• database2: The database to operate on. If not specified, the operations will be performed on a default database (used only in the between operation).
• filesystem: The filesystem to operate on.
• filesystem2: The filesystem to operate on (used only in the between operation).
• operate-on-types: Specify the types to export/import. Usage example: --operate-on-types=Indexes,Documents,Attachments,Transformers.
• filter: Filter documents by a document property. Usage example: --filter=Property-Name=Value.
• negative-filter: Filter documents NOT matching a document property. Usage example: --negative-filter=Property-Name=Value.
• metadata-filter: Filter documents by a metadata property. Usage example: --metadata-filter=Raven-Entity-Name=Posts.
• negative-metadata-filter: Filter documents NOT matching a metadata property. Usage example: --negative-metadata-filter=Raven-Entity-Name=Posts.
• ignore-errors-and-continue: If this option is enabled, smuggler will not halt its operation on errors. Errors still will be displayed to the user.
• system-database: Set to true in order to work on a system database.
• transform: Transform documents using a given script.
• transform-file: Transform documents using a given script file.
• max-steps-for-transform-script: Maximum number of steps that a transform script can have.
• timeout: The timeout (in milliseconds) to use for requests.
• batch-size: The batch size for requests.
• chunk-size: The number of documents to import before new connection will be opened.
• username: A username to use when a database requires client authentication.
• username2: A username to use when a database requires client authentication (used only in the between operation).
• password: A password to use when a database requires client authentication.
• password2: A password to use when a database requires client authentication (used only in the between operation).
• domain: A domain to use when a database requires client authentication.
• domain2: A domain to use when a database requires client authentication (used only in the between operation).
• api-key: An API-key to use, when using OAuth.
• api-key2: An API-key to use, when using OAuth (used only in the between operation).
• incremental: States usage of incremental operations.
• wait-for-indexing: Wait until all indexing activities have been completed (import only).
• excludeexpired: Excludes expired documents created by the expiration bundle.
• limit: Reads at most VALUE documents/attachments.
• strip-replication-information: Remove all replication information from metadata (import only).
• disable-versioning-during-import: Disables versioning for the duration of the import.
• continuation-token: Activates the usage of a continuation token in case of unreliable connections or huge imports.
• skip-conflicted: The database will issue and error when conflicted documents are put. The default is to alert the user, this allows to skip them to continue.
• counter: The counter storage to operate on. If no specified, the operations will be on the default counter storage.
• counter2: The counter storage to operate on. If no specified, the operations will be on the default counter storage (used only in the between operation).
• no-compression-on-import: A flag that if set to true disables compression usage during import of documents.
• help: You can use the help option in order to print the built-in options documentation.
• start-document-etag: Start exporting from the specified documents etag
• start-document-delete-etag: Start exporting from the specified document deletion etag
• start-attachment-etag: Start exporting from the specified attachment deletion etag
• start-attachment-delete-etag: Start exporting from the specified attachment deletion etag

To filter out documents, we introduced a few filtering options that can be used during an import or export process.

1. filter is used to filter documents according to their properties. E.g. if we want to export all documents with a property Name and value John, we have to apply command as follows: --filter=Name=John .
2. negative-filter is a countertype to filter and will filter documents that do NOT match the given property.
3. metadata-filter is similar to filter but works on the document's metadata properties.
4. negative-metadata-filter filters out documents that do NOT match given metadata property.

Example I - basics

To export all documents containing property FirstName with value Robert we need to execute following command:

Raven.Smuggler out http://localhost:8080/ dump.ravendump --database=Northwind --filter=FirstName=Robert --operate-on-types=Documents

Example II - basics

To export all documents from Employees collection and HiLo document for this collection (if exists) following command needs to be executed:

Raven.Smuggler out http://localhost:8080/ dump.ravendump --database=Northwind --metadata-filter=Raven-Entity-Name=Employees --operate-on-types=Documents

Example III - combining filters

To export all documents from Employees collection that contain property FirstName with value Robert and HiLo document for this collection (if exists) following command needs to be executed:

Raven.Smuggler out http://localhost:8080/ dump.ravendump --database=Northwind --filter=FirstName=Robert --metadata-filter=Raven-Entity-Name=Employees --operate-on-types=Documents

Example IV - multiple filters

Same filter can be used multiple times, but one must remember that document must match all the filters (logical AND), so to export all documents that contain property FirstName with value Robert and LastName with value King then following command must be executed:

Raven.Smuggler out http://localhost:8080/ dump.ravendump --database=Northwind --filter=FirstName=Robert --filter=LastName=King --operate-on-types=Documents

Example V - importing

Filter can be also used during import and they work the same way as in export. E.g. if we have an export containing whole Northwind database and we want to only import documents from Employees collection then we must execute following command:

Raven.Smuggler in http://localhost:8080/ dump.ravendump --database=NewNorthwind --metadata-filter=Raven-Entity-Name=Employees --operate-on-types=Documents

Transforms can be used to modify or filter out documents, but they work only during the import process. The scripts have to use JavaScript syntax and be in following format:

function(doc) {
	// custom code here
}

where doc will contain our document with its metadata under the @metadata property.

Change scripts:

E.g. To change document's property Name value to a new one, the following script can be used:

function(doc) {
	doc['Name'] = 'NewValue';
	return doc;
}

Filter scripts:

If we return null, the document will be filtered out.

function(doc) {
	var id = doc['@metadata']['@id'];
	if(id === 'orders/999')
		return null;
	return doc;
}

Alternatively, if you prefer to do export/import from code rather than from the console utility, you can use the SmugglerDatabaseApi class (in order to use this class you need to reference the Raven.Database.dll).

Exporting

// export Documents, Indexes, Attachments and Transformers
// to dump.raven file
// from Northwind database
// found on http://localhost:8080 server
SmugglerDatabaseApi smugglerApi = new SmugglerDatabaseApi(new SmugglerDatabaseOptions
{
	OperateOnTypes =
		ItemType.Documents | ItemType.Indexes | ItemType.Attachments | ItemType.Transformers,
	Incremental = false
});

SmugglerExportOptions<RavenConnectionStringOptions> exportOptions = new SmugglerExportOptions<RavenConnectionStringOptions>
{
	ToFile = "dump.raven",
	From = new RavenConnectionStringOptions
	{
		DefaultDatabase = "Northwind",
		Url = "http://localhost:8080"
	}
};

await smugglerApi.ExportData(exportOptions);

Importing

// import only Documents
// from dump.raven file
// to NewNorthwind database (must exist)
// found on http://localhost:8080 server
SmugglerDatabaseApi smugglerApi = new SmugglerDatabaseApi(new SmugglerDatabaseOptions
{
	OperateOnTypes = ItemType.Documents,
	Incremental = false
});

SmugglerImportOptions<RavenConnectionStringOptions> importOptions = new SmugglerImportOptions<RavenConnectionStringOptions>
{
	FromFile = "dump.raven",
	To = new RavenConnectionStringOptions
	{
		DefaultDatabase = "NewNorthwind",
		Url = "http://localhost:8080"
	}
};

await smugglerApi.ImportData(importOptions, null);

Moving data between two databases

// export Documents and Indexes
// from Northwind database
// found on http://localhost:8080
// and import them to NewNorthwind
// found on the same server
SmugglerDatabaseApi smugglerApi = new SmugglerDatabaseApi(new SmugglerDatabaseOptions
{
	OperateOnTypes = ItemType.Documents | ItemType.Indexes,
	Incremental = false
});

await smugglerApi.Between(
	new SmugglerBetweenOptions<RavenConnectionStringOptions>
	{
		From = new RavenConnectionStringOptions
		{
			Url = "http://localhost:8080",
			DefaultDatabase = "Northwind"
		},
		To = new RavenConnectionStringOptions
		{
			Url = "http://localhost:8080",
			DefaultDatabase = "NewNorthwind"
		}
	});

A smuggler communicates with a server using the HTTP protocol, meaning that it cannot communicate with Embedded instance as long as the UseEmbeddedHttpServer is set to false (which is a default value). If embedded http server cannot be started, the DatabaseDataDumper (found in Raven.Database.Smuggler namespace in Raven.Database.dll) can be used to import or export data.

using (EmbeddableDocumentStore store = new EmbeddableDocumentStore
					   {
						   DefaultDatabase = "Northwind"
					   })
{
	store.Initialize();

	var dataDumper = new DatabaseDataDumper(
		store.DocumentDatabase,
		new SmugglerDatabaseOptions
		{
			OperateOnTypes = ItemType.Documents | ItemType.Indexes | ItemType.Attachments | ItemType.Transformers,
			Incremental = false
		});

	SmugglerExportOptions<RavenConnectionStringOptions> exportOptions = new SmugglerExportOptions<RavenConnectionStringOptions>
	{
		From = new EmbeddedRavenConnectionStringOptions(),
		ToFile = "dump.raven"
	};

	await dataDumper.ExportData(exportOptions);
}