see on GitHub

Bulk inserts

One of the features that is particularly useful when inserting large amount of data is bulk inserting. This is an optimized time-saving approach with few drawbacks that will be described later.

From the developer side, the API can be used as follows:

using (var bulkInsert = store.BulkInsert())
{
	for (int i = 0; i < 1000 * 1000; i++)
	{
		bulkInsert.Store(new User
			{
				Name = "Users #" + i
			});
	}
}

The BulkInsert method is a part of IDocumentStore and is used to start a BulkInsertOperation.

public interface IDocumentStore
{
	BulkInsertOperation BulkInsert(string database = null, BulkInsertOptions options = null);
}

public class BulkInsertOptions
{
	public bool CheckForUpdates { get; set; }

	public bool CheckReferencesInIndexes { get; set; }

	public int BatchSize { get; set; }
}

The available BulkInsertOptions are:

  • CheckForUpdates - enables document updates (default: false)
  • CheckReferencesInIndexes - enables reference checking (default: false)
  • BatchSize - used batch size (default: 512)

public class BulkInsertOperation
{
    public delegate void BeforeEntityInsert(string id, RavenJObject data, RavenJObject metadata);
    
	public event BeforeEntityInsert OnBeforeEntityInsert = delegate { };

	public event Action<string> Report { ... }

	public void Store(object entity) { ... }

	public void Store(object entity, string id) { ... }
}

The BulkInsertOperation consist of following the methods and events:

  • Store - method used to store the entity, with optional id parameter to explicitly declare the entity identifier (will be generated automatically on client-side when overload without id will be used)
  • Report - event that will be raised every time a batch has finished processing and after the whole operation
  • OnBeforeEntityInsert - event that will be raised before entity will be processed

Limitations

There are several limitations to the API:

  • Entity Id must be provided at the client side. The client by default will use the HiLo generator in order to generate the Id.
  • Transactions are per batch, not per operation and DTC transactions are not supported.
  • Documents inserted using bulk-insert will not raise notifications. More about Changes API can be found here.
  • Document Updates and Reference Checking must be explicitly turned on (see BulkInsertOptions).
  • AfterCommit method in Put Triggers will be not executed in contrast to AllowPut, AfterPut and OnPut.