Bulk Insert: How to Work With Bulk Insert Operation

BulkInsertOperation BulkInsert(string database = null, CancellationToken token = default);

BulkInsertOperation BulkInsert(string database, BulkInsertOptions options, CancellationToken token = default);

BulkInsertOperation BulkInsert(BulkInsertOptions options, CancellationToken token = default);

The following methods can be used when creating a bulk insert.

Methods

Limitations

• BulkInsert is designed to efficiently push high quantities of data.
  As such, data is streamed and processed by the server in batches.
  Each batch is fully transactional, but there are no transaction guarantees between the batches. The operation as a whole is non-transactional. If your bulk insert is interrupted mid-way, some of your data might be persisted on the server while some of it might not.
  • Make sure that your logic accounts for the possibility of an interruption where some of your data has not yet persisted on the server.
  • If the operation was interrupted and you choose to re-insert the whole dataset in a new operation, you can configure SkipOverwriteIfUnchanged as true.
    It only overwrites existing documents if a change has been made since the last insertion.
  • If you need full transactionality, the session may be a better option.
    If using the session, because all of the data is processed in one transaction, your server resources must be able to handle the entire data-set included in the transaction.
• Bulk insert is not thread-safe.
  A single bulk insert should not be accessed concurrently.
  • The use of multiple bulk inserts concurrently on the same client is supported.
  • Also the use in an async context is supported.

Example

Create bulk insert

Here we create a bulk insert operation and insert a million documents of type Employee:

using (BulkInsertOperation bulkInsert = store.BulkInsert())
{
    for (int i = 0; i < 1000 * 1000; i++)
    {
        bulkInsert.Store(new Employee
        {
            FirstName = "FirstName #" + i,
            LastName = "LastName #" + i
        });
    }
}

BulkInsertOperation bulkInsert = null;
try
{
    bulkInsert = store.BulkInsert();
    for (int i = 0; i < 1000 * 1000; i++)
    {
        await bulkInsert.StoreAsync(new Employee
        {
            FirstName = "FirstName #" + i,
            LastName = "LastName #" + i
        });
    }
}
finally
{
    if (bulkInsert != null)
    {
        await bulkInsert.DisposeAsync().ConfigureAwait(false);
    }
}

The following options can be configured for BulkInsert.

CompressionLevel

SkipOverwriteIfUnchanged

Prevent overriding documents if there are no changes when compared to the already existing ones.

Enabling this can avoid a lot of additional work including triggering re-indexation, subscriptions, and ETL processes.
It introduces slight overlay into the bulk insert process because of the need to compare the existing documents with the ones that are being inserted.

using (var bulk = store.BulkInsert(new BulkInsertOptions
{
    SkipOverwriteIfUnchanged = true
}));