Backup



Backup Types

Logical-Backup

  • Data, index definitions, and ongoing tasks are backed-up in compressed JSON files.

  • During the restoration, RavenDB -

    • Re-inserts all data into the database.
    • Inserts the saved index definitions. To save space, Logical Backup stores index definitions only.
      After restoration, the dataset is scanned and indexed according to the definitions.
  • See backup contents.

  • Restoration time is, therefore, slower than when restoring from a Snapshot.

  • The backup file size is significantly smaller than that of a Snapshot.

  • In addition to full data backup, Logical Backups can be defined as incremental, saving any changes made since the previous backup.

  • The following code sample defines a full-backup task that would be executed every 3 hours:

    var config = new PeriodicBackupConfiguration
    {
        LocalSettings = new LocalSettings
        {
            // Local path for storing the backup
            FolderPath = @"E:\RavenBackups"
        },
    
        // Full Backup period (Cron expression for a 3-hours period)
        FullBackupFrequency = "0 */3 * * *",
    
        // Set backup type to Logical-Backup
        BackupType = BackupType.Backup,
    
        // Task Name
        Name = "fullBackupTask",
    };
    var operation = new UpdatePeriodicBackupOperation(config);
    var result = await docStore.Maintenance.SendAsync(operation);
    Note the usage of Cron scheduling when setting backup frequency.

Snapshot

  • A Snapshot is a compressed binary duplication of the full database structure. This includes the data file and the journals at a given point in time.
    Therefore it includes fully built indexes and ongoing tasks.
    See file structure for more info.

  • Snapshot backups are available only for Enterprise subscribers.

  • During restoration -

    • Re-inserting data into the database is not required.
    • Re-indexing is not required.
  • See backup contents.

  • Restoration is typically faster than that of a logical backup.

  • Snapshot size is typically larger than that of a logical backup.

  • If Incremental backups are created for a Snapshot-type backup:

    • The first backup will be a full Snapshot.
    • The following backups will be Incremental.
    • Incremental backups have different storage contents than Snapshots.
  • Code Sample:

    // Set backup type to Snapshot
    BackupType = BackupType.Snapshot,

Basic Comparison Between a Logical-Backup and a Snapshot:

Backup Type Stored Format Restoration speed Size
Snapshot Compressed Binary Image Fast Larger than a logical-backup
Logical backup Compressed Textual Data - JSON Slow Smaller than a Snapshot

Make sure your server has access to the local backup path.

Verify that RavenDB is allowed to store files in the path set in LocalSettings.FolderPath.

Backup Scope

As described in the overview, a backup task can create full and incremental backups.

  • A Backup Task can be defined to create either a full data backup or an incremental backup.
    In both cases, the backup task adds a single new backup file to the backup folder each time it runs, leaving the existing backup files untouched.

Full-Backup

  • File Format
    A full-backup is a compressed JSON file if it is a logical backup, or a compressed binary file if it is a snapshot.

  • Task Ownership
    There are no preliminary conditions for creating a full-backup. Any node can perform this task.

  • To run a full-backup
    Set FullBackupFrequency.

    // A full-backup will run every 6-hours (Cron expression)
    FullBackupFrequency = "0 */6 * * *",

Incremental-Backup

  • File Format and Notes About Contents

    • An incremental-backup file is always in JSON format. It is so even when the full-backup it is associated with is a binary snapshot.
    • An incremental backup stores index definitions (not full indexes).
      After the backup is restored, the dataset is re-indexed according to the index definitions.

      This initial re-indexing can be time-consuming on large datasets.

    • An incremental backup doesn't store change vectors.
  • Task Ownership
    The ownership of an incremental-backup task is granted dynamically by the cluster.
    An incremental-backup can be executed only by the same node that currently owns the backup task.
    A node can run an incremental-backup, only after running full-backup at least once.

  • To run an incremental-backup
    Set IncrementalBackupFrequency.

// An incremental-backup will run every 20 minutes (Cron expression)
IncrementalBackupFrequency = "*/20 * * * *",

Backup to Local and Remote Destinations

  • Backups can be made locally, as well as to a set of remote locations including -

    • A network path
    • An FTP/SFTP target
    • Azure Storage
    • Amazon S3
    • Amazon Glacier
    • Google Cloud
  • RavenDB will store data in a local folder first, and transfer it to the remote destination from the local one.

    • If a local folder hasn't been specified, RavenDB will use the temp folder defined in its Storage.TempPath setting.
      If Storage.TempPath is not defined, the temporary files will be created at the same location as the data file.
      In either case, the folder will be used as temporary storage and the local files deleted from it when the transfer is completed.
    • If a local folder has been specified, RavenDB will use it both for the transfer and as its permanent local backup location.
  • Local and Remote Destinations Settings Code Sample:

    var config = new PeriodicBackupConfiguration
    {
        LocalSettings = new LocalSettings
        {
            FolderPath = @"E:\RavenBackups"
        },
    
        // FTP Backup settings
        FtpSettings = new FtpSettings
        {
            Url = "192.168.10.4:8080",
            UserName = "John",
            Password = "JohnDoe38"
        },
    
        // Azure Backup settings
        AzureSettings = new AzureSettings
        {
            StorageContainer = "storageContainer",
            RemoteFolderName = "remoteFolder",
            AccountName = "JohnAccount",
            AccountKey = "key"
        },
    
        // Amazon S3 bucket settings.
        S3Settings = new S3Settings
        {
            AwsAccessKey = "your access key here",
            AwsSecretKey = "your secret key here",
            AwsRegionName = "OPTIONAL",
            BucketName = "john-bucket"
        },
    
        // Amazon Glacier settings.
        GlacierSettings = new GlacierSettings
        {
            AwsAccessKey = "your access key here",
            AwsSecretKey = "your secret key here",
            AwsRegionName = "OPTIONAL",
            VaultName = "john-glacier",
            RemoteFolderName = "john/backups"
        },
    
        // Google Cloud Backup settings
        GoogleCloudSettings = new GoogleCloudSettings
        {
            BucketName = "RavenBucket",
            RemoteFolderName = "BackupFolder",
            GoogleCredentialsJson = "GoogleCredentialsJson"
        }
    
    };
    var operation = new UpdatePeriodicBackupOperation(config);
    var result = await docStore.Maintenance.SendAsync(operation);

Tip

Use AWS IAM (Identity and Access Management) to restrict users access while they create backups.
E.g. -

{
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "VisualEditor0",
                    "Effect": "Allow",
                    "Action": "s3:PutObject",
                    "Resource": "arn:aws:s3:::BUCKET_NAME/*"
                },
                {
                    "Sid": "VisualEditor1",
                    "Effect": "Allow",
                    "Action": [
                        "s3:ListBucket",
                        "s3:GetBucketAcl",
                        "s3:GetBucketLocation"
                    ],
                    "Resource": "arn:aws:s3:::BUCKET_NAME"
                }
            ]
        }

Backup Retention Policy

By default, backups are stored indefinitely. The backup retention policy sets a retention period, at the end of which backups are deleted. Deletion occurs during the next scheduled backup task after the end of the retention period.

Full backups and their corresponding incremental backups are deleted together. Before a full backup can be deleted, all of its incremental backups must be older than the retention period as well.

The retention policy is a property of PeriodicBackupConfiguration:

public class RetentionPolicy
{
    public bool Disabled { get; set; }
    public TimeSpan? MinimumBackupAgeToKeep { get; set; }
}
Parameter Type Description
Disabled bool If set to true, backups will be retained indefinitely, and not deleted. Default: false
MinimumBackupAgeToKeep TimeSpan The minimum amount of time to retain a backup. Once a backup is older than this time span, it will be deleted during the next scheduled backup task.

Example

var config = new PeriodicBackupConfiguration
{
    RetentionPolicy = new RetentionPolicy
    {
        Disabled = false, // False is the default value
        MinimumBackupAgeToKeep = TimeSpan.FromDays(100)
    }
};

Server-Wide Backup

You can create a Server-Wide Backup task to back-up all the databases in your cluster at a scheduled time.
Individual databases can be excluded from the backup. Learn more in Studio: Server-Wide Backup.

Backups can be made locally, as well as to a set of remote locations.

Examples

A server-wide backup configuration that sets multiple destinations:

var putConfiguration = new ServerWideBackupConfiguration
{
    Disabled = true,
    FullBackupFrequency = "0 2 * * 0",
    IncrementalBackupFrequency = "0 2 * * 1",

    //Backups are stored in this folder first, and sent from it to remote destinations (if defined).
    LocalSettings = new LocalSettings
    {
        FolderPath = "localFolderPath"
    },

    //FTP settings
    FtpSettings = new FtpSettings
    {
        Url = "ftps://localhost/john/backups"
    },

    //Microsoft Azure settings.
    AzureSettings = new AzureSettings
    {
        AccountKey = "Azure Account Key",
        AccountName = "Azure Account Name",
        RemoteFolderName = "john/backups"
    },

    //Amazon S3 bucket settings.
    S3Settings = new S3Settings
    {
        AwsAccessKey = "Amazon S3 Access Key",
        AwsSecretKey = "Amazon S3 Secret Key",
        AwsRegionName = "Amazon S3 Region Name",
        BucketName = "john-bucket",
        RemoteFolderName = "john/backups"
    },

    //Amazon Glacier settings.
    GlacierSettings = new GlacierSettings
    {
        AwsAccessKey = "Amazon Glacier Access Key",
        AwsSecretKey = "Amazon Glacier Secret Key",
        AwsRegionName = "Amazon Glacier Region Name",
        VaultName = "john-glacier",
        RemoteFolderName = "john/backups"
    },

    //Google Cloud Backup settings
    GoogleCloudSettings = new GoogleCloudSettings
    {
        BucketName = "Google Cloud Bucket",
        RemoteFolderName = "BackupFolder",
        GoogleCredentialsJson = "GoogleCredentialsJson"
    }
};

var result = await store.Maintenance.Server.SendAsync(new PutServerWideBackupConfigurationOperation(putConfiguration));
var serverWideConfiguration = await store.Maintenance.Server.SendAsync(new GetServerWideBackupConfigurationOperation(result.Name));

A server-wide backup configuration that excludes several databases:

var DBExcludeConfiguration = new ServerWideBackupConfiguration
{
    Disabled = true,
    FullBackupFrequency = "0 2 * * 0",
    LocalSettings = new LocalSettings
    {
        FolderPath = "localFolderPath"
    },
    ExcludedDatabases = new []
    {
        "DB1",
        "DB2",
        "DB5",
        "NorthWind",
        "DB2_Jun_2018_Backup"
    }
};

var result = await store.Maintenance.Server.SendAsync(new PutServerWideBackupConfigurationOperation(DBExcludeConfiguration));

Initiate Immediate Backup Execution

The Backup task is executed periodically on its predefined schedule.
If needed, it can also be executed immediately.

  • To execute an existing backup task immediately, use the StartBackupOperation method.

    // Create a new backup task
    var operation = new UpdatePeriodicBackupOperation(config);
    var result = await docStore.Maintenance.SendAsync(operation);
    
    // Run the backup task immediately
    await docStore.Maintenance.SendAsync(new StartBackupOperation(true, result.TaskId));
    • Definition:

      public StartBackupOperation(bool isFullBackup, long taskId)
    • Parameters:

      Parameter Type Functionality
      isFullBackup bool true: full-backup
      false: incremental-backup
      taskId long The existing backup task ID
  • To verify the execution results, use the GetPeriodicBackupStatusOperation method.

    // Pass the the ongoing backup task ID to GetPeriodicBackupStatusOperation  
    var backupStatus = new GetPeriodicBackupStatusOperation(result.TaskId);
    • Return Value:
      The PeriodicBackupStatus object returned from GetPeriodicBackupStatusOperation is filled with the previously configured backup parameters and with the execution results.
      public class PeriodicBackupStatus : IDatabaseTaskStatus
      {
          public long TaskId { get; set; }
          public BackupType BackupType { get; set; }
          public bool IsFull { get; set; }
          public string NodeTag { get; set; }
          public DateTime? LastFullBackup { get; set; }
          public DateTime? LastIncrementalBackup { get; set; }
          public DateTime? LastFullBackupInternal { get; set; }
          public DateTime? LastIncrementalBackupInternal { get; set; }
          public LocalBackup LocalBackup { get; set; }
          public UploadToS3 UploadToS3;
          public UploadToGlacier UploadToGlacier;
          public UploadToAzure UploadToAzure;
          public UploadToFtp UploadToFtp;
          public long? LastEtag { get; set; }
          public LastRaftIndex LastRaftIndex { get; set; }
          public string FolderName { get; set; }
          public long? DurationInMs { get; set; }
          public long Version { get; set; }
          public Error Error { get; set; }
          public long? LastOperationId { get; set; }
      }

Delay Backup Execution

The execution of a periodic backup task can be delayed for a given time period via Studio or using the DelayBackupOperation store operation.

  • Definition:

    public DelayBackupOperation(long runningBackupTaskId, TimeSpan duration)
  • Parameters:

    Parameter Type Functionality
    runningBackupTaskId long Backup task ID
    duration TimeSpan Delay Duration
  • Example:
    To delay the execution of a running backup task pass DelayBackupOperation the task's ID and the delay duration.

    // Get backup operation info
    var taskBackupInfo = await docStore.Maintenance.SendAsync(
            new GetOngoingTaskInfoOperation(taskId, OngoingTaskType.Backup)) as OngoingTaskBackup;
    
    // Set delay duration to 10 minutes from now
    var delayDuration = TimeSpan.FromMinutes(10);
    var delayUntil = DateTime.Now + delayDuration;
    
    // Delay backup operation
    await docStore.Maintenance.SendAsync(
            new DelayBackupOperation(taskBackupInfo.OnGoingBackup.RunningBackupTaskId, delayDuration));

  • Don't substitute RavenDB's backup procedures with simply copying the database folder yourself.
    The official backup procedure satisfies needs that simply copying the database folder does not. E.g. -

    • A reliable point-in-time freeze of backed up data.
    • An ACIDity of backed-up data, to keep its independence during restoration.
  • Remove old backup files regularly.
    Set the backup retention policy to remove unneeded backup files so that they don't build up.
    While setting how many days to keep your backups, consider how much of a recent database history you would like to have access to.

  • Store backup files in a location other than your database's.
    Note that backup files are always stored in a local folder first (even when the final backup destination is remote).
    Make sure that this local folder is not where your database is stored, as a precaution to keep vacant database storage space.