see on GitHub

Backup



Backup Types

Logical-Backup

  • Data is backed-up in compressed JSON files.

  • During the restoration, RavenDB -

    • Re-inserts all data into the database.
    • Re-indexes the data.
  • Restoration Time is, therefore, slower than that required when restoring a Snapshot.

  • Backup file size is significantly smaller than that of a Snapshot.

  • The following code sample defines a full-backup task that would be executed every 3 hours:

    var config = new PeriodicBackupConfiguration
    {
        LocalSettings = new LocalSettings
        {
            //Local path for storing the backup
            FolderPath = @"E:\RavenBackups"
        },
    
        //Full Backup period (Cron expression for a 3-hours period)
        FullBackupFrequency = "0 */3 * * *",
    
        //Set backup type to Logical-Backup
        BackupType = BackupType.Backup,
    
        //Task Name
        Name = "fullBackupTask",
    };
    var operation = new UpdatePeriodicBackupOperation(config);
    var result = await docStore.Maintenance.SendAsync(operation);
    Note the usage of Cron scheduling when setting backup frequency.

Snapshot

  • A SnapShot is a compressed binary duplication of the database and journals file structure at a given point-in-time.
    Snapshot-backups are available only for Enterprise subscribers.
  • During restoration -

    • Re-inserting data into the database is not required.
    • Re-indexing is not required.
  • Restoration is typically faster than that of a logical backup.

  • Snapshot size is typically larger than that of a logical backup.

  • Code Sample:

    //Set backup type to Snapshot
    BackupType = BackupType.Snapshot,

Basic Comparison Between a Logical-Backup and a Snapshot:

Backup Type Stored Format Restoration speed Size
Snapshot Compressed Binary Image Fast Larger than a logical-backup
Logical backup Compressed Textual Data Slow Smaller than a Snapshot

Make sure your server has access to the local backup path.

Verify that RavenDB is allowed to store files in the path set in LocalSettings.FolderPath.

Backup Scope

As described in the overview, a backup task can create full and incremental backups.

  • A Backup Task can be defined to create either a full data backup or an incremental backup.
    In both cases, the backup task adds a single new backup file to the backup folder each time it runs, leaving the existing backup files untouched.

Full-Backup

  • File Format
    A full-backup is a compressed JSON file if it is a logical backup, or a compressed binary file if it is a snapshot.

  • Task Ownership
    There are no preliminary conditions for creating a full-backup. Any node can perform this task.

  • To run a full-backup
    Set FullBackupFrequency.

    //A full-backup will run every 6-hours (Cron expression)
    FullBackupFrequency = "0 */6 * * *",

Incremental-Backup

  • File Format
    An incremental-backup file is always in JSON format. It is so even when the full-backup it supplements is a binary snapshot.

  • Task Ownership
    The ownership of an incremental-backup task is granted dynamically by the cluster.
    An incremental-backup can be executed only by the same node that currently owns the backup task.
    A node can run an incremental-backup, only after running full-backup at least once.

  • To run an incremental-backup
    Set IncrementalBackupFrequency.

    //An incremental-backup will run every 20 minutes (Cron expression)
    IncrementalBackupFrequency = "*/20 * * * *",

Backup to Local and Remote Destinations

  • Backups can be made locally, as well as to a set of remote locations including -

    • A network path
    • An FTP/SFTP target
    • Azure Storage
    • Amazon S3
    • Amazon Glacier
    • Google Cloud
  • RavenDB will store data in a local folder first, and transfer it to the remote destination from the local one.

    • If a local folder hasn't been specified, RavenDB will use the temp folder defined in its Storage.TempPath setting.
      If Storage.TempPath is not defined, the temporary files will be created at the same location as the data file.
      In either case, the folder will be used as temporary storage and the local files deleted from it when the transfer is completed.
    • If a local folder has been specified, RavenDB will use it both for the transfer and as its permanent local backup location.
  • Local and Remote Destinations Settings Code Sample:

    var config = new PeriodicBackupConfiguration
    {
        LocalSettings = new LocalSettings
        {
            FolderPath = @"E:\RavenBackups"
        },
    
        //FTP Backup settings
        FtpSettings = new FtpSettings
        {
            Url = "192.168.10.4",
            Port = 8080,
            UserName = "John",
            Password = "JohnDoe38"
        },
    
        //Azure Backup settings
        AzureSettings = new AzureSettings
        {
            StorageContainer = "storageContainer",
            RemoteFolderName = "remoteFolder",
            AccountName = "JohnAccount",
            AccountKey = "key"
        },
    
        //Amazon S3 bucket settings.
        S3Settings = new S3Settings
        {
            AwsAccessKey = "your access key here",
            AwsSecretKey = "your secret key here",
            AwsRegionName = "OPTIONAL",
            BucketName = "john-bucket"
        },
    
        //Amazon Glacier settings.
        GlacierSettings = new GlacierSettings
        {
            AwsAccessKey = "your access key here",
            AwsSecretKey = "your secret key here",
            AwsRegionName = "OPTIONAL",
            VaultName = "john-glacier",
            RemoteFolderName = "john/backups"
        },
    
        //Google Cloud Backup settings
        GoogleCloudSettings = new GoogleCloudSettings
        {
            BucketName = "RavenBucket",
            RemoteFolderName = "BackupFolder",
            GoogleCredentialsJson = "GoogleCredentialsJson"
        }
    
    };
    var operation = new UpdatePeriodicBackupOperation(config);
    var result = await docStore.Maintenance.SendAsync(operation);

Tip

Use AWS IAM (Identity and Access Management) to restrict users access while they create backups.
E.g. -

{
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "VisualEditor0",
                    "Effect": "Allow",
                    "Action": "s3:PutObject",
                    "Resource": "arn:aws:s3:::BUCKET_NAME/*"
                },
                {
                    "Sid": "VisualEditor1",
                    "Effect": "Allow",
                    "Action": [
                        "s3:ListBucket",
                        "s3:GetBucketAcl",
                        "s3:GetBucketLocation"
                    ],
                    "Resource": "arn:aws:s3:::BUCKET_NAME"
                }
            ]
        }

Server-Wide Backup

You can create a Server-Wide Backup task to back-up all the databases in your cluster at a scheduled time.

  • Backups can be made locally, as well as to a set of remote locations.

  • Server-wide Backup Code Sample:

    var putConfiguration = new ServerWideBackupConfiguration
    {
        Disabled = true,
        FullBackupFrequency = "0 2 * * 0",
        IncrementalBackupFrequency = "0 2 * * 1",
    
        //Backups are stored in this folder first, and sent from it to remote destinations (if defined).
        LocalSettings = new LocalSettings
        {
            FolderPath = "localFolderPath"
        },
    
        //FTP settings
        FtpSettings = new FtpSettings
        {
            Url = "ftps://localhost/john/backups"
        },
    
        //Microsoft Azure settings.
        AzureSettings = new AzureSettings
        {
            AccountKey = "Azure Account Key",
            AccountName = "Azure Account Name",
            RemoteFolderName = "john/backups"
        },
    
        //Amazon S3 bucket settings.
        S3Settings = new S3Settings
        {
            AwsAccessKey = "Amazon S3 Access Key",
            AwsSecretKey = "Amazon S3 Secret Key",
            AwsRegionName = "Amazon S3 Region Name",
            BucketName = "john-bucket",
            RemoteFolderName = "john/backups"
        },
    
        //Amazon Glacier settings.
        GlacierSettings = new GlacierSettings
        {
            AwsAccessKey = "Amazon Glacier Access Key",
            AwsSecretKey = "Amazon Glacier Secret Key",
            AwsRegionName = "Amazon Glacier Region Name",
            VaultName = "john-glacier",
            RemoteFolderName = "john/backups"
        },
    
        //Google Cloud Backup settings
        GoogleCloudSettings = new GoogleCloudSettings
        {
            BucketName = "Google Cloud Bucket",
            RemoteFolderName = "BackupFolder",
            GoogleCredentialsJson = "GoogleCredentialsJson"
        }
    };
    
    var result = await store.Maintenance.Server.SendAsync(new PutServerWideBackupConfigurationOperation(putConfiguration));
    var serverWideConfiguration = await store.Maintenance.Server.SendAsync(new GetServerWideBackupConfigurationOperation(result.Name));

Initiate Immediate Backup Execution

The Backup task is executed periodically on its predefined schedule.
If needed, it can also be executed immediately.

  • To execute an existing backup task immediately, use the StartBackupOperation method.

    //Create a new backup task
    var operation = new UpdatePeriodicBackupOperation(config);
    var result = await docStore.Maintenance.SendAsync(operation);
    
    //Run the backup task immediately
    await docStore.Maintenance.SendAsync(new StartBackupOperation(true, result.TaskId));
    • Definition:

      public StartBackupOperation(bool isFullBackup, long taskId)
    • Parameters:

      Parameter Type Functionality
      isFullBackup bool true: full-backup
      false: incremental-backup
      taskId long The existing backup task ID
  • To verify the execution results, use the GetPeriodicBackupStatusOperation method.

    //Pass the the ongoing backup task ID to GetPeriodicBackupStatusOperation  
    var backupStatus = new GetPeriodicBackupStatusOperation(result.TaskId);
    • Return Value:
      The PeriodicBackupStatus object returned from GetPeriodicBackupStatusOperation is filled with the previously configured backup parameters and with the execution results.
      public class PeriodicBackupStatus : IDatabaseTaskStatus
      {
          public long TaskId { get; set; }
          public BackupType BackupType { get; set; }
          public bool IsFull { get; set; }
          public string NodeTag { get; set; }
          public DateTime? LastFullBackup { get; set; }
          public DateTime? LastIncrementalBackup { get; set; }
          public DateTime? LastFullBackupInternal { get; set; }
          public DateTime? LastIncrementalBackupInternal { get; set; }
          public LocalBackup LocalBackup { get; set; }
          public UploadToS3 UploadToS3;
          public UploadToGlacier UploadToGlacier;
          public UploadToAzure UploadToAzure;
          public UploadToFtp UploadToFtp;
          public long? LastEtag { get; set; }
          public LastRaftIndex LastRaftIndex { get; set; }
          public string FolderName { get; set; }
          public long? DurationInMs { get; set; }
          public long Version { get; set; }
          public Error Error { get; set; }
          public long? LastOperationId { get; set; }
      }

  • Don't substitute RavenDB's backup procedures with simply copying the database folder yourself.
    The official backup procedure satisfies needs that simply copying the database folder does not. E.g. -

    • A reliable point-in-time freeze of backed up data.
    • An ACIDity of backed-up data, to keep its independence during restoration.
  • Remove old backup files regularly.
    RavenDB does not remove old backup files automatically. As these files continue to aggregate, it is important that you take care of their regular removal.
    You can use services like crontab (a Linux scheduling procedure) to create an old-backups-removal routine.

  • Store backup files in a location other than your database's.
    Note that backup files are always stored in a local folder first (even when the final backup destination is remote).
    Make sure that this local folder is not where your database is stored, as a precaution to keep vacant database storage space.