Backup

Logical-Backup

• Data, index definitions, and ongoing tasks are backed-up in compressed JSON files.

• During the restoration, RavenDB -

  • Re-inserts all data into the database.
  • Inserts the saved index definitions. To save space, Logical Backup stores index definitions only.
    After restoration, the dataset is scanned and indexed according to the definitions.

• See backup contents.

• Restoration time is, therefore, slower than when restoring from a Snapshot.

• The backup file size is significantly smaller than that of a Snapshot.

• In addition to full data backup, Logical Backups can be defined as incremental, saving any changes made since the previous backup.

• The following code sample defines a full-backup task that would be executed every 3 hours:
  

  var config = new PeriodicBackupConfiguration
  {
      LocalSettings = new LocalSettings
      {
          // Local path for storing the backup
          FolderPath = @"E:\RavenBackups"
      },
  
      // Full Backup period (Cron expression for a 3-hours period)
      FullBackupFrequency = "0 */3 * * *",
  
      // Set backup type to Logical-Backup
      BackupType = BackupType.Backup,
  
      // Task Name
      Name = "fullBackupTask",
  };
  var operation = new UpdatePeriodicBackupOperation(config);
  var result = await docStore.Maintenance.SendAsync(operation);

  Note the usage of Cron scheduling when setting backup frequency.

Snapshot

• A Snapshot is a compressed binary duplication of the full database structure. This includes the data file and the journals at a given point in time.
  Therefore it includes fully built indexes and ongoing tasks.
  See file structure for more info.

• Snapshot backups are available only for Enterprise subscribers.

• During restoration -

  • Re-inserting data into the database is not required.
  • Re-indexing is not required.

• See backup contents.

• Restoration is typically faster than that of a logical backup.

• Snapshot size is typically larger than that of a logical backup.

• If Incremental backups are created for a Snapshot-type backup:

  • The first backup will be a full Snapshot.
  • The following backups will be Incremental.
  • Incremental backups have different storage contents than Snapshots.

• Code Sample:
  

  // Set backup type to Snapshot
  BackupType = BackupType.Snapshot,

Basic Comparison Between a Logical-Backup and a Snapshot:

As described in the overview, a backup task can create full and incremental backups.

• A Backup Task can be defined to create either a full data backup or an incremental backup.
  In both cases, the backup task adds a single new backup file to the backup folder each time it runs, leaving the existing backup files untouched.

Full-Backup

• File Format
  A full-backup is a compressed JSON file if it is a logical backup, or a compressed binary file if it is a snapshot.

• Task Ownership
  There are no preliminary conditions for creating a full-backup. Any node can perform this task.

• To run a full-backup
  Set FullBackupFrequency.

  // A full-backup will run every 6-hours (Cron expression)
  FullBackupFrequency = "0 */6 * * *",

Incremental-Backup

• File Format and Notes About Contents

  • An incremental-backup file is always in JSON format. It is so even when the full-backup it is associated with is a binary snapshot.
  • An incremental backup stores index definitions (not full indexes).
    After the backup is restored, the dataset is re-indexed according to the index definitions.

    This initial re-indexing can be time-consuming on large datasets.

  • An incremental backup doesn't store change vectors.

• Task Ownership
  The ownership of an incremental-backup task is granted dynamically by the cluster.
  An incremental-backup can be executed only by the same node that currently owns the backup task.
  A node can run an incremental-backup, only after running full-backup at least once.

• To run an incremental-backup
  Set IncrementalBackupFrequency.

// An incremental-backup will run every 20 minutes (Cron expression)
IncrementalBackupFrequency = "*/20 * * * *",

• Backups can be made locally, as well as to a set of remote locations including -

  • A network path
  • An FTP/SFTP target
  • Azure Storage
  • Amazon S3
  • Amazon Glacier
  • Google Cloud

• RavenDB will store data in a local folder first, and transfer it to the remote destination from the local one.

  • If a local folder hasn't been specified, RavenDB will use the temp folder defined in its Storage.TempPath setting.
    If Storage.TempPath is not defined, the temporary files will be created at the same location as the data file.
    In either case, the folder will be used as temporary storage and the local files deleted from it when the transfer is completed.
  • If a local folder has been specified, RavenDB will use it both for the transfer and as its permanent local backup location.

• Local and Remote Destinations Settings Code Sample:
  

  var config = new PeriodicBackupConfiguration
  {
      LocalSettings = new LocalSettings
      {
          FolderPath = @"E:\RavenBackups"
      },
  
      // FTP Backup settings
      FtpSettings = new FtpSettings
      {
          Url = "192.168.10.4:8080",
          UserName = "John",
          Password = "JohnDoe38"
      },
  
      // Azure Backup settings
      AzureSettings = new AzureSettings
      {
          StorageContainer = "storageContainer",
          RemoteFolderName = "remoteFolder",
          AccountName = "JohnAccount",
          AccountKey = "key"
      },
  
      // Amazon S3 bucket settings.
      S3Settings = new S3Settings
      {
          AwsAccessKey = "your access key here",
          AwsSecretKey = "your secret key here",
          AwsRegionName = "OPTIONAL",
          BucketName = "john-bucket"
      },
  
      // Amazon Glacier settings.
      GlacierSettings = new GlacierSettings
      {
          AwsAccessKey = "your access key here",
          AwsSecretKey = "your secret key here",
          AwsRegionName = "OPTIONAL",
          VaultName = "john-glacier",
          RemoteFolderName = "john/backups"
      },
  
      // Google Cloud Backup settings
      GoogleCloudSettings = new GoogleCloudSettings
      {
          BucketName = "RavenBucket",
          RemoteFolderName = "BackupFolder",
          GoogleCredentialsJson = "GoogleCredentialsJson"
      }
  
  };
  var operation = new UpdatePeriodicBackupOperation(config);
  var result = await docStore.Maintenance.SendAsync(operation);

{
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "VisualEditor0",
                    "Effect": "Allow",
                    "Action": "s3:PutObject",
                    "Resource": "arn:aws:s3:::BUCKET_NAME/*"
                },
                {
                    "Sid": "VisualEditor1",
                    "Effect": "Allow",
                    "Action": [
                        "s3:ListBucket",
                        "s3:GetBucketAcl",
                        "s3:GetBucketLocation"
                    ],
                    "Resource": "arn:aws:s3:::BUCKET_NAME"
                }
            ]
        }

By default, backups are stored indefinitely. The backup retention policy sets a retention period, at the end of which backups are deleted. Deletion occurs during the next scheduled backup task after the end of the retention period.

Full backups and their corresponding incremental backups are deleted together. Before a full backup can be deleted, all of its incremental backups must be older than the retention period as well.

The retention policy is a property of PeriodicBackupConfiguration:

public class RetentionPolicy
{
    public bool Disabled { get; set; }
    public TimeSpan? MinimumBackupAgeToKeep { get; set; }
}

Example

var config = new PeriodicBackupConfiguration
{
    RetentionPolicy = new RetentionPolicy
    {
        Disabled = false, // False is the default value
        MinimumBackupAgeToKeep = TimeSpan.FromDays(100)
    }
};

You can create a Server-Wide Backup task to back-up all the databases in your cluster at a scheduled time.
Individual databases can be excluded from the backup. Learn more in Studio: Server-Wide Backup.

Backups can be made locally, as well as to a set of remote locations.

Examples

A server-wide backup configuration that sets multiple destinations:

var putConfiguration = new ServerWideBackupConfiguration
{
    Disabled = true,
    FullBackupFrequency = "0 2 * * 0",
    IncrementalBackupFrequency = "0 2 * * 1",

    //Backups are stored in this folder first, and sent from it to remote destinations (if defined).
    LocalSettings = new LocalSettings
    {
        FolderPath = "localFolderPath"
    },

    //FTP settings
    FtpSettings = new FtpSettings
    {
        Url = "ftps://localhost/john/backups"
    },

    //Microsoft Azure settings.
    AzureSettings = new AzureSettings
    {
        AccountKey = "Azure Account Key",
        AccountName = "Azure Account Name",
        RemoteFolderName = "john/backups"
    },

    //Amazon S3 bucket settings.
    S3Settings = new S3Settings
    {
        AwsAccessKey = "Amazon S3 Access Key",
        AwsSecretKey = "Amazon S3 Secret Key",
        AwsRegionName = "Amazon S3 Region Name",
        BucketName = "john-bucket",
        RemoteFolderName = "john/backups"
    },

    //Amazon Glacier settings.
    GlacierSettings = new GlacierSettings
    {
        AwsAccessKey = "Amazon Glacier Access Key",
        AwsSecretKey = "Amazon Glacier Secret Key",
        AwsRegionName = "Amazon Glacier Region Name",
        VaultName = "john-glacier",
        RemoteFolderName = "john/backups"
    },

    //Google Cloud Backup settings
    GoogleCloudSettings = new GoogleCloudSettings
    {
        BucketName = "Google Cloud Bucket",
        RemoteFolderName = "BackupFolder",
        GoogleCredentialsJson = "GoogleCredentialsJson"
    }
};

var result = await store.Maintenance.Server.SendAsync(new PutServerWideBackupConfigurationOperation(putConfiguration));
var serverWideConfiguration = await store.Maintenance.Server.SendAsync(new GetServerWideBackupConfigurationOperation(result.Name));

A server-wide backup configuration that excludes several databases:

var DBExcludeConfiguration = new ServerWideBackupConfiguration
{
    Disabled = true,
    FullBackupFrequency = "0 2 * * 0",
    LocalSettings = new LocalSettings
    {
        FolderPath = "localFolderPath"
    },
    ExcludedDatabases = new []
    {
        "DB1",
        "DB2",
        "DB5",
        "NorthWind",
        "DB2_Jun_2018_Backup"
    }
};

var result = await store.Maintenance.Server.SendAsync(new PutServerWideBackupConfigurationOperation(DBExcludeConfiguration));

The Backup task is executed periodically on its predefined schedule.
If needed, it can also be executed immediately.

• To execute an existing backup task immediately, use the StartBackupOperation method.
  

  // Create a new backup task
  var operation = new UpdatePeriodicBackupOperation(config);
  var result = await docStore.Maintenance.SendAsync(operation);
  
  // Run the backup task immediately
  await docStore.Maintenance.SendAsync(new StartBackupOperation(true, result.TaskId));

  • Definition:

    public StartBackupOperation(bool isFullBackup, long taskId)

  • Parameters:

    Parameter	Type	Functionality
    isFullBackup	bool	true: full-backup false: incremental-backup
    taskId	long	The existing backup task ID

• To verify the execution results, use the GetPeriodicBackupStatusOperation method.
  

  // Pass the the ongoing backup task ID to GetPeriodicBackupStatusOperation  
  var backupStatus = new GetPeriodicBackupStatusOperation(result.TaskId);

  • Return Value:
    The PeriodicBackupStatus object returned from GetPeriodicBackupStatusOperation is filled with the previously configured backup parameters and with the execution results.
    

    public class PeriodicBackupStatus : IDatabaseTaskStatus
    {
        public long TaskId { get; set; }
        public BackupType BackupType { get; set; }
        public bool IsFull { get; set; }
        public string NodeTag { get; set; }
        public DateTime? LastFullBackup { get; set; }
        public DateTime? LastIncrementalBackup { get; set; }
        public DateTime? LastFullBackupInternal { get; set; }
        public DateTime? LastIncrementalBackupInternal { get; set; }
        public LocalBackup LocalBackup { get; set; }
        public UploadToS3 UploadToS3;
        public UploadToGlacier UploadToGlacier;
        public UploadToAzure UploadToAzure;
        public UploadToFtp UploadToFtp;
        public long? LastEtag { get; set; }
        public LastRaftIndex LastRaftIndex { get; set; }
        public string FolderName { get; set; }
        public long? DurationInMs { get; set; }
        public long Version { get; set; }
        public Error Error { get; set; }
        public long? LastOperationId { get; set; }
    }

The execution of a periodic backup task can be delayed for a given time period via Studio or using the DelayBackupOperation store operation.

• Definition:
  

  public DelayBackupOperation(long runningBackupTaskId, TimeSpan duration)

• Parameters:

  Parameter	Type	Functionality
  runningBackupTaskId	long	Backup task ID
  duration	TimeSpan	Delay Duration

• Example:
  To delay the execution of a running backup task pass DelayBackupOperation the task's ID and the delay duration.
  

  // Get backup operation info
  var taskBackupInfo = await docStore.Maintenance.SendAsync(
          new GetOngoingTaskInfoOperation(taskId, OngoingTaskType.Backup)) as OngoingTaskBackup;
  
  // Set delay duration to 10 minutes from now
  var delayDuration = TimeSpan.FromMinutes(10);
  var delayUntil = DateTime.Now + delayDuration;
  
  // Delay backup operation
  await docStore.Maintenance.SendAsync(
          new DelayBackupOperation(taskBackupInfo.OnGoingBackup.RunningBackupTaskId, delayDuration));