Customizing Data Files Locations



Why store data on different devices

  • A file or directory can be redirected to a different storage location according to its speed, durability, etc.
  • This allows organizing the files based on their usage pattern and the performance of the different devices you own.
    i.e. slow devices may cause a slow down when reading from Raven.voron.
  • Voron components operate concurrently.
    Storing them to different locations can help avoid traffic jams and achieve better concurrency.

Configuring temp files location

  • Databases temporary files
    By default, all databases' temporary files are written to the Temp folder under each Database directory.
    Customize the files path by setting configuration option Storage.TempPath in your settings.json file.

  • Indexes temporary files
    By default, all indexes' temporary files are written to the Temp folder under each Index directory.
    Customize the files path by setting configuration option Indexing.TempPath in your settings.json file.

  • Backup temporary files
    By default, backup temporary files are written under the Database directory or under Storage.TempPath if defined.
    Customize the files path by setting configuration option Backup.TempPath in your settings.json file.

Configuring data files location

  • You can store RavenDB data files in any directory of your choice.
    This is done by defining junction points (Windows) or mount points (Linux).

  • If data already exists in the directory, and you want to define the junction / mount point,
    you need to create a backup of the data first and copy it back into the directory after executing the command.

  • The database must be offline when moving the database folder itself to a new point.

Example - Moving Journals

A common practice is to store the journals on a very fast drive to achieve better write performance. The following command will point the Journals directory of the Northwind database to a path on a different drive.

# Windows:
C:\RavenDB\Server\RavenData\Databases\Northwind>mklink /J Journals E:\Journals\Northwind

# Linux:
ln -s /mnt/FastDrive/Databases/Northwind/Journals ~/RavenDB/Server/RavenData/Databases/Northwind/Journals

Example - Moving Indexes

To store the data of all indexes of the Northwind database in the custom location, you can use the following command to to point the Indexes directory to a new location:

# Windows:
C:\RavenDB\Server\RavenData\Databases\Northwind>mklink /J Indexes D:\Indexes\Northwind

# Linux:
ln -s /mnt/FastDrive/Databases/Northwind/Indexes ~/RavenDB/Server/RavenData/Databases/Northwind/Indexes

Automate storage definitions

  • To help automate the process, we have added the on directory initialize configuration option.
    Whenever RavenDB creates a directory, it will invoke the process that is defined within that configuration.

  • The process is called just before the directory is created.
    This allows you to create a script with your own logic, defining junction/mount points as needed.


  • RavenDB will invoke the process with the following params:

    • Params passed by the user:

    • Params passed by RavenDB:

      • The environment type (System, Database, Index, Configuration, Compaction)
      • The database name
      • Path of the DataDir directory
      • Path of the Temp directory
      • Path of the Journals directory

Script example - Basic usage

The following is a very simple PowerShell script example, here only to show basic usage.
The script is Not modifying any file location, it will only print out the value of the script params into a text file.

# script.ps1

param([string]$userArg ,[string]$type, [string]$name, [string]$dataPath, [string]$tempPath, [string]$journalPath)
Add-Content $userArg "$type $name $dataPath $tempPath $journalPath\r\n"
exit 0

The output file outFile.txt is supplied as a user argument.
Add the script path and its user arguments to the settings.json file as follows:

{
    "Setup.Mode": "None",
    "ServerUrl": "http://127.0.0.1:8080",
    "License.Eula.Accepted": true,
    "Storage.OnDirectoryInitialize.Exec" :"powershell",
    "Storage.OnDirectoryInitialize.Exec.Arguments" :"c:\\scripts\\script.ps1 c:\\scripts\\outFile.txt"
}

When launching the server and creating the Northwind database with the Northwind sample data, the script is invoked every time a directory is created.
Each line in outFile.txt shows the values passed to the script when it was called.

{
System System C:\RavenDB\Server\System C:\RavenDB\Server\System\Temp C:\RavenDB\Server\System\Journals\r\n
Configuration Northwind C:\RavenDB\Server\Databases\Northwind\Configuration C:\RavenDB\Server\Databases\Northwind\Configuration\Temp C:\RavenDB\Server\Databases\Northwind\Configuration\Journals\r\n
Database Northwind C:\RavenDB\Server\Databases\Northwind C:\RavenDB\Server\Databases\Northwind\Temp C:\RavenDB\Server\Databases\Northwind\Journals\r\n
Index Northwind C:\RavenDB\Server\Databases\Northwind\Indexes\Orders_ByCompany C:\RavenDB\Server\Databases\Northwind\Indexes\Orders_ByCompany\Temp C:\RavenDB\Server\Databases\Northwind\Indexes\Orders_ByCompany\Journals\r\n
Index Northwind C:\RavenDB\Server\Databases\Northwind\Indexes\Product_Search C:\RavenDB\Server\Databases\Northwind\Indexes\Product_Search\Temp C:\RavenDB\Server\Databases\Northwind\Indexes\Product_Search\Journals\r\n
// ... + more lines per index folder created
}

Script example - Point Indexes to different location

The following bash script example will point the Indexes data to a new location.

If the environment type is other than 'Database' the script will exit.
Else, the script will create a soft link for the Indexes directory.

After the script is run, the Indexes link will reside under the Database directory (See Directory Structure).
The link will point to the new location set by variable $INDEXES_TARGET_DIR_NAME where all indexes' data will be written.

#!/bin/bash
# bash ./your-script USER_ARGS Database DB_NAME BASE_PATH TEMP_PATH JOURNALS_PATH

# Use directory names as defined on your machine
RDB_DATA_DIR="/var/lib/ravendb/data"
INDEXES_TARGET_DIR="/mnt/ravendb-indexes"

DIR_TYPE="$1"
DB_NAME="$2"

if [ "$DIR_TYPE" != 'Database' ]; then
exit 0
fi

INDEXES_SOURCE_DIR_NAME="${RDB_DATA_DIR}/Databases/${DB_NAME}/Indexes"
INDEXES_TARGET_DIR_NAME="${INDEXES_TARGET_DIR}/${DB_NAME}/Indexes"


if [ -d "$INDEXES_SOURCE_DIR_NAME" ] && [ ! -L "$INDEXES_SOURCE_DIR_NAME" ]; then
# If Indexes directory exists then exit - need manual handling
echo "FATAL: Directory $INDEXES_SOURCE_DIR_NAME already exists."
exit 1
fi

if [ -L "$INDEXES_SOURCE_DIR_NAME" ]; then
exit 0
fi

mkdir -p "$INDEXES_TARGET_DIR_NAME"

ln -s "$INDEXES_TARGET_DIR_NAME" "$INDEXES_SOURCE_DIR_NAME"

Add the script to the settings.json file as follows:

{
"Setup.Mode": "None",
"ServerUrl": "http://127.0.0.1:8080",
"License.Eula.Accepted": true,
"Storage.OnDirectoryInitialize.Exec" :"bash",
"Storage.OnDirectoryInitialize.Exec.Arguments" :"/scripts/your-script.sh"
}