Securing your Ravens

The internet is often seen as a magical grove where you have the sum total of human knowledge at your fingertips, a land of unlimited opportunities. But if you dig down just a little bit, you'll find that this is not a magic grove, but haunted woods. The internet is a hostile place where very little trust is given or expected. A network packet traveling between two machines is likely to be intercepted, inspected, logged and modified before it reaches its destination.

With rewritten links that download malware and phishing sites that steal your passwords and then your identity, the ideal of a safe-to-browse network is very far from reality. You might not typically notice this because the baseline assumption is, and has been for a while, that the internet is hostile. Websites use certificates and top-of-the-line encryption to identify themselves and hide the content of requests and responses over the wire; digital signatures are sent and validated automatically by your browser and platform.

There is so much encryption around us that we barely notice it. We've reached the stage where we need to use military-grade encryption to upload a lunch selfie to the cloud. We've gotten used to the green lock icon in the address bar and the end-to-end encryption, forward security and authentication employed by common chat platforms. Since the network is assumed to be hostile, only the foolish would go forth without the armor of encryption.

This chapter is not meant to be a thorough discussion of security practices in general. There have been many books published on the subject, guides on topics ranging from how to protect your data and services to how to break into insufficiently protected locations and ransack their contents.

Instead, we are going to focus on understanding the threat models your RavenDB instances face and what kinds of features RavenDB has to deal with such threats. This chapter is all about controlling who can access the data inside your databases, protecting the conversation between client applications and the database from being eavesdropped on or modified as well as protecting the data on disk from being usable if it is stolen.

Running RavenDB in an unsecured mode

You might find this surprising, but it is very common to run RavenDB with no security whatsoever. This typically happens when RavenDB is running on a developer machine. By default, this mode is only allowed as long as RavenDB is listening to either 127.0.0.1, ::1 or localhost. In other words, as long as RavenDB is not listening on the network.¹

In such a mode, none of the security features of RavenDB are accessible. You can neither authenticate users (anyone listening on the network would be able to hijack the connection, after all) nor create or use encrypted databases (anyone can access the server and get the encrypted data, so why bother).

Note that even in development mode, we still strongly recommend that you run in a secured configuration. The RavenDB team has worked hard to make sure that running in a secured mode is easy. Your default mode should be secured.

Even if you're not running development work in a secure network, you'll want to run RavenDB in a secured mode. Security is a core part of the design of RavenDB; we've made it easy for you to run RavenDB in a secured mode, so secured should be your default.

Introduction to transport-level security and authentication

I'm going to assume that you have little-to-no knowledge of security. This is likely false, but it's better to give you information you already have than risk missing something critical. Securing a RavenDB server requires that you do the following:

1. Allow the client to verify that the database server is indeed the one it wants to talk to.
2. Allow the server to verify that the client is valid and decide what access the client should have.
3. Prevent anyone else from eavesdropping on the communication between server and client, hijacking the client credentials, etc.

RavenDB is not the first to have this set of requirements. Instead of rolling our system², RavenDB uses the TLS 1.2 protocol.³ You might not be familiar with the term; a more common use name for this is HTTPS. RavenDB uses TLS 1.2 in the following ways:

• The database server identifies itself to clients with a X509 certificate.
• Clients use the X509 client certificate to identify themselves to the server.
• All traffic between the database and clients is encrypted.

TLS 1.2 is the recommended protocol for secure communication by NIST and PCI 3.1, and in Dec 2017, it was the chosen protocol for close to 90% of all encrypted web traffic worldwide.

If you have previously deployed a website or application using HTTPS, you're already familiar with the key problem that arises from using TLS/HTTPS: certificates. Or more to the point: getting certificates.

A certificate, according to Wikipedia, is "an electronic document used to prove the ownership of a public key", but that might not mean much to a layperson. Basically, it's a way for a service to identify itself in a cryptographically secured manner. There are two types of certificates that interest us: self-signed certificates and certificates signed by a certificate authority (CA).

A self-signed certificate is similar to a name tag at a class reunion, while a certificate signed by a CA is more like an official government ID. Figure 13.1 might make this easier to understand.

Certificates use cryptographic signatures to identify themselves. You can assume that if a connection is using a certificate, then no outside party can listen to the contents of the traffic between the client and the server. But that, as it turns out, is just half the issue. Imagine going to your class reunion, seeing a name tag that says "Your Best Friend From High School", and then whispering a dire secret to the person wearing that name tag.

You know that your secret has only reached that person's ears, but the mere fact that they have a name tag doesn't guarantee that they are the person you think they are. In the real world, you typically use a government-issued ID to verify someone's identity. There's a much higher level of trust given to a driver's license than to a hand-written name tag.

In the same sense, you have a self-signed certificate (which ensures that the connection is private but doesn't guarantee who you are talking to) and you have a CA-signed certificate. A CA does some level of validation before issuing a certificate. And a CA will sign the certificate using its own cryptographic signature. This way, if you have a list of trusted CAs (called root CAs), by default you'll trust that certificates signed by those CAs are valid.

What does that mean, CA validation?

There are different types of certificates that you can use. They range from domain validation (DV), code signing, extended validation (EV) and many more. For the purpose of RavenDB, we only care about DV certificates, which allow you to verify that the server you are talking to is actually the server you think you are talking to.

A DV certificate ensures that when you type https://db.example.org, your client will validate that the certificate served from the server is actually for the right domain and that it is signed by a trusted party (root CA).

It's important to note that this is all a DV certificate does. Most CAs will only check that you have control over a domain before issuing a certificate for it. And if you have the certificate for a site, you can absolutely pretend to be that site. If you have bought the domain example.org, you can get a certificate for db.example.org, but unless you have really good eyes and a good font choice, you wouldn't know that the a character in the domain is not U+0061 but actually U+0251, a completely different letter, resulting in a different domain.

I'm using the Unicode letters because they are so sneaky, but the same rules apply to, say, exampel.org and examp1e.org. The first can mislead you with a typo, and the second with 1 instead of l. At the time of writing, both domains were actually available for purchase, by the way.

Another aspect of security to consider is whom you trust. For example, inside an organization, the admins usually have a root certificate installed on all machines so they can generate certificates for any sites they wish to.

A CA can be one of the global root CAs (such as Comodo, IdenTrust, DigiCert or Let's Encrypt), which are trusted by most browsers and operating systems, or it can be a local CA. For example, your operations team might define a root certificate that is trusted by all your machines and use that root CA to generate more certificates. A real-world parallel for having a local root CA would be accepting employee ID cards as proof of identity inside a company.

Certificate usage inside RavenDB

RavenDB uses certificates for all communication and authentication needs. This has the advantage that everything can talk to RavenDB because HTTPS is so widely supported. Operations teams are also familiar with handling certificates, securely storing them, renewal, revocation, etc.

By default, the client API will trust the server if the server's certificate is a match to the expected URL, the certificate is valid and the certificate is signed by a trusted party. You can override the client API's decision by setting RequestExecutor.ServerCertificateCustomValidationCallback and doing your validation logic there. This is useful if you are using self-signed certificates that aren't trusted by the client, or if you want to verify additional properties on the certificate.

When running in a secured mode, RavenDB requires that the client also authenticates using a X509 certificate. This is called a client certificate, and it allows the client and server to establish a mutually authenticated channel between them. A client trusts the server because the server's certificate is signed by a trusted party. For the server to trust a client's certificate, that certificate needs to be explicitly registered in the cluster. RavenDB does not delegate trust to 3rd parties or the PKI infrastructure.

Reducing optional attack surface

RavenDB's security was designed with a single on/off switch. The unsecured mode for RavenDB uses plain text transport only (HTTP), supports no authentication and cannot use encrypted databases. It can also only talk to other unsecured servers and be part of an unsecured cluster.

On the other hand, the secured mode uses HTTPS and TLS 1.2 for all communications, requires authentication for all operations (using client certificates) and can only talk to other secured servers and be part of a secured cluster. This also allows us to have encrypted databases.

We require secured servers to have authentication in order to prevent operator mistakes from exposing production machines to the world. Sadly, this is a very common occurrence and something that we have strived to make difficult to do by accident.

Client certificates are easy to get: the only thing that needs to trust them is RavenDB, and we can either register certificates with the cluster directly or ask the cluster to generate them for us. Server certificates are usually much more complex to get ahold of because you're dealing with issues of trust. In many organizations, the ability to generate trusted certificates is tightly controlled (and rightly so).

Let's us see what RavenDB does to make secured setup easier.

Setting RavenDB up in a secured mode

Setting up RavenDB in a secured mode is a fairly simple process. You can do it yourself by setting the right configuration values and deploying it directly. In this case, we'll be using the setup wizard to get things going. This simplifies the process of setting up a secured RavenDB instance because the setup wizard takes care of all the details for you.

The first time you start a RavenDB instance, you'll be directed to the setup wizard where you will be asked to make important decisions about how RavenDB will be deployed. You can see what this looks like in Figure 13.2.

You have four setup options available. You can generate a certificate through Let's Encrypt, use your own certificate, set up RavenDB in an unsecured mode or complete the setup process for a cluster node that was started on another instance.

Using Let's Encrypt

One of the reasons TLS and HTTPS aren't used everywhere is that they require certificates. In order for a certificate to be useful, you need it to be trusted, which means that you need to get it from one of the root CAs. Up until recently, that was something that you could only get in a commercial setting, which severly limited HTTPS adoption.

Enter Let's Encrypt. Its mission is to bring HTTPS everywhere by providing free, trusted, fully automated certificates to all. The only requirement of Let's Encrypt is that you must prove that you own the domain for which you are requesting a certificate.

The online documentation does an excellent job of walking you through all the ways you can set up RavenDB, so I'm not going to go into detail on each and every option. Instead, I'm going to guide you through the process of setting up a RavenDB cluster in production mode in as few steps as possible. Right now, my focus is on the security aspects of the solution; we'll talk a lot more about deployment options in later chapters.

First, uncompress the RavenDB server package (available on the "https://ravendb.net/downloads") three times to raven/srv-a, raven/srv-b and raven/srv-c. We are going to run three instances of RavenDB on a single machine, but you can follow the exact same steps to set up RavenDB on multiple machines. Just be sure to get the IP addresses correct and open the relevant ports in the firewalls.

Let's Encrypt is completely optional

It's important to note that this part of the process is entirely optional. Certificate and domain management is the bread and butter of most operations teams, and RavenDB couldn't care less about where you got your certificate from. You can run your RavenDB instance using your own certificates and your own domains. You can use the same setup process with your own certificates by selecting the Provide Your Own Certificate option. Let's Encrypt is available to make it easier for you to set up RavenDB in a secure manner, should you choose to use it.

First, run the raven/srv-a/run.sh (Linux) or raven/srv-a/run.ps1 (Windows) script. This should start the RavenDB instance and open your browser automatically. If this does not happen, look for the "Server available on: <url>" line on the command line and go the the specified URL.

You should see a screen similar to the one in Figure 13.2.

Automatic certificate generation requires a (free) license

Let's Encrypt's free certificate generation is helpful, but to be able to actually generate a certificate, you need to use a domain that you own. In many cases, you are only deploying internally without having any external access.

To alleviate much of the complexity of such a deployment, RavenDB takes it upon itself to handle the details of updating DNS records and generating the certificate. The RavenDB team is managing a set of root-level domains from which it allocates subdomains to users; RavenDB uses these subdomains to create certificates.

These domains looks like <name>.development.run or <name>.ravendb.community. You can provide any <name> you want as long as it hasn't been taken by someone else.⁴

Names are provided on a first come, first served basis, and once they have been given to a user, they are associated with that user permanently. This is to avoid the possibility of two users having certificates to the same domain, which would allow them to capture traffic meant for the other.

You can request a free license from the RavenDB site, use that license during the setup process to register your domain name and then get a certificate to run RavenDB securely. That license is then tied to the domain and can be used later to modify the domain's settings and to generate another certificate.

On the setup screen, select Generate Let's Encrypt Certificate, provide the license that you received over email (you can get a free license by registering on the RavenDB website) and click next. You can see the next screen in Figure 13.3.

The domain selection screen in Figure 13.3 shows the screen where you'll select your domain name. This will also be how you'll typically refer to your cluster. If this is the first time you're running through the process, you'll need to select a name. As long as it hasn't already been taken by someone else, you will be granted the name. If you have already set up RavenDB, you'll be able to select from the domain names you have previous registered. After selecting a domain name, click next.

You'll be faced with the cluster setup, as shown in Figure 13.4. We're setting a local cluster, so set the HTTP Port to 443 and the IP Address to 127.0.1.1 for node A. Note that this is 127.0.1.1, not 127.0.0.1, which you are more used to. In this case, we are relying on the fact that the entire 127.x.x.x IP range has been reserved for the loopback device. Why use 127.0.1.1, then? Because we want to bind to the HTTPS port 443 by default, and it is unlikely that someone has already bound to 127.0.1.1:443.

For production, you obviously won't need to fight some other program for access to a port - I'm looking at you, Skype - but I'm assuming that you will run this exercise on your own machine to start with. You can see what this should look like in Figure 13.4.

If you look at the left portion of Figure 13.4, you'll see a list of nodes (A, B and C). As we have already seen in Chapter 6, we usually use the letters of the alphabet to name the nodes in the cluster. Click on Add node to add node B on your system and give it the IP 127.0.1.2. Then do this again for node C with the IP 127.0.1.3.

If you are deploying to multiple machines, make sure that the IPs are correct, that the ports are available and that the firewall permits the machines to talk to one another. In particular, note that RavenDB actually uses two ports: one for HTTPS (usually 443) and one for TCP (usually 38888). You'll need to open both ports for incoming connections in the firewall. The setup process supports binding to an internal IP and registering the DNS name to an external IP (common in cloud platforms such as AWS, Azure, etc.).

Assuming you went with 127.0.1.1 and a local cluster, you are now ready and can hit the next button. The next little bit should take about a minute or two, as RavenDB contacts Let's Encrypt and completes the DNS challenge to prove that you are in control of the domain you have selected. (In my case, this is raven.development.run; yours will be different and unique to you, of course.) Along with generating the certificate via Let's Encrypt, RavenDB does something else that's very important: it updates the global DNS records to set up the domain name you have chosen with the IPs you have set up.

Troubleshooting the setup

As mentioned previously, this isn't going to be a step-by-step guide to setting up RavenDB. The online documentation goes through that process in great detail. The following quick list should cover the most common issues you may run into during setup, and the online docs have full troubleshooting for anything not covered here.

First, read the error. We have gone to great lengths to make sure that RavenDB errors are clear and concise, telling you what is wrong and, in many cases, how to fix it. Common reasons for setup failure include:

• Another application is already listening on the port.
• On Linux, using port 443 usually requires root privileges. You can either use a higher port (such as 8443) or use sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/Raven.Server to grant the RavenDB binary the right to bind to a low-numbered port.
• Your antivirus / proxy is preventing RavenDB from binding to the network or preventing clients from accessing it.
• Your firewall is blocking incoming network connections, so you cannot connect to RavenDB.
• Your DNS server is aggressively caching and you can't see the DNS updates made by RavenDB to the domain you have chosen. You can set your DNS server to 8.8.8.8 to avoid that, or you can wait about 30 minutes for the usual refresh cycle.

Upon successful completion of the setup, you'll see a message with details about your newly defined cluster. Your browser will have also downloaded a .zip file. This file is critical and should be saved carefully. You can see the contents of this file in Figure 13.5.

The file contains a client certificate with admin-level privileges to the cluster (admin.client.certificate.raven), the license used and a README file with additional setup instructions. In Figure 13.5, you can also see that there are three directories there: A,B and C. These contain the configuration and the server certificate for each of the nodes in the cluster. Depending on your organization's policies, you might need to store them separately. We'll touch on that towards the end of this chapter.

You should install the admin client certificate into your browser. If you are using Chrome, double clicking on the certificate will usually prompt the wizard to begin installation. If it doesn't, go to Settings, Manage certificates, Import... inside Chrome to import the client certificate.

The certificate is generated locally

This info regarding the security of the generated certificate is for the technically inclined. The RavenDB setup process is running locally on your machine. The only involvement of an external party (aside from Let's Encrypt, of course) comes when the api.ravendb.net service registers the Let's Encrypt challenge in the DNS.

The actual certificate sign request (CSR) is generated on your local machine, as is the private key for the certificate. Neither Let's Encrypt nor api.ravendb.net are exposed to your private key, and neither party can recreate the certificate / private key. The certificate is generated locally under your own control, which means the safety of the certificate is in your hands. We'll discuss some security management strategies for your certificates toward the end of this chapter.

Once you've installed the admin client certificate, you can click on the Restart Server button at the end of the setup screen. The browser should then redirect you to the RavenDB management Studio. At this point in the process, you are actually running a secured server, which means you need to authenticate yourself to the server. This is the purpose of installing the admin certificate. You should get a dialog similar to the one shown in Figure 13.6 and you should select the appropriate certificate (if you have more than one).⁵

You should now see the Studio's dashboard. A new feature that is now available is the "Who am I?" notification. Look at the bottom of the window and you'll see a bright green lock icon. Hovering over that icon will reveal what certificate you are using and what permissions are available to you. You can see an example of this in Figure 13.7.

At this point, we've finished setting up a single node. But we haven't actually set up the cluster, have we? In the Studio, go to Manage Server and then Cluster and you'll see that node A is up, but both B and C are showing up red. This is because we haven't even started setting up B or C.

Setting up the rest of the cluster

You can now run the raven/srv-b/run.sh (Linux) or raven/srv-b/run.ps1 (Windows) script. This will start up a new instance of RavenDB, also in initial setup mode. Unlike the when we set up the first node, we don't need to do much this time. At the bottom of the setup window, you'll see Continue cluster setup for new node, as you can see in Figure 13.2. Click on that button and then upload the .zip file that you got from the first node, select node B in the drop-down, and then click next. You can see what this looks like in Figure 13.8.

You should see the success screen again with details about the setup. Click on Restart server at the bottom, and you're done. If you look under Manage Server and Cluster in node A, you'll see that the server has recognized that node B is functional, connected to it, and joined it to the cluster. You can now repeat the process with node C to complete the cluster setup.

This is it: you now have a cluster of three nodes. You have a valid certificate that is broadly trusted and is being used to secure all access to the cluster. Authentication to the cluster is done via the generated admin client certificate (you'll learn more about authentication in RavenDB later in this chapter). The setup process has also updated the DNS system for your cluster, so you can put https://a.raven.development.run to go to node A, or https://b.raven.development.run to go to node B, etc.⁶

Updating the certificate over time

A certificate has a time limit built into it. Let's Encrypt generates certificates for a maximum of 90 days. This means, operationally speaking, that you'll need to replace your certificates. But don't worry: just like RavenDB automatically generates certificates for you during setup, RavenDB will make sure to update your certificates when needed (with plenty of buffer time to avoid any issues).

RavenDB will take care of updating certificates, distributing new certificates to the rest of the cluster and orchestrating certificate replacement (done live, without any downtime) when all the nodes have the new certificate.

If there are any issues during the process, RavenDB will alert the operations team so that you have plenty of time to resolve it. It will also retry on regular basis to avoid being blocked by any transient errors.

In other words: you are pretty much done as far as setup is concerned. Your remaining steps are creating databases and creating certificates so applications and users can connect to the cluster (and not as the cluster admin!). And that's it. Even ongoing maintenance in the form of refreshing the certificates is done for you automatically and transparently.

Security concerns with the automated setup

The automated setup is nice. I'm really happy about how it saves a lot of time and effort. It's a feature aimed primarily at developers running a secure system locally. It is also useful for small applications that don't get a lot of operator attention. However, that has a cost. In order to make this process seamless, RavenDB uses api.ravendb.net to answer the Let's Encrypt DNS challenge and update the DNS records to point to your server.

This service is provided (free of charge) as a courtesy to RavenDB users. For mission critical systems, we recommend that your operations team takes ownership of the process. The actual certificate is generated on your machine, not on a machine owned by us. But given that you don't own the domain name for your cluster, you will have to go through api.ravendb.net to make changes.

We are not in the business of providing hosting, nor do we offer 24/7 support for issues like, "I need to change the server IP address on this node". For production systems, you are encouraged to run RavenDB on your own DNS and use your own certificates.

A security audit of your system will also point out that since we own the domain name, we can generate our own certificates for your cluster domain. We obviously promise not to (and you can check the certificate transparency logs to verify that), but running a critical system is best done on your own systems and under your control.

Certificate transparency

Each certificate generated by a root CA is registered in the public certificate transparency logs. This means that in order for a certificate to be trusted, it must be logged. We have carefully designed the structure of the RavenDB default domain name so we can't generate certificates for your cluster's domain without them becoming immediately visible in the certificate transparency logs.

The technical details are as follows. Your nodes' URLs look like <node-tag>.<cluster-name>.development.run, where RavenDB controls the development.run domain. We can generate a wildcard certificate for *.development.run, but that will not be valid for nested domains. In the case of the raven cluster name, you'll need a certificate that will be valid for URLs such as https://a.raven.development.run. A wildcard certificate on the root domain will not do; only a certificate for *.raven.development.run will be acceptable, which will be very visible in the certificate transparent logs.

Consider the automated setup to be something akin to training wheels on a bicycle: really useful to get you going, but something that you intend to grow out of in time.

The setup process is actually faster when you provide your own certificate. It offers the same user experience without you having to worry about third parties.

Authentication and authorization

We now have a RavenDB cluster set up with encrypted channels of communication and strong authentication methods for clients. We can use the admin client certificate that was generated during setup to authenticate ourselves to the cluster as the admin, which gives us the ability to create additional certificates and manage permissions.

Certificates are not users

It's tempting to consider a certificate as a simple replacement for a user/pass, just another way for a user to authenticate. I would suggest avoiding this line of thinking. Instead, think about granting access to the cluster (and specific databases inside it) on a per-application basis, rather than a per-user basis.

In other words, don't give out certificates to specific users, but to applications (for example, orders.northwind or fulfilment.northwind). This strategy can have profound implications for the way you view the security of the system.

The problem with trying to model security on a per-user basis is that your users aren't (and shouldn't be) accessing and manipulating the data directly. Instead they are going through your application where business rules and validation and authorization decisions are made taking into account full knowledge of your domain and application.

Here's an example I like to use: consider an employee management system and a VacationRequest. Any employee can create a VacationRequest document at any time, modify it freely and put it in the Submit state. At this point, the employee's manager can then Approve or Deny the request. From the point of view of the database, all of the operations (Request, Submit, Approve and Deny) are business operations. As far as the database is concerned, those are all just reads and writes.

Attempting to push business-level authorization decisions (such as "only an employee's manager can Approve a VacationRequest") to the database can lead to a lot of complexity for both the application and the database and can cause data security leaks.

Managing certificates and permissions is done via the Studio in the Manage Server, Certificates page. (Naturally, you can also define certificates through an API, so this is easily automated.) You can see what this looks like in Figure 13.9.

In order for RavenDB to accept a client certificate as valid, it must be registered with the cluster.⁷ Note that as long as the certificate has been registered with RavenDB, RavenDB does not validate the certificate aside from verifying that it has not expired.

To be more specific: RavenDB does not check the certificate chain of trust or the validity period of the certificate that signed the client certificate. RavenDB does not place any requirements at all on the certificate or its properties aside from that it must be usable for Client Authentication. This is important because the generated admin client certificate, for example, is signed by the server certificate that RavenDB is using.

In the case of the Let's Encrypt setup that we ran in the previous section, the generated server certificate is only valid for three months, but the admin client certificate (which is signed by the generated server certificate) is valid for five years. Even if the signing server certificate has expired (and has since been replaced), RavenDB will continue to accept the admin client certificate until the client certificate itself expires.

Plan for certificates to expire

RavenDB generates client certificates with a default validity period of five years. If you need a different duration, you can generate your own client certificates and set different time limits. But regardless of a certificate's validity period, it is important to recognize that certificates can and do expire. A certificate that smoothly runs in production for years could expire one day, causing you to scramble to figure out exactly where it is stored and how to register a new certificate.

RavenDB will warn you about soon-to-be-expired certificates (both cluster certificates and client certificates), but you should have a plan in place to ensure rotation of certificates at regular intervals. It's not only a good security practice, but a good operations practice; it means that your operations team is in the habit of actually replacing certificates so they will be ready to do so when needed.

Creating a new client certificate is easy. Click on the Generate client certificate button, and you should see a screen similar to that shown in Figure 13.10.

In Figure 13.10, you can see the creation of a new certificate, uat.northwind, and that we have granted it read/write access to the test database. Once you click on the Generate button, the new certificate will be generated on the server and registered with the appropriate permissions. The browser will download the newly generated certificate file.

It's important to understand that on the server side, RavenDB only keeps track of the certificate's public information. In other words, RavenDB does not require the private key of the client certificate. If you want to register an existing client certificate with RavenDB, you can do so with Upload client certificate; you don't need to upload the private key, only the public information about the certificate.

Certificates and permissions are cluster-wide

You can make modifications to your trusted certificates and to the permissions assigned to them from any node in the system. Modifying a certificate is a cluster operation and will be reflected on every node in the cluster.

Internally, the only certificate details that RavenDB keeps are the certificate's subject, expiration date and most importantly, its thumbprint, which is used to identify the certificate when a connection comes in.

Security clearances and permissions in RavenDB

A complex security system has many places where it can fail simply due to its complexity. RavenDB's security system was designed to be simple and obvious. There are three clearance levels available to the system, although only two of them are generally applicable for your needs. The commonly used clearances are:

• Cluster Administrator - That's you. The admin cluster certificate that was generated during the initial setup is one example. It has access to everything and can do any operation. This is the equivalent of root in Linux.
• User - This is a certificate that needs to be assigned permissions to specific databases. It is limited to the permissions explicitly given to it.

There is also the Operator clearance level, which is similar to the Cluster Administrator in scope but is limited in terms of the kinds of operations it can perform on the cluster. The Operator clearance level is typically only used in cloud environments where the host manages the cluster and the operator can modify settings within the cluster, but cannot impact the cluster itself.

The Cluster Node clearance alias

If you look closely at Figure 13.9, you'll see that there is another clearance level that I didn't mention: Cluster Node. What's the story there? The Cluster Node is basically an alias for the Cluster Administrator role. It exists because communication between different RavenDB servers also requires authentication (discussed in more detail in the next section) and we wanted a clear distinction between operations initiated by a user and operations initiated directly by a node.

For instance, a node may decide to forward a request to another node, and that is done with the node's own certificate. RavenDB ensures that all validation and authorization checks have been made on the original node before forwarding the request, using the original node's own certificate to authorize it.

Permissions in RavenDB apply only to certificates with the User security clearance level. Permissions apply to a particular database or group of databases and determine whether a certificate has access to that database and, if so, at what level. The available permissions are:

• Database admin - Can read and write documents, query, create static indexes, set up backups, engage ETL processes, define revisions retention and the conflict resolution policies, etc. Basically, this user can perform any operation that is in scope to the single database for which they are the administrator.
• Read/Write - Can read and write documents, query over documents and existing indexes, define and use subscriptions and in general perform any non-maintenance task on the databases to which this user has access. The ability to query also includes the ability of the query optimizer to generate automatic indexes on the fly. It does not include the ability to define new indexes manually. That is reserved to the database administrator.

Typically, a certificate will have access to a single database (its own application database). It might also have access to a shared database for ETL, etc.

RavenDB explicitly does not offer a read only access mode. If you want a user to be able to access your data without being able to modify it, set up external replication (or even RavenDB ETL, to decide exactly what is going to be exposed) and give access to that. This way, the user cannot read or modify your source data; instead they have their own copy of the data that they can work with in isolation.

Reminder: Only database admins can create static indexes

An important note: a static index allows you to write code that will be executed as part of the indexing process. If a user has permission to create a static index, they have the ability to write code that does anything at all.

For this reason, static index creation is limited to database administrators only. But that still means that a sufficiently motivated user with database admin access can access other databases. For the most part, we've assumed that if you've given a database administrator access, you trust that user.

If you require a true security boundary between administrators of different databases, we recommend that you split the databases into independent clusters so that the first administrator only has access to their own cluster.

Because static index creation requires higher security privileges, it is common to use the Read/Write permissions in production and split the static index creation function off into a separate tool that will run with elevated permissions.

Server-to-server authentication

We have talked about how RavenDB uses certificates to identify clients, but what about servers? How do the different servers in the cluster decide whether to trust each other?

The answer is simple. Any time a RavenDB server talks to a remote node, it will identify itself using its local server certificate. We just need to recognize that certificate, and from then on the server can be handled just like any other client.

I lost the keys to the kingdom, what now?

Edge cases are always the hardest things to deal with in development. You need to consider what will happen if you manage to lose all the keys to your security system. What will happen if you lose the client certificate that talks to RavenDB? RavenDB will have no way of recognizing you and will (rightly) reject any requests as unauthorized.

There are two ways to handle this. First, you can use the rvn admin-channel tool, which uses the operating system to confirm that you should have access to the RavenDB process. This tool gives the root user (or Administrator on Windows) a back channel into RavenDB and the ability to run a few commands to get you back on your feet (such as trustClientCert to register a new certificate with Cluster Administrator clearance).

If you have access to the server, you can also get the server's certificate and use that, since RavenDB always trusts clients that connect using the same certificate as the server.

When running in a cluster, we typically use a wildcard certificate (such as *.raven.development.run) or a certificate that contains multiple Subject Alternative Names such as:

• a.raven.development.run
• b.raven.development.run
• c.raven.development.run

This means that all the nodes in the cluster are using the same certificate. That leads to some interesting behavior when a client connects to a server and is authenticating using the same certificate that the server itself is using. When this happens, we consider the connection to be trusted and grant it Cluster Node privileges. There's no need to register the certificate for this to work.

So if you need to connect to the server and don't have a client certificate registered, but you do have access to the certificate that the server is using, you can use the server's certificate and your connection will be trusted without the admin having to take any action.

Putting these two facts together, you can see that when we bring up a new node in the cluster that has the same certificate as the other nodes, the new node intrinsically knows that it can trust the other side. This greatly simplifies the process of setting up a secured cluster.

You can run a RavenDB cluster with a different certificate for each node, but you'll need to register all the certificates in the cluster. That presents a bit of a chicken-and-egg problem because you can't have a cluster until the nodes trust each other. The rvn admin-channel tool solves this issue by providing the trustServerCertificate command that you can run on each of the nodes, instructing each node to trust the others.

Updating the cluster certificate at runtime is a delicate dance. You need to first distribute the new certificate (including the private key) to all the nodes, then register the new certificate as a Cluster Node, then start switching over all new connections to use the new certificate. Different nodes may do that at different times, which is why we need to ensure that the new certificate is registered explicitly.

RavenDB manages this dance for you. When you are using Let's Encrypt, RavenDB will automatically refresh the cluster certificate about a month before it is due to expire. Then the actual certificate replacement will take place once all the nodes are confirmed to have the new certificate. You can also trigger this process manually with your own certificate using the Replace cluster certificate button, as shown in Figure 13.9.

Authentication between clusters

We've talked so far about certificates and trust within a single cluster. We also need to talk about how certificates and authentication work when you're running multiple clusters that need to communicate securely among themselves. Whenever a RavenDB server communicates with the outside world, it uses its own server certificate as the client certificate so that it will be properly identified on the other side.

Within the same cluster, the nodes know that they can trust the connection. But when we are dealing with separate clusters, we'll need to explicitly tell the remote cluster that it can trust our cluster.

In Figure 13.9, you can see the Export cluster certificates button. Clicking this button will download a .pfx file containing the certificates of all the nodes in the cluster sans their private keys (everything that has a Cluster Node clearance level). You can take that .pfx file to a different cluster, register it using Upload client certificate and give it the appropriate permissions for your needs.

Cross-cluster authentication is often used as part of external replication and RavenDB ETL processes. Note that you don't have to specify credentials as part of the ETL connection string or when you're setting up external replication. The credentials are implicit in the cluster itself.

Handling automatic certificate refresh in cross-cluster authentication

You will encounter a wrinkle if you're using cross-cluster authentication and also automatically updating your certificates. In this case, the remote cluster will not be familiar with the new certificates, and you'll need to manually add the new certificates to the remote cluster.

Externalizing certificate management

Certificates are of high importance. If you have the certificate for a domain, you can eavesdrop on communication to that domain, modify responses, pretend to be that domain and in general do a whole bunch of pretty nasty stuff from a security point of view.

In many organizations, there are strict policies regarding how you are expected to manage certificates, due to their sensitive nature. These policies range from, "Let's not put them on publicly shared folders" to, "Only store certificates on hardware security modules with level 4 certification".

There is an astounding variety of policies regarding certificate management, and RavenDB makes no attempt to accommodate all of them. Instead, we provide the following options:

• Store the certificate on the file system (potentially secured via file system permissions).
• Use your own policy for certificate storage and provide a script that RavenDB will call to fetch the certificate as needed.

In the Let's Encrypt scenario we've explored, the actual certificate is located in the server's folder with the Security.Certificate.Path configuration parameter pointing to the file. The certificate file is not encrypted on the disk.

Default security of the certificate

We could have added additional security measures to the Let's Encrypt certificate, such as storing it in an encrypted form on the file system. But then we would have needed to store the certificate password somewhere. RavenDB does support encrypted certificates and you can provide passwords using the Security.Certificate.Password configuration option. But this just shifts the problem to how to secure the certificate password.

Given that this certificate is routinely refreshed by RavenDB and can be regenerated at will using the license file (which is also typically sitting right next to the certificate), there is no need to put overdue efforts into securing it. If your threat model calls for securing the certificate from an attacker that has physical access to the files on disk, you should not be using the Let's Encrypt setup mode. Instead, you should use more advanced methods to provide your own certificate, as we'll soon explore.

If you need a more sophisticated setup than a certificate file on disk (with an optional password), you need to tell RavenDB how to get the certificate yourself. This can be done by two configuration options: Security.Certificate.Exec and Security.Certificate.Exec.Arguments.

You can see an example of how this can be done in Listing 13.1.

Listing 13.1 Partial configuration of shelling out to a user defined method for obtaining the certificate

{
    "ServerUrl": "https://0.0.0.0",
    "PublicServerUrl": "https://a.raven.development.run",
    "Security.Certificate.Exec": "powershell",
    "Security.Certificate.Exec.Arguments":  
        "get-cert-by-id.ps1 90F4BC16CA5E5CB535A6CD8DD78CBD3E88FC6FEA"
}

The idea is that instead of RavenDB having to support all the various options, policies and rules around the storing of certificates, you tell RavenDB that whenever it wants to get a certificate, it just needs to run the specified process and read the certificate from the standard output. Listing 13.2 shows the implementation of the get-cert-by-id.ps1.

Listing 13.2 PowerShell script to get a certificate by id from the current user's key store

try
{
    $thumbprint = $args[0]
    $cert = gci "cert:\CurrentUser\my\$thumbprint"
    $exportedCertBinary = $cert.Export("Pfx")
    $stdout = [System.Console]::OpenStandardOutput()
    $stdout.Write($exportedCertBinary, 0, 	
        $exportedCertBinary.Length)
}
catch
{
    write-error $_.Exception
    exit 3
}

The output of the script is the .pfx binary data, sent to the standard output. You can report errors using the standard error, which will be included in the RavenDB error message. In this way, you can integrate RavenDB into your existing policies with very little hassle.

For example, if you wanted to get a certificate from Azure Vault, you'd use: Get-AzureKeyVaultSecret cmdlet. You're also not limited to just using PowerShell; you're free to shell out to any process or method you'd like.

Refreshing the certificates manually

If you can tolerate the certificate being store on disk, then refreshing a certificate is easy. Either RavenDB will handle this for you on its own if you are using the Let's Encrypt mode, or you can trigger the refresh manually by using Replace cluster certificate. But how do you handle a server certificate refresh if your certificate cannot be stored on disk?

RavenDB will periodically (once an hour) reload the certificate from store (either the file on disk or using the executable command you specified) and compare it to the certificate that is already in memory. If the certificate has been updated, RavenDB will begin to use the new certificate instead.

But before we can replace the certificate, we need to register the new certificate in the cluster. You should only replace the certificate once that registration is complete. The reason being, you don't want some of the servers to load the new certificate, immediately start using it and then not be able to talk to the other nodes that haven't noticed the new certificate yet.

This two-step approach - first registering the certificate in the cluster as a known entity, then replacing it and letting each node pick it up in its own time - ensures that there will be no interruption in service while a server certificate is being updated.

Auditing accesses to RavenDB

Controlling who can get into RavenDB and what they can access is just one part of the security story. We also need to be able to keep track of who has connected to the system and when. RavenDB supports the process of auditing accesses at the level of database connection.

Configuring auditing is easy. You need to set the Security.AuditLog.FolderPath and optionally the Security.AuditLog.RetentionTimeInHours (which defaults to a year). Once these values are set, RavenDB will record the following events in a dedicated audit log:

• A connection was made to RavenDB, including what certificate was used and what privileges it was granted;
• A connection was rejected by RavenDB as invalid;
• A database was created or removed (from all nodes or a single node);
• An index was created or removed.

The audit log folder will contain the audit entries and is usually loaded into centralized audit and analysis by dedicated tools. RavenDB does nothing with the audit logs except write to them.

It is important to understand that the audit logs are also local (local to the originator node). That is, if we have a database residing on node C that is removed by a command originating on node B, the audit entry will be in the audit log of node B, not node C.

Another important consideration is that RavenDB writes connections to the audit log, not requests. This is done for performance and manageability reasons; otherwise, you'd have extremely large and unwieldy audit logs. With HTTP 1.1, a single TCP connection is used for many different requests. The only items you'll find in the audit log are the time of the TCP connection, the certificate being used and the level of access granted to this certificate at the time of the connection.

If you require more detailed logs (at the level of individual HTTP requests), you can use a proxy in front of RavenDB that will log the appropriate requests as they are being made.

Summary

This topic ended up taking significantly more time than I intended it to. I'm talking both about the time it took to write the chapter (and the length of the topic) and the time it took us to properly implement authentication and authorization in RavenDB.

Security is a big topic, one that needs to be taken seriously and handled with care. Recognizing this, we ran RavenDB through an external security audit in January 2018 and made the resulting report public.⁸

We started this chapter with a brief review of the state of the network under the baseline assumption that the network is hostile and we should take explicit steps to protect ourselves, our systems and our data. RavenDB handles this requirement by using TLS 1.2 for communications and using X509 for mutual authentication between clients and servers. We touched briefly on the notion of trust and how that is conferred using digital signatures from trusted parties, the root CAs (either the global list or specific CAs for your organization). We then moved to discussed what this means for deploying RavenDB.

One of the more common causes of security breaches is security systems that are so complex that they are either never enabled or enabled with poor defaults that offer nothing but a security theater. RavenDB's system was carefully designed to be both simple and secure.

Instead of offering a lot of options and configurations and tweaks, RavenDB has a simple on/off mode for security. If you don't need security (for example, if you're working locally on a development machine), you can choose to run in an unsecured mode. In that case, you will use HTTP and unencrypted TCP, have no authentication, be unable to use encrypted databases and be unable to be part of secure clusters or replication.

The secured mode, on the other hand, uses HTTPS and TCP encrypted using TLS 1.2 for all remote communications, requires authentication of clients using X509 client certificates, can use encrypted databases and can only be part of secure clusters and replications.

The automated setup mode via Let's Encrypt takes you through the entire process of setting up a RavenDB cluster and takes upon itself the task of setting up everything. This includes generating a certificate, setting up DNS entries so your servers will have nice URLs, generating a configuration package to complete the setup of the other nodes in the cluster and everything else that is required to properly set things up.

We also talked in this chapter about how the automated setup, while very convenient, represents a loss of control for your system (the DNS entries are not owned by your organization, for example) and is most suitable for development and small applications. If you're setting up everything yourself, you can use the same setup process; the only difference will be that you'll be the one to take care of the DNS entries and the certificate generation.

Next, we dived into authentication and authorization and how they work inside RavenDB. We learned how RavenDB maps certificates internally handling security clearances and permissions for specific databases, and that the server's own certificate can also be used as a client certificate (with admin privileges). This fact makes it very easy for us to run clusters since all the nodes in the cluster share the same certificate.

We then talked about server-to-server authentication both within the same cluster, where RavenDB manages this almost entirely for you, and between clusters, where it's the administrator's job to ensure that the clusters trust each other and have the right permissions for the tasks that they need to do.

We talked about how to actually manage certificates, from just storing the plain text certificate on the disk to using hardware security modules or relying on external secret services (such as the machine key store or Azure Vault). RavenDB allows you to define exactly how it will fetch certificates, though remember that this approach makes certificate refresh a somewhat manual process. We also talked about what is required from the administrator to enable online certificate refresh with no downtime, as well as the reasoning behind these steps.

Finally, we discussed how you can audit access to your RavenDB cluster by configuring RavenDB to log all connections and the security clearance that these connections are granted.

In this chapter, we talked about security in transit: encrypting data as it goes through the network, authenticating who we are talking to and authorizing that users and systems are allowed to perform the operations they've requested. In the next chapter, we are going to talk about encryption at rest: how we can ensure that our data is safe, even if our hard disk is not.