Security: Common Errors & FAQ
-
This article explains some of the common security configuration errors and how to handle them.
-
In this page
Setup Wizard Issues
- Server cannot bind to the provided private IP address
- Ports are blocked by the firewall
- DNS is cached locally
- Long DNS propagation time
- If I already have the Zip file, can I avoid repeating the setup process?
Server cannot bind to the provided private IP address
If the IP/port is not accessible on your machine, you'll get the following error.
System.InvalidOperationException: Setting up RavenDB in Let's Encrypt security mode failed. --->
System.InvalidOperationException: Validation failed. --->
System.InvalidOperationException: Failed to simulate running the server with the supplied settings using: https://a.example.ravendb.community:4433 --->
System.InvalidOperationException: Failed to start webhost on node 'A'. The specified ip address might not be reachable due to network issues.
It can happen if the ip is external (behind a firewall, docker). If this is the case, try going back to the previous screen and add the same ip as an external ip.
Settings file: D:\temp\RavenDB-4.0.0-windows-x64\Server\settings.json.
IP addresses: 10.0.0.65:4433.
---> Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvException: Error -4092 EACCES permission denied
This can be caused by two different reasons:
- Your private IP address is not reachable inside the machine or you provided the wrong IP/port.
- You are running behind a firewall (VM, docker...) and accidentally provided the external IP address during setup.
Make sure you provide the private IP address in the "IP Address / Hostname" field as seen in this example.
Ports are blocked by the firewall
When configuring a VM in Azure, AWS or any other provider, you should define firewall rules to allow both the HTTP and TCP ports you have chosen during setup. This should be done both inside the VM operating system and in the web dashboard or management console.
If ports are blocked you'll get the following error.
Setting up RavenDB in Let's Encrypt security mode failed.
System.InvalidOperationException: Setting up RavenDB in Let's Encrypt security mode failed. --->
System.InvalidOperationException: Validation failed. --->
System.InvalidOperationException: Failed to simulate running the server with the supplied settings using: https://a.example.development.run:443 --->
System.InvalidOperationException: Client failed to contact webhost listening to 'https://a.example.development.run:443'.
Are you blocked by a firewall? Make sure the port is open.
Settings file: D:\RavenDB-4.0.0-windows-x64\Server\settings.json.
IP addresses: 10.0.1.4:443.
DNS is cached locally
Most networks cache DNS records. In some environments you can get an error such as this:
Setting up RavenDB in Let's Encrypt security mode failed.
System.InvalidOperationException: Setting up RavenDB in Let's Encrypt security mode failed. --->
System.InvalidOperationException: Validation failed. --->
System.InvalidOperationException: Failed to simulate running the server with the supplied settings using: https://a.onenode.development.run --->
System.InvalidOperationException: Tried to resolve 'a.onenode.development.run' locally but got an outdated result.
Expected to get these ips: 127.0.0.1 while the actual result was: 10.0.0.65
If we try resolving through google's api (https://dns.google.com), it works well.
Try to clear your local/network DNS cache or wait a few minutes and try again.
Another temporary solution is to configure your local network connection to use google's DNS server (8.8.8.8).
This error probably means that the DNS is cached. You can wait a few minutes or reset the network DNS cache, but in many cases, the easiest solution is to temporarily switch your DNS server to 8.8.8.8 You can click the Try Again button to restart the validation process of the Setup Wizard.
Long DNS propagation time
If you are trying to modify existing DNS records, for example running the Setup Wizard again for the same domain name, you may encounter errors such as this:
Setting up RavenDB in Let's Encrypt security mode failed.
System.InvalidOperationException: Setting up RavenDB in Let's Encrypt security mode failed. --->
System.InvalidOperationException: Validation failed. --->
System.InvalidOperationException: Failed to simulate running the server with the supplied settings using: https://a.example.development.run --->
System.InvalidOperationException: Tried to resolve 'a.example.development.run' using google's api (https://dns.google.com).
Expected to get these ips: 127.0.0.1 while google's actual result was: 10.0.0.65
Please wait a while until DNS propagation is finished and try again. If you are trying to update existing DNS records,
it might take hours to update because of DNS caching. If the issue persists, contact RavenDB's support.
If this happens, there is nothing you can do except wait for DNS propagation. When it's updated on dns.google.com click the Try Again
button.
You can keep track of your RavenDB clusters and their associated DNS records at the Customer's Portal.
If I already have the Zip file, can I avoid repeating the setup process?
Yes.
You can use the Zip file to re-install or deploy the server/cluster elsewhere.
Download a fresh copy of RavenDB and run the setup wizard. Then choose Continue Cluster Setup
and select node A.
This will use the existing Zip file and the same configuration and certificate which were previously chosen.
When building a cluster, repeat this step with nodes B, C, and so on.
Changing Configurations and Renewals Issues
- After installing with Let's Encrypt, can I change the DNS records?
- Can I change the (private) IP address RavenDB binds to?
- The Let's Encrypt certificate is about to expire but doesn't renew automatically
- What should I do when my license expires?
- Let's Encrypt certificate permission errors after renewal
After installing with Let's Encrypt, can I change the DNS records?
Yes.
- The Customers Portal allows you to easily edit DNS records that are associated with your license.
- You can run the setup wizard again.
If you supply different IP addresses then the wizard will update the DNS records of your domain.
If you use a new domain or if you add/remove nodes in the new configuration then the wizard will also fetch a new certificate.
Can I change the (private) IP address RavenDB binds to?
Yes.
Open the settings.json file located in the RavenDB Server installation folder,
change the ServerUrl
setting and restart the server.
The Let's Encrypt certificate is about to expire but doesn't renew automatically
If you are getting the following error you must update the RavenDB server.
Failed to update certificate from Lets Encrypt, EXCEPTION: System.InvalidOperationException:
Your license is associated with the following domains: ravendb.community but the PublicServerUrl
configuration setting is: Raven.Server.Config.Settings.UriSetting.There is a mismatch, therefore
cannot automatically renew the Lets Encrypt certificate. Please contact support.
If it's not the same error as above, please open settings.json in your Server installation and make sure you have all of the fields defined properly. Take a look at the following example:
{
"DataDir": "RavenData",
"License.Eula.Accepted": true,
"Security.Certificate.LetsEncrypt.Email": "your-email@example.com",
"Setup.Mode": "LetsEncrypt",
"Security.Certificate.Path": "cluster.server.certificate.aws.pfx",
"ServerUrl": "https://172.31.30.163",
"ServerUrl.Tcp": "tcp://172.31.30.163:38888",
"ExternalIp": "35.130.249.162",
"PublicServerUrl": "https://a.aws.development.run",
"PublicServerUrl.Tcp": "tcp://a.aws.development.run:38888"
}
Things to check:
-
"Setup.Mode" must be "LetsEncrypt"
The automatic renewal process only works if you acquired your certificate through the RavenDB setup wizard and used LetsEncrypt.
If you did not set up your cluster with the setup wizard and with LetsEncrypt, you are responsible to renew your certificate periodically.- To enable RavenDB's automatic certificate renewal, set up a new cluster with the setup wizard, create parallel databases, reconfigure the document store to connect to the new databases, and import the data.
- Security.Certificate.LetsEncrypt.Email
must be identical to the e-mail which is associated with your license. - PublicServerUrl and PublicServerUrl.Tcp
must contain the same domain as the one chosen during the setup wizard and is associated with your license. -
ExternalIp
should be defined only if you are running behind a firewall (cloud VM, docker, etc...). -
If all of this looks right, and the certificate still doesn't renew automatically and there are no alerts telling you what's wrong, you can contact support.
- Make sure to supply the server logs with your ticket. When running in a cluster, please provide the logs from all nodes.
- If your logs are turned off, open
Manage Server
->Admin Logs
in the Studio, and keep them open while you click theRenew
button in the certificate view.
What should I do when my license expires?
By default, the RavenDB license update server automatically renews licenses. If your server is running offline or a firewall is blocking the connection with the license server, you will need to renew your license key manually.
If your license is due to expire, be sure to renew the license.
Do not set up a new license because this will cause domain name issues.
If you get automatic Let's Encrypt certificate renewals, contact RavenDB support to renew the license because changing the license ID will cause a mismatch that will cancel the auto-certificate renewal.
Copy your new license code key from the email that was given when registering your license.
Paste it either:
-
Directly into the settings.json
- e.g.
"License": paste your license code including curly brackets here
- e.g.
-
Into the .json file
license.json
, to which the file path is provided in the settings.json- e.g.
"License.Path": "D:\\RavenDB\\Server\\license.json"
- e.g.
To renew your license via studio, navigate to the Studio "About" tab:
Renew License in Studio
-
Renew.
Click to send an email with the renewal key to the email account that is associated with the license. -
Submit License
Paste the renewal key from the email into the interface. -
Submit button
Click to complete the procedure.
Let's Encrypt certificate permission errors after renewal
If you have External Replication or ETL to another cluster, or if you use your own Let's Encrypt certificates as client certificates, the next certificate renewal may cause permission issues that need to be handled manually.
Learn how to handle this issue here.
Authentication Issues
- Authentication Error Occurred using Edge
- Authentication Error Occurred using Chrome
- RavenDB is running as a service in Windows and Chrome doesn't use the client certificate from the OS store
- Authentication Error Occurred in Firefox
- Cannot Import the Client Certificate to Firefox
- Getting the full error using PowerShell
- Not using TLS
- How to regain access to a server when you have physical access but no client certificate
- Certificate is not recognized when setting up on Azure App Services
- Automatic cluster certificate renewal following migration to 4.2
Authentication Error Occurred using Edge
You cannot access Studio using Edge, though during setup you checked the "Automatically register the admin client certificate in this (local) OS" checkbox and the setup wizard ended successfully.
There were problems authenticating the request:
This server requires client certificate for authentication, but none was provided by the client.
- Try closing all instances of the browser and restarting it,
or opening an incognito tab and pasting the server URL into the address bar. -
If clearing the cache didn't help, manually register the client certificate in the OS store.
- Under Windows:
Double-click the .pfx certificate.
Repeat clickingnext
for the default settings or provide your own settings. - Under Linux:
Import the certificate directly to the browser.
- Under Windows:
-
If the browser has presented several certificates and you selected the wrong one, you can -
- Remove the certificate from the browser's cache and reinstall the .pfx certificate as described above
- Or open an Incognito tab and paste the server URL into the address bar.
- In case none of the above works, you can use your own certificate and have RavenDB trust it.
You can use any client certificate that works under your OS and browser, even if it wasn't generated by RavenDB.
See trusting an existing certificate.
If your browser runs under Windows 7 or Windows Server 2008 or older:
The first thing to try would be installing the SERVER certificate to the OS where your server is running, closing all instances of the browser and restarting it.
If the issue persists, please also visit the Trusted Issuers List and follow method 3 (Configure Schannel to no longer send the list of trusted root certification authorities during the TLS/SSL handshake process) to set the following registry entry to false:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
Value name: SendTrustedIssuerList
Value type: REG_DWORD
Value data: 0 (False)
Authentication Error Occurred using Chrome
You cannot access Studio using Chrome, though during setup you checked the "Automatically register the admin client certificate in this (local) OS" checkbox and the setup wizard ended successfully.
There were problems authenticating the request:
This server requires client certificate for authentication, but none was provided by the client.
- Try closing all instances of the browser and restarting it,
or open an incognito tab (Ctrl+Shift+N) and paste the server URL into the address bar. -
If clearing the cache didn't help, manually register the client certificate.
- Chrome versions earlier than 105 look for certificatees registered with the OS.
Windows users can register a certificate with the OS by double-clicking its .pfx file and repeatedly clickingnext
for the default settings (or providing custom settings).
Linux users can import the certificate directly to the browser. - Chrome versions 105 and on look for certificatees registered with the browser's root store.
A failure to locate the certificate may be the result of registering it with the OS rather than with the browser.
This failure typically occures when self-signed certificates are used rather than with let's encrypt certificates issued during setup, since let's encrypt certificates are automatically installed in the Chrome root store.
Settings > Privacy and Security > Security > Manage device certificates
When a "Certificates" window opens, click Import and select your PFX certificate.
- Chrome versions earlier than 105 look for certificatees registered with the OS.
-
If the browser has presented several certificates and you selected the wrong one, you can -
- Either remove the certificate from the browser (Settings -> Privacy and security -> Security > Manage device certificates) and reinstall the .pfx certificate as described above,
- Or open an incognito tab (Ctrl+Shift+N) and paste the server URL into the address bar.
- In case none of the above works, you can use your own certificate and have RavenDB trust it.
You can use any client certificate that works under your OS and browser, even if it wasn't generated by RavenDB.
See trusting an existing certificate.
RavenDB is running as a service in Windows and Chrome doesn't use the client certificate from the OS store
Your RavenDB service may run under a certain user, for which the certificate was installed, while you
are currently using a different user for which no certificate was installed.
Or you may have registered the certificate with the OS, but are using a Chrome version higher than 105 that
looks for the certificate not at the OS root but at the Chrome root store.
To solve these issues: Using a user that the service is available for, install or import the certificate PFX as described above.
Authentication Error Occurred in Firefox
You cannot access the Studio using Firefox even though you have finished the setup wizard successfully and you also checked the box saying "Automatically register the admin client certificate in this (local) OS".
There were problems authenticating the request:
This server requires client certificate for authentication, but none was provided by the client.
Firefox doesn't use the OS certificate store like Chrome or Edge. Please import the certificate manually (In Firefox, "Settings" -> "Privacy and Security" -> scroll down to Security and click "View Certificates" -> "Import").
Then close all instances of the browser and restart it.
Cannot Import the Client Certificate to Firefox
You're trying to import the client certificate received from RavenDB to Firefox but get the following error:
The PKCS#12 operation failed for unknown reasons.
Firefox fails to import a certificate that is not password protected. To overcome this issue, use the RavenDB CLI to generate a password protected certificate. You can also add a password to the current certificate by using OpenSSL or by importing it to the OS store and exporting it back with a password.
Firefox sometimes fails to import a perfectly good certificate for no apparent reason and without a proper error message.
You can try to generate a new password-protected certificate using the RavenDB CLI and import that instead.
If it didn't help, you can use any other client certificate you have that works with Firefox (even if it wasn't generated by RavenDB) and have RavenDB trust it. See trusting an existing certificate.
You can also generate your own self-signed client certificate by using OpenSSL or Powershell.
This is a known issue which has been reported many times to Mozilla.
Some references:
Bugzilla: #1049435
Bugzilla: #458161
mozilla.dev.tech.crypto issue
Getting the full error using PowerShell
You can use PowerShell to make requests using the REST API.
If you are having trouble using certificates, take a look at this example which prints the full error (replace the server URL and the /certificates/whoami
endpoint with yours).
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$cert = Get-PfxCertificate -FilePath C:\secrets\admin.client.certificate.example.pfx
try {
$response = Invoke-WebRequest https://a.example.development.run:8080/certificates/whoami -Certificate $cert
}
catch {
if ($_.Exception.Response -ne $null) {
Write-Host $_.Exception.Message
$stream = $_.Exception.Response.GetResponseStream()
$reader = New-Object System.IO.StreamReader($stream)
Write-Host $reader.ReadToEnd()
}
Write-Error $_.Exception
}
Not using TLS
The RavenDB clients use TLS 1.2 by default. If you want to use other clients please make sure to use the TLS security protocol version 1.2 or 1.3.
{
"Url":"/admin/secrets/generate",
"Type":"Raven.Client.Exceptions.Security.InsufficientTransportLayerProtectionException",
"Message":"RavenDB requires clients to connect using TLS 1.2, but the client used: 'Tls'.",
"Error":"Raven.Client.Exceptions.Security.InsufficientTransportLayerProtectionException: RavenDB requires clients to connect using TLS 1.2, but the client used: 'Tls'.
at Raven.Server.RavenServer.AuthenticateConnection.ThrowException() in C:\\Builds\\RavenDB-Stable-4.0\\src\\Raven.Server\\RavenServer.cs:line 570
at Raven.Server.Routing.RequestRouter.TryAuthorize(RouteInformation route, HttpContext context, DocumentDatabase database) in C:\\Builds\\RavenDB-Stable-4.0\\src\\Raven.Server\\Routing\\RequestRouter.cs:line 168
at Raven.Server.Routing.RequestRouter.<HandlePath>d__6.MoveNext() in C:\\Builds\\RavenDB-Stable-4.0\\src\\Raven.Server\\Routing\\RequestRouter.cs:line 89
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
at Raven.Server.RavenServerStartup.<RequestHandler>d__11.MoveNext() in C:\\Builds\\RavenDB-Stable-4.0\\src\\Raven.Server\\RavenServerStartup.cs:line 159"
}
In PowerShell it can be solved like this:
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
How to regain access to a server when you have physical access but no client certificate
An admin client certificate can be generated through the [RavenDB CLI](../../server/administration/cli#generateclientcert If RavenDB runs as a console application, the CLI is just there. When running as a service, please use the rvn admin-channel
.
Use either the generateClientCert command, or (if you already have a certificate) the trustClientCert command.
Another way to gain access for an existing certificate is to add the Security.WellKnownCertificates.Admin configuration to settings.json
with your existing certificate's thumbprint.
In this case, a server restart is required.
Certificate is not recognized when setting up on Azure App Services
You may have gotten an error message like:
The credentials supplied to the package were not recognized (The SSL connection could not be established, see inner exception.)
1) In the app settings of your Azure App Services application, add the WEBSITE_LOAD_USER_PROFILE = 1
option.
2) Another solution is to use the MachineKeySet
flag during certificate creation:
[DocumentStore].Certificate = new X509Certificate2("[path to your pfx file]",
(string)null, X509KeyStorageFlags.MachineKeySet);
Automatic cluster certificate renewal following migration to 4.2
Security.Certificate.Exec
was deprecated in 4.2 and replaced by Security.Certificate.Load.Exec
. You can use your old Security.Certificate.Exec
executable by simply moving it to this new path. The settings Security.Certificate.Renew.Exec
and Security.Certificate.Change.Exec
have been added
for automatically persisting the certificate to the whole cluster. If you have your own mechanism for doing this, or are using a single node cluster,
you still need to place empty scripts in the Security.Certificate.Renew.Exec
and Security.Certificate.Change.Exec
paths or an exception will be
thrown.
Encryption Issues
Insufficient Memory Exception
Memory exception occurred: System.InsufficientMemoryException:
Failed to increase the min working set size so we can lock 4,294,967,296 for
D:\stackoverflow\RavenData\Databases\SO\Indexes\Auto_Questions_ByBody\Temp\compression.0000000000.buffers.
With encrypted databases we lock some memory in order to avoid leaking secrets to disk.
Treating this as a catastrophic error and aborting the current operation.
When encryption is turned on, RavenDB locks memory in order to avoid leaking secrets to disk. Read more here.
By default, RavenDB treats this error as catastrophic and will not continue the operation. You can change this behavior but it's not recommended and should be done only after a proper security analysis is performed, see the Security Configuration Section.
If such a catastrophic error occurs in Windows, RavenDB will try to recover automatically by increasing the size of the minimum working set and retrying the operation.
In Linux, it is the admin's responsibility to configure higher limits manually using:
sudo prlimit --pid [process-id] --memlock=[new-limit-in-bytes]
To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.