Setting Up RavenDB Cluster on AWS EKS

Introduction

Deploying RavenDB in a Kubernetes environment, particularly on Amazon EKS, offers a scalable and resilient database solution.

Setting up any database in Kubernetes is… well… complex. It’s a stateless world, and databases are stateful. Databases need to have a fixed setup and configuration. Auto-recovery indicates that things won’t permanently be fixed but more fluid. These things don’t mix well, as you can see.

However, many software solutions cover these pain points. We use controllers to ensure fixed setup and configuration within a fluid environment. They will ensure that automatic instance re-creation won’t break your cluster logic and that everything will return to normal after recovery.

Also, remember that with great power comes great responsibility. EKS is an expansive tool that requires careful configuration to maximize its potential without any hiccups. This guide will walk you through the steps to deploy RavenDB on EKS, ensuring a secure, performant, and adequately connected setup.

Understanding the Key Concepts

Before we dive into the deployment steps, let’s briefly cover some core concepts you’ll work with in this guide:

Basics

• Declarative Infrastructure: Kubernetes operates on a declarative model, meaning you define the desired state of your infrastructure, and Kubernetes ensures it remains in that state.
• Resources: These include pods, services, storage volumes, roles, and networking components that make up your cluster.
• Controllers: Components like EBS CSI (storage), Load Balancer Controller (LBC), and networking controllers that automate various cluster operations. They operate on your cluster to achieve configured goals, like automatically linking storage disks, assigning pod VPC IP address, or load balancer creation & connecting to re-created Kubernetes pods. It can even control your Route53 to ensure a valid DNS configuration for new load balancers.

Amazon EKS

• EKS (Elastic Kubernetes Service): AWS-managed Kubernetes service that simplifies cluster management.
• EKS Cluster Add-ons: Pre-packaged AWS-managed components that extend cluster functionality. Available to quick-add from UI.

NOTE: Manually installed controllers (not Add-ons) often require specific IAM roles and service accounts to operate correctly and ensure proper resource access. You can usually assign an IAM role to a controller pod by Pod Role Association, but most guides recommend setting up OIDC. By the end of this guide, we’ll cover that.

Guide to Boot-up: Preparing Your EKS Cluster

This guide showcases the required components and actions to build a healthy cluster. We’ll try to show how you can set it up using AWS Web UI. However, some steps will require you to use commands from your terminal. To preserve this article, we will link resources that should be updated, like official AWS documentation, to go through particular setup steps.

We will provide detailed instructions on setting the cluster up. That’s why most of the space of this guide is taken by screenshots for a step-by-step guide.

You’ll need more manual intervention when installing the Load Balancer Controller and ExternalDNS, but we will provide our know-how and link external & updated deployment guides that match our case here. If you want to deploy the .yaml, LBC, and ExternalDNS, scroll down to point 3.

0. Prerequisites

Before starting, ensure you have:

• An AWS Account with sufficient permissions.
• kubectl installed and configured for EKS.
• eksctl (useful for some steps).
• Registered private domain (for internet exposure)

1. Create a new EKS cluster

Go to EKS > Clusters and create a new cluster.

First, we will need to give the new EKS cluster privileges so it can control AWS resources. In the configuration, let’s name the cluster and create the IAM role. Click “Create Recommended Role.”

Let’s pick the EKS – Cluster Use case.

Then, click Next.

Review and save.

Well – as you can see, the recommended role ain’t working…

Let’s fix this – Click Edit in the IAM console.

Attach policies to your role. Click Add permissions > Attach policies. Select the listed policies and save.

Policies added successfully – it should look like this:

Head back to the cluster creator.

Well, we need to add trust policies, too. Let’s fix this

Open the previous view (Edit in IAM console). Click “Trust relationships”, then “Edit trust policy”.

The policy editor will open. In the panel on the right, search for “TagSession.” Click the checkbox, save, and go back.

We leave those options up to you:

Go next to the ‘Networking’ section.

We recommend creating a separate VPC for the EKS cluster. For details, visit this page: https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html

Let’s create three subnets in our VPC. We will assign 256 IP addresses each. Each one should have a different AZ.

Success:

We’ve also created a security group that allows all traffic. You can modify it to achieve better (any) security.

Now, we have a very important step.

We need to modify the subnets we’ve just created to be publicly available. This is our part of the setup, where we make public subnets. You can go private if you need to.

For more information on public and private subnets and overall cluster networking, visit this page:

https://aws.amazon.com/blogs/containers/de-mystifying-cluster-networking-for-amazon-eks-worker-nodes

Open your subnets overview.

First, go to Actions>Edit subnet settings.

Enable Automatic IPv4 assignment.

Save. Then, we need to edit the route tables, as our subnets won’t be able to exchange traffic from the Internet. First, let’s create an Internet Gateway. We will need it to configure the route tables.

Search for Internet Gateways, open their panel, and create a new one.

Name it appropriately, and click Create. The green bar “Attach to VPC” should appear.

Click it and select your VPC.

Success.

Let’s go back to our subnets.

Click “Route tables”, and go to the bound route table by clicking its’ name (highlighted in image).

It automatically searched for a valid route table.

Click it’s ID (2nd column).

As you can see, the route table is bound to all of our subnets.

Let’s modify its routes.

Add 0.0.0.0/0 Destination with a target being the Internet Gateway we’ve just created. Save changes.

Let’s go back to the EKS cluster configuration.

Our cluster networking should look like this:

Configure observability up to your needs; we’ll leave it default:

Next, to “Add-ons,” we’ll install defaults and the EBS CSI Driver. We’ll need all of them for proper networking; EBS CSI is a controller that we’ll need for dynamic storage space provisioning.

Don’t forget to create a dedicated IAM Role for EBS CSI

Create your cluster. You should get a new cluster dashboard. Head to Add-ons. Verify this view

2. Compute resources – node group

Create a node group with at least 3 machines that will serve your cluster as host machines to all these add-ons… and, of course, your RavenDB instances 😉

Click “Add node group”.

Let’s create a recommended role, just like previously.

Use Case now is “EC2”.

Let’s use it.

Then go Next.

Select the machine type you want to use. RavenDB supports both X64 and ARM architectures. If you wish to spread RavenDB instances across the cluster, increase the number of machines to a minimum of 3.

Go Next and select previously created Subnets.

Go next, review your config, and click Create.

Your node group should be up in a longer while.

We created and configured the EKS Kubernetes cluster with a node group attached. Now, let’s deploy RavenDB on it.

3. Connect with kubectl, and deploy your RavenDB.yaml

Now, let’s deploy this RavenDB cluster in a .yaml. It has a defined separate namespace, RBAC on a secret, a config map with scripts, a storage class based on EBS…

All of this is to run the RavenDB cluster on your node group.

Necessary: The linked .yaml has many “todo” items. Before deploying, you must fill them with your domain, license, certificates, and other information.

To deploy it, you need to bind your kubectl to your EKS cluster.

Here’s the guide on how to do it: https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html

Let’s try to get all pods:

It works! Now you see why we assigned so many IP addresses. Each pod network interface has a separate IP!

Now let’s deploy the file:

Command “kubectl logs” should show RavenDB Server starting:

The next steps will focus on exposing your instance since it is currently in a black box on Kubernetes’ internal network. Even though it’s running, there’s no way to reach your server yet.

4. Setup LBC (load-balancer-controller)

We will use Network Load Balancing, an AWS feature, to route external traffic inside the cluster.

Long story short, it can route complete traffic (OSI Layer 4) and won’t drop the TLS payload containing the client’s access certificate.

But to do so, we need a fixed target to which the load should be balanced. It’s kind of the opposite of K8’s fluid nature, right? Right… We need another controller—a load balancer controller. It will detect changes in our Kubernetes cluster and re-configure Network Load Balancers on every change to maintain connectivity.

We’ll need to install it from “hand” as no LBC Add-on is currently available.

Follow the guide using eksctl and helm to automate some setup parts. The eksctl usually deploys components using Amazon CloudFormation. The helm is a package manager for Kubernetes. More info here: https://helm.sh/

Here are guides that you should follow:

1. First, create an OIDC provider, which will be needed for the AWS Load Balancer Controller. Follow this guide: https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html
2. Follow the steps from this tutorial to install the AWS Load Balancer Controller: https://docs.aws.amazon.com/eks/latest/userguide/lbc-helm.html

(optionally) Here, you can also make sure the guide is up to date with the newest release: https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/

This way, your Services of type “LoadBalancer” will have a connection to the outside world, as LBC will provision Network Load Balancers.

Your network load balancers will have random domain names that will change upon failure. To keep things solid, we need the last controller, which will allow you to:

• Connect to your instances by your domain name, always the same
• Connect your nodes into a cluster, as your assigned PublicServerUrl will officially work, and other nodes will use it to talk to each other without problem.

5. Setup ExternalDNS

Installing ExternalDNS in the cluster is relatively easy if you installed LBC by yourself, but also by another long step, which we won’t cover here, as there are official guides to follow, and our article would grow at an uncontrollable speed. https://kubernetes-sigs.github.io/external-dns/latest/docs/tutorials/aws/

Long story short? After you install ExternallDNS in your cluster, you’ll be able to annotate your Service component like this (take a look at the line with comment):

It will allow you to rely on your instance address. This is an absolute game-changer: now, you can connect RavenDB nodes in a cluster in a completely fluid and autonomous environment.

It should spawn DNS records like this, routing traffic from your domain to LBC-dynamically spawned Network Load Balancer:

Conclusion

Setting up a database on EKS from scratch may seem complicated, but this article provides the know-how to make your journey into this dark world much less painful. We’ve successfully set up a working EKS RavenDB cluster, fully automated (automated storage, computing, networking, security, and external access) by controllers.