In this guide, we’ll walk through the steps needed to take enable Kubernetes’ high availability capabilities with a cluster installed by kURL and managed by KOTS.
This only applies to ‘embedded’ cluster installs using Kurl.
As with the rest of the guides, this is meant as a ‘hello world’ example to help get you familiarized with the process of standing up a highly available Kubernetes cluster using the Replicated toolset. The architecture we will follow is presented below:
The diagram shows 4 Virtual Machines (VMs) running on Google Cloud Platform (GCP), with one of them used solely as a load balancer. Note that there aren’t any requirements to use GCP. You can follow along with VMs provisioned on a different cloud provider or on premise.
The guide is broken into two parts. The first part will walk you through how to stand up the cluster, while the second part will go through a couple of testing scenarios to validate the cluster’s resiliency and some troubleshooting tips.
There are a few things to keep in mind about this guide:
- While the VMs are going to be created in Google Cloud Platform (GCP), nothing in this guide is dependent on GCP specific services.
- The Load Balancer we’ll use for this exercise is HAProxy.
- The sample application has a database component as described here, which will help us validate data retention in our testing scenarios.
This exercise will need 4 VMs that will need to communicate with each other. Please refer to the Procuring the (virtual) hardware section for guidance on system requirements for these.
In the example commands and screenshots, we are using a sample application called ‘AppDirect’. Please refer to the Sample Application section for more details if you wish to follow along and use it.
This is an advanced topic, so this guide will assume you’ve already completed one of the Getting Started Guides, and have already a level of familiarity with Replicated.
The KOTS application we’ll use in this guide is available in the kotsapps repository.
The application consists of two components:
- Python Flask App Deployment - This flask application contains ‘routes’ to help test connecting, reading and writing to a database.
- Postgres StatefulSet - This is the database for the flask application.
A full description of the application is available in the repository.
If you want to follow this example, but with your own application, you may need to modify the Kubernetes Installer to use EKCO. For the purposes of this guide, EKCO can be thought of as a helper service that helps ‘move’ scheduled pods from a node that is down to ones that are up.
To add EKCO to your Kubernetes installer, follow the steps below. If you are not familiar with the Kubernetes installer, review this blog first to get familiar with the process.
Adding EKCO to Embedded Installs
In your Kubernetes installer, add the following under spec:
Promote the installer to same channel(s) where you have promoted your application’s releases.
Part I - Setting Up the Cluster
To stand up the cluster, we will be following these steps:
Provisioning the (Virtual) Hardware
For this exercise, we are going to create 4 Virtual Machines. One of the VMs will be used to install, configure and run HAProxy. The remaining 3 VMs will be the 3 Primary Nodes for our cluster.
Provisioning the HAProxy VM
For the VM that will host HAProxy, a bare bones VM should suffice as we will only use it to route traffic. As an example, the command below creates a VM instance with enough juice to run HAProxy, at least for testing purposes:
gcloud compute instances create app-direct-ha-proxy --boot-disk-size=10GB --labels=app=app-direct --image-project ubuntu-os-cloud --image-family ubuntu-1804-lts --machine-type n1-standard-1
In the command output, you should note the public and internal IP address. The public address of this VM is what we’ll use to connect to both the KOTS Admin Console, the application’s UI and a Postgres client of your choice. We’ll also use the interal IP address when we install KOTS in HA mode.
If you’d like to explore HAProxy requirements for heavier workloads, please check out HAProxy’s documentation
Provisioning the cluster VMs
Like with the HAProxy VM, keep the KOTS System Requirements in mind when provisioning the VMs that will form the cluster. The command below creates a VM that meets those requirements, at least for testing purposes.
gcloud compute instances create app-direct-node-01 --boot-disk-size=200GB --labels=app=app-direct --image-project ubuntu-os-cloud --image-family ubuntu-1804-lts --machine-type n1-standard-4
To create the remaining nodes, simply run the same command but increment the node number (i.e., “app-direct-node-02”).
Also note the VMs public and private ip addresses as each VM is provisioned. We’ll use the internal ip addresses in the HAProxy configuration in the next step.
Setting Up & Configuring a Load Balancer
The main purpose of the load balancer is to give us a single point of access. All end-user interactions (i.e., access the web UI of the deployed application) should go through the load balancer.
For this guide, HAProxy will be the load balancer and will be running on its own VM, provisioned already in the previous step. HAProxy is configured by editing the haproxy.cfg file that by default resides in ‘/etc/haproxy’.
Here are the high level steps we’ll take to install & configure HAProxy.
- Create the config file in our home folder using our favorite editor.
- Use SCP to copy the file in the VM running HAProxy.
- Install HAProxy.
- Copy the config file to the proper location.
- Restart HAProxy.
Note that another way of accomplishing this is by connecting to the instance via SSH and creating the file and editing from the command line inside the VM.
To create the config file, use the editor of your choice and call it ‘haproxy.cfg’. For the purposes of this exercise, we’ll create the file in our home folder. Below is an example config file for the sample application. Keep in mind the following when creating your file:
- Replace any IP addresses with the internal IP addresses of your VMs.
- If you used a different naming convention for your instance names, you will need to replace ‘app-direct-node-0(123)’ with your instance names.
- There are two entries specific to the Application, one for each component (app-direct-frontend/backend & postgres frontend/backend). If you are using a different application, adjust based on the endpoints you want to access through the load balancer. The remaining entries are for the KOTS Admin Console and the Kubernetes API and should be left as-is other than the instance names and ip addresses.
timeout client 60s
timeout server 60s
timeout connect 60s
stats uri /
server app-direct-node01 10.240.0.37:80 check
server app-direct-node02 10.240.0.39:80 check
server app-direct-node03 10.240.0.71:80 check
server app-direct-node01 10.240.0.37:5432 check
server app-direct-node02 10.240.0.39:5432 check
server app-direct-node03 10.240.0.71:5432 check
server app-direct-node01 10.240.0.37:8800 check
server app-direct-node02 10.240.0.39:8800 check
server app-direct-node03 10.240.0.71:8800 check
server app-direct-node01 10.240.0.37:6443 check
server app-direct-node02 10.240.0.39:6443 check
server app-direct-node03 10.240.0.71:6443 check
For more details on the structure of the config file, please refer to the HAProxy documentation
To copy the config file into the VM, we are going to use scp. The command below will copy it to your home folder on the VM.
gcloud compute scp haproxy.cfg app-direct-ha-proxy:~/
Now we install HAProxy using apt.
sudo apt-get update
sudo apt install -y haproxy
The install will create a default config file. We will back it up an then move ours to the /etc/haproxy directory created by the install.
sudo mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.bak
sudo mv ~/haproxy.cfg /etc/haproxy/haproxy.cfg
Last step is to restart HAProxy
sudo systemctl restart haproxy
To validate the install, browse to http://:8080. Below is a screenshot of what it should look like. The dashboard at this point does not contain much valuable information as all the frontend/backend services we configured are not running yet.
Run the KOTS Installer & Deploy Application
The next step is to install KOTS on the first node, in our case, app-direct-node-01. It’s important to note that in reality, you could run the installer on any of the three nodes.
To run the installer, ssh into the node and run the ‘one line installer’. You can get this command from the Vendor portal as shown below.
However, in order to enable High Availability we are going to add the ‘-s ha’ advanced option
curl -sSL https://k8s.kurl.sh/appdirect-unstable | sudo bash -s ha
When you include the ‘-s ha’ option, you will be prompted right away for the load balancer address. As shown in the figure below, we are providing the ‘internal’ IP address. This address is used by the Kubernetes services in the nodes to talk to each other. Using a public IP address would add extra layers that are not needed.
This process will take several minutes as it will install a Kubernetes ‘stack’ that will include the add-ons defined in the Kubernetes Installer, including KOTS. Once it is finished, you should see something similar as shown below:
Highlighted in red in the screenshot above are the URL and password to login to the Admin Console. Also highlighted are the commands to run on other nodes to join the cluster. In this exercise we are not going to run these commands yet and instead continue deploying the application. The commands to join the cluster are also available in the Admin Console, which we will cover later in this guide.
Next, we are going to log in to th Admin Console to complete the install and deploy the application. Since this guide assumes you have a level of familiarity with Kots, we are not going to go into detail on every step to get the application deployed.
Log in to the Admin Console using the address and password from the install. Once you have logged in using the provided password, you will need to provide a license. The license is generated in the Vendor Portal and this guide assumes you already know how to do this. There are no options that need to be enabled or turned on in order to support an HA cluster, so if you already have a development license, it should work for this exercise.
Once you upload your license it may take a few minutes to load and continue with the deployment.
Once the license is loaded, the next window will depend on the application being deployed as this window renders whatever has been codified in the application’s Config.yaml.
After the configuration window, you should see the preflight checks and eventually land on a window similar to what is shown below
Verify the Deployment
To see everything that has been deployed, ssh into one of the nodes and run:
kubectl get pods --all-namespaces -o wide
The output will show you all the pods running on all namespaces. The output also shows you the node that each pod is currently running on, which will come in handy during this exercise. Since we only have a one node cluster, all pods should be running on node 1. Check the status of the pods to make sure there are no issues.
If you are following along, you should be able to access the application by browsing to http://. The sample app uses flask ‘routes’ to call various methods to interact with Postgres. To test if it can write to the database, use the
/sql-check route which will check the connection to the default database and list the databases that are available to the ‘postgres’ user. To write to the database, first use the
/sql-create route which creates a database (appdirectdb) with a table (tblrecords) in it. Finally, use the
/sql-add route to add a row to the table with the timestamp. Once the database and table are created, simply use this route to add more records to the database.
To verify that the data is in fact being written to Postgres, use the Postgres client of your choice. The screenshots included in this guide are from pgAdmin with a connection to the database using the Load Balancer’s IP address.
Once connected using the client of your choice, you can retrieve the records by running a query like this:
SELECT * FROM tblrecords
You should see records for each time that the /sql-add route is accessed. Below is a screenshot of what it looks like in pgAdmin:
Adding Remaining Nodes to Cluster
To add the remaining nodes to the clusters, follow these steps:
Get the Command to Join the Cluster
In the Admin Console, under Cluster Management you can view the status of the embedded cluster. At this time, it only has one node.
To get the command to run on the other nodes, click on the ‘Add Node’ button. This will display the command to use for Primary and Secondary nodes. For our exercise we will add Primary nodes to give us a proper HA cluster.
SSH into the Nodes & Run the Command
SSH into both nodes and run the commands by pasting it into the terminal.
This should run a process similar to the initial install of the cluster and will take a few minutes. Once it is finished you should see something similar to the output below:
Verify Pods Are Running on All Nodes
To verify that the nodes have in fact joined the cluster you can run the following commands:
The output should show us all three nodes of our cluster and have a status of
kubectl get pods --all-namespaces -o wide
The output should show that other than the default namespace, there are pods running on all three nodes.
Part II: Testing Scenarions & Troubleshooting
There are various testing scenarios for an HA Cluster. For starters we are focusing on the data retention/loss scenario. The sample application is already configured to write to the embedded Postgres StatefulSet.
This section assumes you followed the above steps in order and are using the sample application. Since we are testing data retention, make sure to use the routes described in the Verify the Deployment section.
To verify that data is not lost, you will need a postgres client. As mentioned already in thid guide, you can use the Postgres client of your choice. The screenshots in this guide are from pgAdmin.
To observe how the pods are scheduled as we conduct our test scenarions, ssh into Node 1 and run:
watch -d 'kubectl get pods --all-namespaces -o wide'
watch -d the output will automatically get refreshed so you can watch pod activity as you delete pods and node.
Deleting a Pod
If you followed along with the steps and sample application, you should see that the Postgres pod is running on Node 1. In this test, we are going to delete the Postgres pod. The expected result is that a new pod will be scheduled (possibly on another node) and all the data in the database will be retained and not lost.
To delete the Postgres pod deployed with the sample app, ssh into one of the Nodes, and run the following command:
kubectl delete pod app-direct-postgresql-0
This command will schedule a new pod and then terminate the existing one. You can verify this on the terminal where you ran the watch command. After the Postgres is back up and runing, verify that the data is there by either using the
/sql-check route in the sample application or a Postgres client.
Shutting Down a Node
There are a few things you need to keep in mind:
The Postgres instance in this application, and the application itself are not configured or set up for High Availability. This test is meant to show the resiliancy capabilities of the Kubernetes embedded cluster installed by KURL and configured through KOTS.
As described in the Kubernetes Documentation, there is a default 5 minute timeout duration. This means that there will be 5 minutes between the time the node goes down and that Kubernetes will finally decide that the node is not coming back and will schedule pods on this node to another node.
To simulate a node becoming unresponsive or simply crashed, you can simply stop the VM corresponding to the node. The only caveat here is that you are not running the watch command on the node you are planning to take down.
You are likely not to see much change in the pods during the first five minutes, other than some pods erroring out. After 5 minutes or so, you will start to see pods being terminated and scheduled on the remaining nodes.
In this scenario, the downtime for Postgres was about 6 - 7 minutes. This includes the Kubernetes time out of 5 minutes, 1 minute for EKCO to delete the node and remove it from the ceph cluster, and some remaining time is for Postgres to start up on a new node.
As you go through this exercise, you may run into some of the issues below:
Postgres pod will not come up after deleting the pod
In the first test above, after deleting the Postgres pod, the new one would not come up. After looking at the logs of the pod found errors like
find: ‘/var/lib/postgresql/data/pgdata/pg_stat_tmp/global.stat’: Structure needs cleaning
chown: cannot access '/var/lib/postgresql/data/pgdata/pg_stat_tmp/global.stat': Structure needs cleaning
This appeared to be more of an OS issue, so I deleted this pod. When the new one was scheduled all was fine.
Postgres unable to mount to data directory
On one occassions when I shut down the node that Postgres was running on, the new pod that that got scheduled on one of the remaining nodes was not able to come up due to volume mount issues. I ran
kubectl describe on the Postgres pod and saw the following error:
It turned out that I did not have my EKCO configuration set up correctly. To resolve the issue, I updated my Kubernetes installer as described in the configuring ekco section.
Other Persistent Data Issues
The default configuration of the Kurl installer includes the rook add-on, which creates and manages a Ceph cluster along with a storage class for provisioning Persistent Volume Claims (PVCs). If you are running into other issues with persistent storage, following are some commands that were helpful when I ran into the issues mentioned above.
These commands check the health of ceph in your cluster. Keep in mind that the screenshots below show a
healthy cluster that has all nodes up and running.
All related pods are runnning in the rook-ceph namespace, and a are good place to start to troubleshoot any issues. By running
kubectl get pods -n rook-ceph you should get an output similar to the one below:
One thing to note here is that you will see pods running on all three nodes. Orchestrating all of the data accross all nodes is the
rook-ceph-operator-... pod. To troubleshoot issues, we’ll need to exec into this pod in order to run some commands. For example, to exec into the pod on the screenshot above, I would run
kubectl exec -it rook-ceph-operator-6fbfdccc57-p477z.
The check the status of ceph, run
The output provides you with a high level overview of the ceph cluster. Note the value for ‘health’ is set to
HEALTH_OK. When a node is unresponsive, the ‘health’ is set to
To check the Object Storage Deamons (OSDs) in more detail, run
ceph osd status
To get another look at the OSDs, run
ceph osd tree
The guide covers a basic 3 node HA cluster. However, there are other layers of complexity that could be added depending on your use case:
- Add worker nodes to the cluster and run the same tests.
- Set up https access
- Set up Postgres in HA mode and compare down time against a single Postgres deployment.