Docker Swarm HA (gluster)

In this guide, I will try to explain how to set up a Docker Swarm system that is completely highly available. In total we need to setup five Ubuntu-Server machines (3x Docker machines).

Components to make Docker Swarm high available:

By default volumes of Docker containers are not synchronized between the docker swarm machines. To overcome this limitation we are installing GlusterFS on all three docker swarm machines. INFO: Wikipedia: GlusterFS is a distributed file system that presents storage elements from multiple servers as a unified file system.
As a reverse proxy we are installing Nginx Proxy Manager as a container on our Docker Swarm.
For a graphical interface to manage our Docker Swarm we are installing Portainer as a container on our Docker Swarm.
We need a virtual IP-Address and a load balancer to distribute all incoming request to one of our three Docker machines. Therefore we are setting up keepalived on our three Docker machines.

Requirements for VMs

A disk dedicated for ubuntu operating system. (20GB)
A second disk that is used for the glusterFS (shared filesystem) (50 GB or more).
All 3 machines should be in the same IP subnet (low-latency).
For the three docker swarm VMs I recommend at least 4GB RAM.

Content

Topology
1. Install Keepalived
2. Install Docker Swarm
3. Build Docker Swarm
4. Install GlusterFS
5. Install Portainer
6. Install Nginx Proxy Manager
7. Perform port forwarding on your router
8. (Optional) Set up demo web server as service

Topology

1. Install Keepalived

Keepalived is a routing software that provides high availability and load balancing for Linux-based systems. It is primarily used to manage virtual IP addresses (VIPs) and to ensure that services remain available even in the event of server failures. Keepalived uses the Virtual Router Redundancy Protocol (VRRP) to achieve this.

After installing ubuntu-server on all three machines proceed to installation of keepalived via apt:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo apt update -y
sudo apt install -y keepalived

Now we need to setup the keepalived config on dockerswm1:

!!! Only on machine dockerswm1 !!!:

Open the nano editor:

sudo nano /etc/keepalived/keepalived.conf

!!! Only on machine dockerswm1 !!!:

Insert the following code block into the nano editor. You need to replace ens192 with the primary ethernet interface of the machine. Command to list ethernet interfaces: ip addr

vrrp_instance VI_1 {
        state MASTER
        interface ens192
        virtual_router_id 51
        priority 255
        advert_int 1
        authentication {
              auth_type PASS
              auth_pass 12345
        }
        unicast_peer {
            192.168.0.101
            192.168.0.102
        }
        virtual_ipaddress {
              192.168.0.99/24
        }
}

Now we need to setup the keepalived config on dockerswm2:

!!! Only on machine dockerswm2 !!!:

Open the nano editor:

sudo nano /etc/keepalived/keepalived.conf

!!! Only on machine dockerswm2 !!!:

Insert the following code block into the nano editor. You need to replace ens192 with the primary ethernet interface of the machine. Command to list ethernet interfaces: ip addr

vrrp_instance VI_1 {
        state BACKUP
        interface ens192
        virtual_router_id 51
        priority 254
        advert_int 1
        authentication {
              auth_type PASS
              auth_pass 12345
        }
        unicast_peer {
            192.168.0.100
            192.168.0.102
        }
        virtual_ipaddress {
              192.168.0.99/24
        }
}

Now we need to setup the keepalived config on dockerswm3:

!!! Only on machine dockerswm3 !!!:

Open the nano editor:

sudo nano /etc/keepalived/keepalived.conf

!!! Only on machine dockerswm3 !!!:

Insert the following code block into the nano editor. You need to replace ens192 with the primary ethernet interface of the machine. Command to list ethernet interfaces: ip addr

vrrp_instance VI_1 {
        state BACKUP
        interface ens18
        virtual_router_id 51
        priority 253
        advert_int 1
        authentication {
              auth_type PASS
              auth_pass 12345
        }
        unicast_peer {
            192.168.0.100
            192.168.0.101
        }
        virtual_ipaddress {
              192.168.0.99/24
        }
}

Now we need to start the Keepalived service and enable autostart of the service on each machine:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo systemctl start keepalived
sudo systemctl enable keepalived

2. Install Docker Swarm

Good documentation can be found on Docker docs: https://docs.docker.com/engine/install/ubuntu

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo apt update -y
sudo apt install ca-certificates curl gnupg lsb-release -y

Add the official Docker GPG keyrings:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Add the Docker repository:

The following command is repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo apt update -y
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

3. Build Docker Swarm

Good documentation can be found on docker docs: https://docs.docker.com/engine/swarm/swarm-mode

INFO: There are two different roles that a Docker machine takes in a Docker swarm

Manager: Delegates tasks to the worker nodes. Takes over administrative tasks. Also takes over functions of a worker node.
Worker: Performs common tasks of a worker node, e.g. provisioning and operating containers.

In this tutorial we are adding all the Docker machines as Manager to the Docker Swarm.

dockerswm1 (192.168.0.100) init Swarm

Initialize the Docker Swarm:

!!! Only on machine dockerswm1 !!!:

sudo docker swarm init --advertise-addr 192.168.0.100

sudo docker swarm join-token manager
//////////// Example output to add Manager-Node: ////////////

    docker swarm join \
    --token SWMTKN-1-59egwe8qangbzbqb3ryawxzk3jn97ifahlsrw01yar60pmkr90-bdjfnkcflhooyafetgjod97sz \
    192.168.0.100:2377

/////////////////////////////////////////////////////////////

dockerswm2 (192.168.0.101) add to Swarm as Manager

Now the machine dockerswm2 is added to the swarm as an additional manager node:

Please add the machine as Master-Node to the docker swarm !!!

!!! Only on machine dockerswm2 !!!:

docker swarm join \
    --token SWMTKN-1-59egwe8qangbzbqb3ryawxzk3jn97ifahlsrw01yar60pmkr90-bdjfnkcflhooyafetgjod97sz \
    192.168.0.100:2377

dockerswm3 (192.168.0.102) add to Swarm as Manager

Now the machine dockerswm3 is added to the swarm as an additional manager node:

Please add the machine as Master-Node to the docker swarm !!!

!!! Only on machine dockerswm3 !!!:

docker swarm join \
    --token SWMTKN-1-59egwe8qangbzbqb3ryawxzk3jn97ifahlsrw01yar60pmkr90-bdjfnkcflhooyafetgjod97sz \
    192.168.0.100:2377

OPTIONAL: Check if docker nodes are connected to each other

sudo docker node ls

4. Install GlusterFS

Good documentation can be found on GlusterFS documentation page: https://docs.gluster.org/en/latest/Install-Guide/Install and https://docs.gluster.org/en/latest/Install-Guide/Configure/#partition-the-disk

In this step, a GlusterFS is installed on each of the three machines (dockerswm1, dockerswm2 and dockerswm3). This service is used to synchronize the persistent data of containers (volumes). This is necessary because by default persistent data is not synchronized between nodes in a Docker Swarm.

Installation of GlusterFS:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo apt install glusterfs-server -y

Format the partition: Assuming you have an empty disk at /dev/sdb: (You can check the partitions on your system using sudo fdisk -l)

And then create a single XFS partition using fdisk Format the partition:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo mkfs.xfs /dev/sdb

Create a directory where you will mount the device:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo mkdir /mnt/glusterfs

Add to /etc/fstab for automatic mounting:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

echo "/dev/sdb /mnt/glusterfs xfs defaults 0 0" | sudo tee -a /etc/fstab
sudo mount -a
sudo systemctl daemon-reload

The hostnames and IP addresses of all machines are now entered in the /etc/hosts file on all machines. Thus everyone knows everyone.

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

echo "192.168.0.100 dockerswm1" | sudo tee -a /etc/hosts
echo "192.168.0.101 dockerswm2" | sudo tee -a /etc/hosts
echo "192.168.0.102 dockerswm3" | sudo tee -a /etc/hosts

After that, the GlusterFS daemon service must be started and activated on all machines:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo systemctl start glusterd
sudo systemctl enable glusterd

Now we create a memory cluster, starting with one of the machines and adding the others with this command:

!!! Only on machine dockerswm1 !!!:

sudo gluster peer probe dockerswm2
sudo gluster peer probe dockerswm3
sudo gluster pool list

Now we create a second folder (/mnt/dockerswarmvol) in which we mount the GlusterFS volume so that containers can store their persistent data there.

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

sudo mkdir /mnt/dockerswarmvol

Now the replication-volume is set up over all three Docker Swarm machines. I for example set the replication-volume name to staging-gfs:

!!! Only on machine dockerswm1 !!!:

sudo gluster volume create staging-gfs replica 3 dockerswm1:/mnt/glusterfs dockerswm2:/mnt/glusterfs dockerswm3:/mnt/glusterfs force
sudo gluster volume start staging-gfs

The volume is now ready to use, but we need to make sure that the volume is mounted again on reboot (or under other circumstances). We will mount the volume to the /mnt/dockerswarmvol directory. This is done by issuing the following commands on all machines:

The following commands are repeated for the following systems (dockerswm1; dockerswm2; dockerswm3):

echo 'localhost:/staging-gfs /mnt/dockerswarmvol glusterfs defaults,_netdev,backupvolfile-server=localhost,noauto,x-systemd.automount 0 0' | sudo tee -a /etc/fstab
sudo mount.glusterfs localhost:/staging-gfs /mnt/dockerswarmvol
sudo chown -R root:docker /mnt/dockerswarmvol
sudo reboot

You can now create new files in the directory /mnt/dockerswarmvol, which will then be synchronized to all machines.

5. Install Portainer

Portainer is a container for managing Docker containers via a graphical web interface.

Create a folder for the Portainer container (persistend data) (container volume):

!!! Only on machine dockerswm1 !!!:

sudo mkdir /mnt/dockerswarmvol/portainer

Now a YAML file (docker-compose) is created to start the Portainer container in the Swarm environment. Please use nano to create the file portainer.yaml file and fill it with the code contained in the following code block and save it.

!!! Only on machine dockerswm1 !!!:

nano portainer.yaml

Insert the following content into the nano editor, save the file and close the nano editor:

services:
  agent:
    image: portainer/agent:sts
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ce:sts
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
      - "9000:9000"
      - "8000:8000"
    volumes:
      - /mnt/dockerswarmvol/portainer:/data
    networks:
      - agent_network
    deploy:
      restart_policy:
        condition: any
        delay: 60s
        max_attempts: 3
        window: 180s
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]
networks:
  agent_network:
    driver: overlay
    attachable: true

Now the Portainer container is started with the following command:

!!! Only on machine dockerswm1 !!!:

sudo docker stack deploy portainer -c portainer.yaml

Now we access the Portainer web interface in a browser (https://192.168.0.99:9443) and we define a secure password for the admin user. We can call the Portainer web interface on all three IP addresses of the Docker Swarm but it does not matter which IP address we choose.

6. Install Nginx Proxy Manager

Nginx Proxy Manager is a modern HTTP reverse proxy that makes deploying microservices easy.

Now we need to create a overlay network on the docker swarm cluster like so:

!!! Only on machine dockerswm1 !!!:

sudo docker network create --driver overlay --attachable nginx_ingress

Create a folders for the Nginx Proxy Manager container (persistend data) (container volume):

!!! Only on machine dockerswm1 !!!:

sudo mkdir /mnt/dockerswarmvol/nginxproxymanagerstack
sudo mkdir /mnt/dockerswarmvol/nginxproxymanagerstack/npm_data
sudo mkdir /mnt/dockerswarmvol/nginxproxymanagerstack/npm_letsencrypt

First we open the Portainer web interface in our web browser:
https://192.168.0.100:9443
or
https://192.168.0.101:9443
or
https://192.168.0.102:9443
It does not matter which IP address you use to access the Portainer web interface.

Now we create a stack: The name of the stack can be freely defined.

Now we insert the content of the following code-block in the web editor of Portainer and deploy the stack (docker-compose).

services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    volumes:
      - /mnt/dockerswarmvol/nginxproxymanagerstack/npm_data:/data
      - /mnt/dockerswarmvol/nginxproxymanagerstack/npm_letsencrypt:/etc/letsencrypt
    networks:
      - nginx_ingress

networks:
  nginx_ingress:
    external: true

Now we access the Nginx Proxy Manager web interface in a browser (http://192.168.0.99:81). Use the default email admin@example.com and default password changeme to access the configuration page. Immediately after logging in with this default user you will be asked to modify your details and change your password.

7. Perform port forwarding on your router

Now all that remains is to set up and enable port forwarding on your router for port 80 and 443 for the virtual IP address 192.168.0.97.

Also setup a dynamic DNS on your router. There are many dynamic DNS provider ( https://www.noip.com, https://www.duckdns.org, ...) that can translate your public IP address of your router to a predefined Domain name you can define.

Comparison of different dynamic DNS provider: https://www.ionos.de/digitalguide/server/tools/dyndns-anbieter-im-ueberblick/

8. (Optional) Set up demo web server as service

In this step, we will deploy a demo web server as a service in the docker swarm using Portainer and Nginx Proxy Manager.

Now we create a stack: The name of the stack can be freely defined.

Please always use the nginx_ingress overlay network for ALL your services, so that Nginx Proxy Manager can communicate with your services

If the service requires a persistent volume you need to create a folder in the GlusterFS shared folder BEFORE you deploy the stack via portainer

Example:

sudo mkdir /mnt/dockerswarmvol/<FOLDER_NAME>

Example part inside the YAML config:

services:
...........................................
   volumes:
     - /mnt/dockerswarmvol/<FOLDER_NAME>:/var/www/html
...........................................

Now we insert the content of the following code-block in the web editor of Portainer.

services:
  webserver:
    image: nginxdemos/hello
    networks:
      - nginx_ingress
    ports:
      - 8082:80
networks:
  nginx_ingress:
    external: true

Now we access the Nginx Proxy Manager web interface http://192.168.0.99:81

In the top menu bar click on the option Hosts and then on Proxy Hosts. Now we need to click on the button Add Proxy Host on the right side of the empty table

Please modify the Domain name webserver.mydnydns.com to youre own domain name!!!

Because the Nginx Proxy Manager-container and the webserver-container

services:
  webserver: <---- This is the service-name of the container
...............

(demo web server) are using the same overlay network, the service-name and the internal port of the demo web server container is used in the proxy-host-config like so:

Previous Post Next Post

Add a comment

Name *

Comment *