Setup a Docker Swarm cluster Part VII - CI/CD workflow

This is the Part VII of more global topic tutorial. Back to first part for intro.

Self-hosted VCS 🍵 #

This specific VCS part is optional and is only for developers that would be completely independent of any cloud VCS providers, by self-hosting its own system.

Of course, in this low-budget cluster, forget about running properly a self-hosted GitLab, you will be forced to have an additionnal worker node with at least 4Gb fully dedicated just for running it. I will privilege here a super lightweight solution, Gitea. Besides, the last version 1.16 finally support dark mode !

Install Gitea 💽 #

You guess it, it’s just an additional stack to run !

Do sudo mkdir /mnt/storage-pool/gitea and create next stack :

version: '3.8'

services:
  gitea:
    image: gitea/gitea:1.16
    volumes:
      - /etc/hosts:/etc/hosts
      - /mnt/storage-pool/gitea:/data
    networks:
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.gitea.entrypoints=https
        - traefik.http.services.gitea.loadbalancer.server.port=3000
        - traefik.tcp.routers.gitea-ssh.rule=HostSNI(`*`)
        - traefik.tcp.routers.gitea-ssh.entrypoints=ssh
        - traefik.tcp.services.gitea-ssh.loadbalancer.server.port=22
      placement:
        constraints:
          - node.role == manager

networks:
  traefik_public:
    external: true

Now go to https://gitea.sw.dockerswarm.rocks and go through the installation procedure. Change default SQLite provider by a more production purpose database.

Create a new gitea PostgreSQL database as usual from pgAdmin or psql for pro-CLI user, and set the according DB info access to Gitea installer. Host should be data-01.

Don’t forgive to change all domain related field by the proper current domain URL. You should set proper SMTP settings for notifications.

For information all these settings will be saved in /mnt/storage-pool/gitea/gitea/conf/app.ini file. You can change them at any time. You may want to disable registration by changing DISABLE_REGISTRATION.

Next just create your first account. The 1st account will be automatically granted to administrator.

You should now test creating some repos and be sure that git cloning works on both HTTPS and SSH protocol. For SSH be sure to add your own SSH public key in your profile.

Private docker registry #

Before attack the CI/CD part, we should take care of where we put our main docker images that will be automatically be built when every code pushes. You have the choice to use main Docker hub of course but honestly, we have a full cluster now, let’s use it fully !

Install official docker registry 💽 #

We’ll use the official docker registry with addition of nice simple UI for images navigation. It’s always the same, do sudo mkdir /mnt/storage-pool/registry and create next stack :

version: '3.8'

services:
  app:
    image: registry:2
    environment:
      REGISTRY_STORAGE_DELETE_ENABLED: 'true'
    volumes:
      - /mnt/storage-pool/registry:/var/lib/registry
    networks:
      traefik_public:
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.registry.entrypoints=https
        - traefik.http.routers.registry.middlewares=admin-auth
        - traefik.http.routers.registry.rule=Host(`registry.sw.dockerswarm.rocks`) && PathPrefix(`/v2`)
        - traefik.http.services.registry.loadbalancer.server.port=5000
      placement:
        constraints:
          - node.role == manager

  ui:
    image: joxit/docker-registry-ui
    environment:
      DELETE_IMAGES: 'true'
      SINGLE_REGISTRY: 'true'
    networks:
      traefik_public:
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.registryui.entrypoints=https
        - traefik.http.routers.registryui.middlewares=admin-auth
        - traefik.http.services.registryui.loadbalancer.server.port=80
      placement:
        constraints:
          - node.role == manager

networks:
  traefik_public:
    external: true

Go to https://registry.sw.dockerswarm.rocks and use Traefik credentials. We have no images yet let’s create one.

Test our private registry #

Create a Dockerfile sample file somewhere :

FROM alpine:latest
RUN apk add --no-cache git

Then build and push the image :

# enter proper credentials
docker login registry.sw.dockerswarm.rocks

# build the image
docker build -t alpinegit .
docker tag alpinegit registry.sw.dockerswarm.rocks/alpinegit
docker push registry.sw.dockerswarm.rocks/alpinegit

Go back to above https://registry.sw.dockerswarm.rocks. You should see 1 new image !

Delete the image test through UI and from local docker with docker image rm registry.sw.dockerswarm.rocks/alpinegit.

Register registry in Portainer #

For our future app deployments from our built docker image, we need to register our new private registry with proper credentials. Go to Registries menu of Portainer, and create the new registry.

Save it, and you have registered repository, which will allow proper pulling from it for next custom app stack deployments.

CI/CD with Drone 🪁 #

It’s finally time to use our currently unused runner-01 ! We’ll use Drone as free self-hosted solution for docker image building. The all CI/CD process can be summarized to this flow chart :

Let’s follow the official docs for generating a OAuth2 application on Gitea, which is necessary for Drone integration. Set https://drone.sw.dockerswarm.rocks as redirect UI after successful authentication.

Save and keep the client and secret tokens. Then create a new drone PostgreSQL database and create next stack :

version: '3.8'

services:
  drone:
    image: drone/drone:2
    volumes:
      - /etc/hosts:/etc/hosts
    environment:
      DRONE_DATABASE_DRIVER: postgres
      DRONE_DATABASE_DATASOURCE: postgres://drone:${DRONE_DATABASE_PASSWORD}@data-01:5432/drone?sslmode=disable
      DRONE_GITEA_CLIENT_ID:
      DRONE_GITEA_CLIENT_SECRET:
      DRONE_GITEA_SERVER: https://gitea.sw.dockerswarm.rocks
      DRONE_RPC_SECRET:
      DRONE_SERVER_HOST:
      DRONE_SERVER_PROTO:
      DRONE_USER_CREATE: username:adr1enbe4udou1n,admin:true
    networks:
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.drone.entrypoints=https
        - traefik.http.services.drone.loadbalancer.server.port=80
      placement:
        constraints:
          - node.role == manager

  runner-docker:
    image: drone/drone-runner-docker
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      DRONE_RPC_SECRET:
      DRONE_RPC_HOST: ${DRONE_SERVER_HOST}
      DRONE_RPC_PROTO: ${DRONE_SERVER_PROTO}
    deploy:
      placement:
        constraints:
          - node.labels.environment == build

networks:
  traefik_public:
    external: true

It’s time to go to https://drone.sw.dockerswarm.rocks and generate your first Drone account through OAuth2 from Gitea. You should be properly redirected to Gitea, where you’ll just have to authorize Drone application.

Finalize registration, and you should finally arrive to main Drone dashboard. If you have already created some repositories, they should appear in the list.

Test with basic project #

We have all the minimal CI/CD pipeline in place ! Let’s create a basic backend Dotnet API and test all CI/CD workflow !

Create new API project ✨ #

Firstly, create a new my-weather-api private repository through Gitea. I’ll use main as default branch here.

Next I presume you have .NET SDK locally installed. Create a new ASP.NET Core Web API project and push into Gitea.

dotnet new webapi -o my-weather-api --no-https

cd my-weather-api

dotnet new gitignore

git init
git add .
git commit -m "first commit"
git remote add origin [email protected]:adr1enbe4udou1n/my-weather-api.git # if you use ssh
git push -u origin main

Project should be pushed correctly :

Drone configuration 🛠️ #

Let’s now activate the repo in Drone. Click on SYNC button. Click on new repo and activate the repository.

It will create a webhook inside repository settings, triggered on every code push.

Now generate a new SSH key on manager-01 :

ssh-keygen -t ed25519 -C "[email protected]"
cat .ssh/id_ed25519 # the private key to set in swarm_ssh_key
cat .ssh/id_ed25519.pub # the public key to add just below
echo "ssh-ed25519 AAAA... [email protected]" | tee -a .ssh/authorized_keys

Then configure the repository settings on Drone. Go to Organization > Secrets section and add some global secrets.

Configure the pipelines (the CI part) 🏗️ #

For working, Drone needs a .drone.yml file in root of repository. This file will describe all steps of our build pipeline. Let’s create and explain it :

kind: pipeline
type: docker
name: default

steps:
  - name: build
    image: mcr.microsoft.com/dotnet/sdk:6.0
    commands:
      - dotnet publish -c Release -o ./publish

  - name: image
    image: plugins/docker
    settings:
      registry: registry.sw.dockerswarm.rocks
      repo: registry.sw.dockerswarm.rocks/adr1enbe4udou1n/my-weather-api
      tags: latest
      username:
        from_secret: registry_username
      password:
        from_secret: registry_password

trigger:
  event:
    - push
    - pull_request

It’s just simple 2 steps :

1. build : Here is the step for project dependencies import, compilation, testing, and code linting / formatting. This is a very basic project here, so we start with a simple building. The image mcr.microsoft.com/dotnet/sdk:6.0 is the required docker image for proper .NET building. The publish command will generate a publish subdirectory.
2. image : We use the official docker plugin for building our final production based docker image, done from the below Dockerfile. Use repo as the final docker image name.

Next create the Dockerfile which will be used for image step :

FROM mcr.microsoft.com/dotnet/aspnet:6.0
RUN apt-get install -y tzdata

COPY /publish /app
WORKDIR /app

ENTRYPOINT ["dotnet", "my-weather-api.dll"]

We use production suited .NET runtime image mcr.microsoft.com/dotnet/aspnet:6.0. Note as WE MUST do the simplest commands possible in order to have the lightest image layers, as it’s the production image. All we have to do is to copy the final published binaries from above build drone step.

Commit both above files and push to remote repo. Drone should be automatically triggered for building and activate the runner. The runner will clone the project and process all pipeline’s steps.

If all’s going well, the final image should be pushed in our docker registry. You can ensure it by navigating to https://registry.sw.dockerswarm.rocks.

Deployment (the CD part) 🚀 #

Our application is now ready for production deployment ! Let’s create our new shiny stack :

version: '3.8'

services:
  app:
    image: registry.sw.dockerswarm.rocks/adr1enbe4udou1n/my-weather-api
    environment:
      ASPNETCORE_ENVIRONMENT: Development
    networks:
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.my-weather-api.entrypoints=https
        - traefik.http.services.my-weather-api.loadbalancer.server.port=80
      placement:
        constraints:
          - node.labels.environment == production

networks:
  traefik_public:
    external: true

Finally, deploy and see the result in https://weather.sw.dockerswarm.rocks/swagger. You should access to the swagger UI, and API endpoints should correctly respond.

Continuous deployment #

Now it’s clear that we don’t want to deploy manually every time when the code is pushed.

First be sure that following docker service update --image registry.sw.dockerswarm.rocks/adr1enbe4udou1n/my-weather-api:latest weather_app --with-registry-auth command works well in manager-01. It’s simply update the current weather_app service with the last available image version from the private registry.

Now we must be sure that the runner-01 host can reach the manager-01 server from outside. If you have applied the firewall at the beginning of this tutorial, only our own IP is authorized. Let’s add the public IP of runner-01 to your firewall-external inside Hetzner console.

Now let’s add a new deploy step inside .drone.yml into our pipeline for automatic deployment !

#...
  - name: deploy
    image: appleboy/drone-ssh
    settings:
      host: sw.dockerswarm.rocks
      port: 2222
      username: swarm
      key:
        from_secret: swarm_ssh_key
      script:
        - docker service update --image registry.sw.dockerswarm.rocks/adr1enbe4udou1n/my-weather-api:latest weather_app --with-registry-auth
#...

Let’s test by editing Program.cs file :

builder.Services.AddSwaggerGen(c =>
{
    c.SwaggerDoc("v1", new OpenApiInfo
    {
        Title = "Weather API",
        Version = "0.0.1",
    });
});

Push and back to your API, and the title and version should be automatically updated !

6th check ✅ #

We’ve done all the main CI/CD part.

Now let’s go even further in the exploitation of our new cluster development environment in the last part.