Automate GitLab Runner Registration and Unregistration

Continuing from Configuring & Managing GitLab Runners with Reusable Templates: My Real-World Workflow, you might see that the operations of configuring and managing GitLab Runners seem to include so many manual steps.

Regarding my workflow in the previous blog above, I extended the custom Bash script to become a wrapper for the gitlab-runner binary combined with the GitLab REST API. It automates the process of creating and registering a new Runner, as well as the process of unregistering and deleting the created Runner.

There are many approaches to automate the process, and this depends on which platform (such as EC2 or GCP) you want to run the runner on.

I believe automation is essential in terms of operation and management, especially when you have a fleet of many runners.

Prerequisites

• Prepare a Linux server with Docker CE installed. I’m still using Ubuntu 24.04 in this demo, which plays the role of a host machine where multiple runners can run. It’s called the gitlab-runner machine.
• I’m using GitLab SaaS (https://gitlab.com) in this example. If you run a self-hosted GitLab, the process is almost the same. There’s a minor difference in the URL which you use to register a runner.

Other notes:

• For self-hosted GitLab only, you must be an administrator to create instance runners.
• To create group runners: Require you to be an administrator (for self-hosted) or have the Owner role (for SaaS) for the group.
• To create project runners: Require you to be an administrator (for self-hosted) or have the Maintainer role (for SaaS) for the project

Preparation

We will download and install the custom Bash script gl-runner-auto.sh that I’ve created for automating runner registration and unregistration. You can find it at https://gitlab.com/binhdt2611/gl-runner-auto.

On the host machine (is gitlab-runner server in my case), install package dependencies needed for the gl-runner-auto.sh script:

# Install package dependencies
sudo apt update
sudo apt install jq curl -y

# Set up the official GitLab repository
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
 
# Install the latest package
sudo apt install gitlab-runner -y

Then, download and set up the gl-runner-auto.sh script, run:

SCRIPT_PATH="/usr/local/bin/gl-runner-auto.sh"
sudo curl -fsSL "https://gitlab.com/binhdt2611/gl-runner-auto/-/raw/main/gl-runner-auto.sh?ref_type=heads" -o $SCRIPT_PATH > /dev/null
sudo chmod +x $SCRIPT_PATH

Can test whether the script has been installed successfully by running:

sudo gl-runner-auto.sh help

Output:

Basically, to automate the registration and unregistration processes, my script relies on the GitLab API to create/delete a runner, and uses the gitlab-runner binary to register/unregister the created runner.

Create a personal access token

It’s up to you to choose which type of access tokens you want to use. Ideally, if you just want to create runners for a project or a group, it’s best that you create an access token for the project or group, respectively. You can check out this guide for the creation of a project/group access token.

However, to reduce the complexity of managing multiple tokens, I’ll create a personal access token in this case, and this is just a personal choice.

Log in your GitLab account, and do the following:

1. On the left sidebar, select your avatar -> select Edit profile.
2. On the left sidebar, select Access tokens -> Select Add new token.
3. Enter a meaningful name and desired expiry date for the token. I can choose up to 400 days.
4. In the Select scopes section
   • If you just want to use my script to automate Registration process, select the create_runner checkbox.
   • If you want to automate Registration/Unregistration, select manage_runner as well. (I selected both)
5. Select Create.

Output:

Once you create it, please copy the generated personal (project/group) access token and save it in a secure place. For example, Keeper Password Manager or anywhere you think it’s safe. We will use this token in later steps.

Automate Runner Registration

For the Runner registration process, the previous blog shows us that we need to:

• Retrieve a runner authentication token. By creating a new Runner through GitLab Web UI.
• Run gitlab-runner with the retrieved authentication token to register the created Runner on the host machine.

The gl-runner-auto.sh script is to automate the above registration process, and also add a few custom operations. It simply does:

• Send a POST request to the API endpoint /user/runners to create a new runner (require personal/project/group access token).
• Retrieve a runner authentication token from the response of the POST request. The API request returns the fields: id, token, and token_expires_at.
• Saving the values of id, token to /home/<username>/.gitlab-runner/state-<runner-name>.json. With <runner-name> is a unique name of the created runner. We need a unique name for a runner because it’s for the unregistration process.
• Run gitlab-runner with the generated token to register the created runner.

Create template config files

We need to create config templates first. By default, if we use gitlab-runner to register a runner, we don’t really need to provide a template config file. It will automatically generate a default configuration for the runner. However, I prefer to manage runners with template config files rather than using default ones. That’s because it allows me to control how many resources I should allocate to runners rather than giving them unlimited resources. Also, the template config file allows us to specify custom configurations that a Runner should have.

In this example, I’ll create two template config files for demonstration.

• Template 1: For generic runners, we aim to use this config file for general-purpose runners.
• Template 2: For a specific runner, in this case is the runner we use to run db-migration tasks, and this runner should contain special configuration.

On the gitlab-runner server (host machine), run the following commands to create a directory where it stores runner template files:

# Create directory to store template files	
sudo mkdir -p /data/runner_template

Template 1: Create a generic template file for general-purpose runners, run

sudo nano /data/runner_template/generic-template.toml

And add the following content:

[[runners]]
  request_concurrency = 1
  environment = ["DOCKER_DRIVER=overlay2"]
  [runners.docker]
    memory = "256m"
    memory_swap = "256m"
    memory_reservation = "128m"
    cpus = ".1"

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

Runners that use template 1 don’t require much CPU/Memory, so I assume that’s enough for them.

Template 2: Create a specific template file for the runner that we aim to run the db-migration task.

sudo nano /data/runner_template/db-migration-runner-template.toml

And add the following content:

[[runners]]
  request_concurrency = 1
  environment = ["DOCKER_DRIVER=overlay2"]
  [runners.docker]
    memory = "512m"
    memory_swap = "512m"
    memory_reservation = "256m"
    cpus = ".2"
    volumes = ["/data/secret:/data/secret:rw"]

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

Because we use template 2 for the db-migration task, this runner should have higher memory/CPU than normal ones, and for example, should have a /data/secret volume on gitlab-runner server mounting to /data/secret inside a Docker container that the runner’s executor uses to run. The /data/secret volume contains database credentials that the application needs to execute db-migration tasks. Note that this is a simple example.

Register a project runner

Because I’m testing with GitLab SaaS, we’re not able to create an instance runner (they’re managed by GitLab), so I will test creating a project runner instead.

Still on gitlab-runner server, run:

sudo gl-runner-auto.sh register \
  --name demo-runner-1 \
  --template /data/runner_template/generic-template.toml \
  --token "PUT-YOUR-PERSONAL/PROJECT/GROUP-ACCESS-TOKEN" \
  --runner-type "project_type" \
  --project-id "your-project-id" \
  --tag-list general,test,dev \
  --max-timeout 1800

where:

• --name (optional): name of the registered runner. The script adds a suffix to the supplied name with a string of random numbers and letters to create a unique runner name. E.g. demo-runner-1-cff505618e8c.
• --template (required): is the path of the runner config template file.
• --token (required): is your personal/personal/group access token.
• --runner-type (optional): specify the type of runner that you want to create. Should be one of three values: instance_type, project_type, or group_type. Default to instance_type but it works for self-hosted GitLab only.
• --project-id (optional): is the ID of the project that you want the runner associate to. It’s required when we specify the runner type project_type. Similar to the group runner type, when we specify runner type is group_type, --group-id is required. Can’t specify both.
• --tag-list (optional): specify multiple tags separated with commas for the runner.
• --max-timeout (optional): set timeout in seconds for a job that uses this runner. Defaults to 3600 if not specified.

Output:

To know more about other options of sub-command register, run:

sudo gl-runner-auto.sh register -h

Next, check runner is created and registered successfully, run:

sudo gitlab-runner list

Output:

We see that the script has registered the runner demo-runner-1-7addf3dee305 successfully in my example.

Now, we go to the GitLab UI to check the created runner. On the project that we created the runner, select Settings > select CI/CD -> Expand the Runners section. I’ll have the result below:

A runner is created with the expected tag list.

If I want to create/register a second runner with the name e.g. demo-runner-2, I just change the –name option. And I want this second use /data/runner_template/db-migration-runner-template.toml, run:

sudo gl-runner-auto.sh register \
  --name demo-runner-2 \
  --template /data/runner_template/db-migration-runner-template.toml \
  --token "PUT-YOUR-PERSONAL/PROJECT/GROUP-ACCESS-TOKEN" \
  --runner-type "project_type" \
  --project-id "your-project-id"\
  --tag-list db-migration\
  --max-timeout 1800

We have a similar result, the second runner has tag “db-migration” is created:

Optional: an example to create a group runner, run:

sudo gl-runner-auto.sh register \
  --name demo-group-runner-1 \
  --template /data/runner_template/generic-template.toml \
  --token "PUT-YOUR-PERSONAL/GROUP-ACCESS-TOKEN" \
  --runner-type "group_type" \
  --group-id "your-group-id"\
  --tag-list "tag1,tag2,tag3"\
  --max-timeout 1800

Optional: an example to create an instance runner, run:

sudo gl-runner-auto.sh register \
  --name demo-instance-runner-1 \
  --template /data/runner_template/generic-template.toml \
  --token "PUT-YOUR-PERSONAL-ACCESS-TOKEN" \
  --url "https:://your-self-hosted-gitlab.example.com"

Testing

I’ve added two jobs to .gitlab-ci.yml with the content like below, and push the change to the remote repository:

stages:
  - build

# test:build uses demo-runner-1 by calling tags: 'dev'
test:build:
  stage: build
  tags:
    - dev
  script:
    - echo "Test registered runner - demo-runner-1"
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

# test:build uses demo-runner-2 by calling tags: 'db-migration'
test:db-migration:
  stage: build
  tags:
    - db-migration
  script:
    - echo "Test registered runner for db-migration task - demo-runner-2"
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

Result for test:db-migration job:

Result for test:build job:

Ok, we have successfully automated the runner registration process. If you just need to automate GitLab Runner registration, this part is enough for you. But feel free to explore further in the later part of the unregistration process.

Automate Runner Unregistration

In a manual way, if we want to unregister a runner, we normally run:

sudo gitlab-runner unregister --name <your-runner-name>

or

# Change url to your own url if you run a self-hosted GitLab instance
sudo gitlab-runner unregister --token <your-runner-authentication-token> --url https://gitlab.com

Both of commands just unregister the runner from the host machine but leaves the created runner remain exist on GitLab instance. If we want to delete the created runner, we have to log in the GitLab account, and go to the place where the runner is created, and manually delete it.

The gl-runner-auto.sh script also automates this process with sub-command unregistrer. The script simply does:

• It unregisters the registered runner on host machine via a supplied runner’s name.
• Retrieve the associated token to the unregistered token from /home/<username>/.gitlab-runner/state-<runner-name>.json
• Send a POST request to the API endpoint /runners to delete the runner by using its authentication token.

Unregister the registered runner

Before we unregister the runner, we need to get the runner’s name, simply run:

sudo gitlab-runner list

Grep the desired runner’s name from the output. In my case, I want to unregister demo-runner-1-7addf3dee305 and delete its associated runner.

To unregister the runner, run:

# Change value of --name to your desired runner's name
sudo gl-runner-auto.sh unregister --name demo-runner-1-7addf3dee305

Output:

Run sudo gitlab-runner list to check whether the runner is unregistered:

Output:

Ok, that has worked! We only see the demo-runner-2-d8b486c91a96 runner left.

Log in the GitLab UI to check the associated runner is deleted, we get:

We find that the runner with tags "general,test,dev" has been deleted successfully.

Noted: This workflow is just a personal way. It may not suit your preference if you want to reuse the authentication token to register the runner on multiple host machines. Feel free to take this as reference if it suits you.

Conclusion

With the workflow of managing runners above, each created runner will be used to register on a particular host machine. It helps clean up the stale runners when we unregister them without leaving them there while they’re not registered to any host. It has the limitation (maybe more than that) that if we reuse the authentication to register on another host machine, and later we use the gl-runner-auto.sh script to unregister the runner on the first host machine, it deletes the created runner while it’s being registered on the other host machines.

I think it’s quite dependent on how you manage your runners, so feel free to adjust the script according to your requirements. I’ll let your creativity evolve 🙂 . The main goal of this article is to provide a script to automate GitLab Runner registration, which we have achieved. Thanks for reading.