Automating Nagios Host and Service Discovery with Consul

Following Setting Up Nagios Core with Nginx on Ubuntu 24.04 blog, configuring Nagios monitoring for a host (whether a VM, bare-metal server, etc.) can be a tedious process. Each host requires multiple boilerplate configuration templates just to monitor its basic metrics and services. This approach quickly becomes inefficient as the number of hosts grows — for example, when managing more than ten hosts. In a dynamic cloud environment, where hosts are frequently resized or replaced, repeatedly creating and adding configuration files for new instances becomes even more time-consuming and prone to human error.

Automation Methods

There are a few ways to automate the Nagios Host and Service Discovery process, and it depends on which Nagios version you’re using.

• Nagios Core:
  • Consul discovery: service registry–based discovery (metadata-driven). Install Consul Agent on all hosts and let Nagios fetch the node’s details from the Consul server.
  • SNMP discovery: network-based discovery (polling hosts/devices).
  • Configuration Management: Use Puppet, Ansible, Chef, or similar tools. We then define Nagios configuration templates and use these tools to apply them to target hosts. Changes apply only when config management runs (not real-time). That means Nagios only realizes the host after config management tools apply pre-defined configurations. I actually don’t prefer this way and would rather use this method to install Consul/SNMP on the host, and implement either Consul or SNMP for Nagios configurations.
  • Configuration Management Database: contains authoritative host/service metadata. It’s known for large enterprises. One of the famous providers is ServiceNow. I never use CMD, so I don’t have much to discuss. Just share what I’ve found if you’re interested.
  • Custom Script or APIs: Develop a custom script (by Python/Golang/Bash, etc) to automate the process with your own logic. It’s more prone to error with this kind of implementation due to different platforms. There are other caveats that we may face along the way.
• Nagios XI: Everything from Nagios Core plus:
  • Nagios Auto-Discovery Addons: Nagios agents (like NCPA) can register themselves with the Nagios server upon installation, sending host data automatically.

In this blog, I want to share a use case of utilising Consul so that we can detect a host/service in real time and automate the generation of Nagios configuration files for monitoring this host. And it aims to support Nagios Core.

Requirements

• Require knowledge of Consul and Nagios setup. At least you know how to install and configure Consul in server/agent mode, and also know how to add a host manually to Nagios.
• Have set up a Nagios server. If you haven’t, please visit this blog for the setup guide.
• Have built a Consul Cluster of at least 3 nodes. If you want to test, you can build a standalone Consul server (running in dev mode). You may want to look at sections 2 and 3 in Consul Service Discovery: Automate Nginx Upstream updates for dynamic load balancing blog to know more about how to set up a single Consul server. I also wrote this Puppet Series – Automate Consul Cluster Setup to build a Consul Cluster with Puppet in case you’re interested. Otherwise, please consult HashiCorp documentation for more information.
• Prepare an Ubuntu 24.04 server (I call it app-01), and we will add this server as a host where Nagios will monitor it.
• Nagios and app-01 should have Consul Agent installed, and they have joined the Consul Cluster.

I’ve already used Puppet to deploy and configure the Consul servers, as well as to install the Consul agent on all other server nodes. Therefore, I won’t cover the installation steps in this post. Instead, this blog focuses on how to use Consul to detect whether a host is alive and automatically generate Nagios configuration files for it.

To allow Consul to create these files automatically, we still need a proper automation process to set up Consul on each host so that it registers itself with the Consul cluster. This setup can be done using tools like Puppet, Ansible, Chef, or any similar automation framework. I’ll walk through the manual steps to demonstrate the process, and later, you can choose whichever tool best fits your workflow.

Additionally, I wrote Puppet Series – Automating Nagios Setup and Auto-Discovery Using Consul to demonstrate the use case of Puppet in automating all steps that I will outline below. After you go through all the steps here, feel free to check it out to know more if you’re interested.

(Optional) Verify Consul Installation

We can skip this step if you have Consul installed successfully. Basically, if you installed Consul on consul-0* servers (in server mode) and on app-01 and nagios-server (in agent mode), you can access your Consul Web UI to check whether the server nodes are there. My link: http://consul-01.srv.local:8500/ui/dc1/nodes (replace the URL with your IP or your own domain). Here is mine:

Note: I’m running a local DNS that allows me to use domain instead of IP address. If you want to have local DNS server so that you won’t have to enter your IP address, please check out How to Set Up a Local DNS Server with Dnsmasq in Ubuntu 24.04 (Fast & Easy) to facilitate your testing.

As you see, all servers have joined the Consul cluster. At the moment, nagios-server server is not configured to query Consul cluster to retrieve the nodes’ metadata, which is information about all server nodes.

(Optional) Set up NRPE and nagios-plugins package

So, to monitor a few basic resources (load, disk, processes, etc) on the remote host, nagios-server needs to run programs (plugins) on the host remotely so that it can retrieve results of resources that we define to monitor. To allow nagios-server to run those programs remotely, we need to install NRPE (Nagios Remote Plugin Executor) and nagios-plugins on the remote hosts and on the nagios-server itself.

Basically, NRPE is a client-side daemon that enables Nagios server to execute plugins on remote systems, while nagios-plugins is the package that refers to a collection of standalone executables or scripts that enable Nagios Core to monitor various aspects of an IT infrastructure such as system resources, services, and applications etc.

You can skip this section if you have installed NRPE and nagios-plugin packages.

On app-01 and consul servers , to install both packages, we run:

sudo apt update
sudo apt install nagios-nrpe-server nagios-plugins -y

On nagios-server server, in the previsous blog, because I installed Nagios and nagios-plugins from source code. Therefore, I just need nagios-nrpe-server and nagios-nrpe-plugin packages in this case.

sudo apt update
# nagios-nrpe-plugin provides us "check_nrpe" script which doesn't include in nagios-plugins package.
sudo apt install nagios-nrpe-server nagios-nrpe-plugin -y

We can also install NRPE from source, and you can check out this doc for more information. I install them through APT to make it faster without the compilation process.

All executable files from the nagios-plugins package will be stored in /usr/lib/nagios/plugins/ by default.

Now, on all servers, change the default NRPE config to allow nagios-server server to talk to NRPE on remote hosts, and also pass arguments (if have) when Nagios runs the checks. Simply run:

# Change IP to the Nagios server's IP. 
sudo sed -i '/^allowed_hosts=/s/$/,192.168.68.54/' /etc/nagios/nrpe.cfg
# Allow Nagios server to pass arguments to the remote host.
sudo sed -i 's/^dont_blame_nrpe=.*/dont_blame_nrpe=1/g' /etc/nagios/nrpe.cfg

The NRPE loads the config file from /etc/nagios/nrpe.cfg. I will use the default checks from this file to test, but we need to adjust the disk check.

sudo nano /etc/nagios/nrpe.cfg

and locate the lines that contain the following:

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -r -w .15,.10,.05 -c .30,.25,.20
# Comment this command[check_hda1] and create your own check_disk. My disk is /dev/sda2
# command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_sda2]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/sda2
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

It should be modified as below:

These are checks we will generate automatically later. If you want to enable more checks, please look at the commented parts of other #command[check_options] parts or search in the internet for more examples.

Restart nagios-nrpe-server service:

sudo systemctl restart nagios-nrpe-server.service
sudo systemctl status nagios-nrpe-server.service

Output:

On nagios-server server, we also need to add Nagios Command Definition for /usr/lib/nagios/plugins/check_nrpe script to the commands.cfg file so that Nagios knows where to load check_nrpe script.

My Nagios loads Command Definitions from /usr/local/nagios/etc/objects/commands.cfg, you will have to find which file corresponds to your config. Run:

sudo nano /usr/local/nagios/etc/objects/commands.cfg

And append this definition to the content:

define command {
    command_name    check_nrpe
    command_line    /usr/lib/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -t 60 -c $ARG1$
}

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

Reload Nagios service:

sudo systemctl reload nagios

If you use Puppet/Ansible/Chef/etc, you would want to include these steps in the configuration management tool so that it will automatically install on any host that you provision.

Configure Nagios Host Auto-Discovery

Consul allows us to retrieve the node’s metadata, but to really generate Nagios configuration files with this metadata, we need to install consul-template, a binary file (a CLI tool) that makes API calls to the Consul server to get server node details as well as service details running on those nodes.

Therefore, in this part, we will do three steps:

• Install consul-template on nagios-server server
• Create Nagios template files with an extension .ctmpl, e.g. nagios-hosts.cfg.ctmpl that are managed by consul-template. These files include some pieces of consul-template‘s language so that it can parse the corresponding node’s metadata when generating real Nagios config files. We also have to configure consul-template‘s config file to watch for these template files.
• Edit the main Nagios config file (nagios.cfg) to add another cfg_file path. This is to instruct Nagios to load generated config files from the folder that consul-template will produce.

Just noted that my nagios-server at the moment only monitor itself and web-01 server:

You can ignore the web-01 host as I manually defined there.

Step 1: Install consul-template on nagios-server server

On nagios-server server, run:

# Update APT
sudo apt-get update -y
  
# Download GPG key
wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
  
# Register APT source in order to install "consul" through APT
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
  
# Update APT again and install consul-template
sudo apt update && sudo apt install consul-template -y

The above commands are for Ubuntu/Debian. If you want to install consul-template for other OS platforms, please see this installation doc.

Step 2: Create Nagios config template files and configure consul-template

Create Nagios config template files

Next, we create a /opt/nagios-templates directory where we store Nagios‘s config template files managed by consul-template. These files are to tell consul-template to watch servers’ details through the Consul servers and produce real Nagios config files.

Still on nagios-server server, run:

# Create a directory to store Nagios config template files 
sudo mkdir -p /opt/nagios-templates

We continue creating two template files, one for the Host Definition config template and the other for the Service Definition config template. I’ll name them nagios-hosts.cfg.ctmpl and nagios-common-services.cfg.ctmpl, respectively.

Create /opt/nagios-templates/nagios-hosts.cfg.ctmpl file

sudo nano /opt/nagios-templates/nagios-hosts.cfg.ctmpl

And add the following content, please change the value “all-servers” in the use field to your Name of host template to use:

## This file is managed and matained by consul-template
{{ range nodes }}
define host {
    address                        {{.Address}}
    alias                          {{.Node}}
    host_name                      {{.Node}}
    use                            all-servers
}
{{ end }}

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

Create /opt/nagios-templates/nagios-common-services.cfg.ctmpl file:

sudo nano /opt/nagios-templates/nagios-common-services.cfg.ctmpl

And add the following content, please change value “hosts-service” in the use field to your Name of service template to use:

{{ range nodes }}
define service {
    check_command                  check_nrpe!check_users
    host_name                      {{.Node}}
    notification_period            24x7
    service_description            Check users login
    use                            hosts-service
}

define service {
    check_command                  check_nrpe!check_load
    host_name                      {{.Node}}
    notification_period            24x7
    service_description            Check load service
    use                            hosts-service
}

define service {
    check_command                  check_nrpe!check_sda2
    host_name                      {{.Node}}
    notification_period            24x7
    service_description            Check disk space
    use                            hosts-service
}

define service {
    check_command                  check_nrpe!check_zombie_procs
    host_name                      {{.Node}}
    notification_period            24x7
    service_description            Check Zombie Procs
    use                            hosts-service
}

define service {
    check_command                  check_nrpe!check_total_procs
    host_name                      {{.Node}}
    notification_period            24x7
    service_description            Check Total Procs
    use                            hosts-service
}

{{ end }}

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

Set up configurations for consul-template

Next, create a main config file for consul-template to instruct it to watch both of the created template files above.

# Create a directory to store main config file of consul-template 
sudo mkdir -p /etc/consul-template
# Create main config file for consul-template
sudo touch /etc/consul-template/config.hcl
# Grant appropriate permissions to "consul" user to read the main config as well as the template files
sudo chown -R consul:consul /etc/consul-template
sudo chown -R consul:consul /opt/nagios-templates
sudo chmod 644 /etc/consul-template
sudo chmod 644 /opt/nagios-templates

And run:

sudo nano /etc/consul-template/config.hcl

And add the following content:

consul {
  address = "localhost:8500"

  retry {
    enabled  = true
    attempts = 15
    backoff  = "250ms"
  }
}

template {
  source      = "/opt/nagios-templates/nagios-hosts.cfg.ctmpl"
  destination = "/usr/local/nagios/etc/objects/consul_sd_configs/nagios-hosts.cfg"
  perms       = 0644
  command     = ["systemctl", "reload", "nagios"]
}

template {
  source      = "/opt/nagios-templates/nagios-common-services.cfg.ctmpl"
  destination = "/usr/local/nagios/etc/objects/consul_sd_configs/nagios-common-services.cfg"
  perms       = 0644
  command     = ["systemctl", "reload", "nagios"]
}

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

Now, we continue creating /etc/systemd/system/consul-template.service, a systemd service config file for controlling consul-template operations, run:

sudo nano /etc/systemd/system/consul-template.service

And add the following content:

[Unit]
Description=Consul-Template Daemon Service
Wants=basic.target
After=basic.target network.target

[Service]
User=consul
Group=consul
ExecStart=sudo /usr/bin/consul-template \
  -config /etc/consul-template
SuccessExitStatus=12
ExecReload=/bin/kill -SIGHUP $MAINPID
ExecStop=/bin/kill -SIGINT $MAINPID
KillMode=process
Restart=always
RestartSec=42s
LimitNOFILE=4096

[Install]
WantedBy=multi-user.target

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

To reload systemd to load a new service configuration file:

sudo systemctl daemon-reload

Remember to check the real path of the consul-template command by simply running which consul-template. I installed consul-template through apt, so it’s /usr/bin/consul-template in my case. If you manually download the consul-template binary file, it should correspond to the path you put the binary file in, e.g., /usr/local/bin/consul-template.

Then add sudo permission to allow the consul user to run consul-template with sudo permission. Run:

sudo nano /etc/sudoers.d/sudo-consul

And add this content:

consul ALL = (root) NOPASSWD: /usr/bin/consul-template *

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

Make sure you can execute consul-template as sudo user under the consul user. Run this command to test:

sudo -u consul sudo consul-template --version

If it outputs consul-template’s version without asking for password, then you’re good.

Step 3: Update the main Nagios config file

We’re instructing consul-template to produce Nagios’s generated config files to /usr/local/nagios/etc/objects/consul_sd_configs/. Therefore, we need to edit the main Nagios config file to let it know where to load these generated files.

Firstly, we create a directory to store configuration files generated by consul-template in Nagios’s config path. Depending on where your Nagios main configuration path is, my main Nagios path is at /usr/local/nagios/etc, so I run:

sudo mkdir -p /usr/local/nagios/etc/objects/consul_sd_configs/

Next, we edit the main Nagios config file:

sudo nano /usr/local/nagios/etc/nagios.cfg

Find the cfg_file field, and append the following line to it

cfg_file=/usr/local/nagios/etc/objects/consul_sd_configs/

Output example:

Press Ctrl + O and press Enter to save the content. Then press Ctrl + X to exit.

Now, we start consul-template.service so that it automatically generates all Host and Service definitions for server nodes. Run:

sudo systemctl start consul-template
sudo systemctl enable consul-template
sudo systemctl status consul-template

Once started consul-template.service, if you check the nagios-hosts.cfg file by running:

sudo cat /usr/local/nagios/etc/objects/consul_sd_configs/nagios-hosts.cfg

Output:

We have all hosts defined in Host Definition blocks automatically generated.

We will have a similar result in nagios-common-services.cfg file, run:

sudo cat /usr/local/nagios/etc/objects/consul_sd_configs/nagios-common-services.cfg

If I access Nagios server’s URL: http://nagios-server.srv.local/. I will see that all hosts are appearing there with service checks that are defined in nagios-common-services.cfg.

Congratulations! You have automated the creation of Nagios config files for each host. As you’ve seen, we don’t manually create Nagios config files for each host. All is generated once the host has Consul set up and joined the Consul cluster. Nagios then uses consul-template to fetch the corresponding node’s details and generate Nagios config files for active hosts.

As I mentioned, Consul itself doesn’t solve all manual processes when configuring Nagios for a host. We still have to set up Consul on each host to let Consul do its job. However, the setup of Consul can be achieved through any configuration management tool. That’s when you build a robust system that you can reuse for any host. Hope it helps you form some ideas in automating your monitoring system.