Enabling Puppet Intermediate CA for Enhanced Security

In this blog, we will learn about enabling Puppet Intermediate CA (Certificate Authority) to enhance its security.

Prerequisite

• Require to install Puppet Server (Puppet 8) – You can refer to Setup Puppet 8 on Ubuntu 24.04 – Configuration Management for a scaling enterprise if you haven’t set up yet.

Introduction

Remember, when we launch a new server and want to register it as a Puppet Agent with the Puppet Server (formerly known as Puppet Master). On the Puppet Server, we need to sign the certificate sent from the new server to approve it. Simply by running:

sudo /opt/puppetlabs/bin/puppetserver ca sign --certname new-hostname.example.com

That’s when the new server can retrieve its own configuration catalogs from the Puppet Server.

You may wonder what that process is. And why we need to enable the Intermediate CA.

Basically, Puppet Server supports two types of CA architectures:

• Simple CA (by default): Puppet Server uses a self-signed root cert. When a Puppet Agent generates a certificate itself, and sends this cert to the Puppet Server to approve. Puppet Server uses this self-signed root cert as a CA signing cert to approve the Agent’s generated certificate.
• Intermediate CA (more secure): Also uses the self-signed root cert, but issuing an intermediate CA cert to sign incoming certificate requests instead of using the self-signed root cert to sign.

Configuring Intermediate CA

To configure the Intermediate CA, we have two cases:

Case 1: With a fresh installation and the Puppet server wasn’t started

If you just installed a fresh Puppet Server and haven’t started puppetserver.service yet, simply run:

sudo /opt/puppetlabs/bin/puppetserver ca setup

Case 2: the puppetserver.service was started already

If you have started the Puppet Server already and have also even signed a request from a Puppet Agent to let it connect to the Puppet Server, when you run the sudo /opt/puppetlabs/bin/puppetserver ca setup, you might encounter this error:

That’s because when we started puppetserver.service at the beginning, Puppet automatically generated a self-signed root certificate and used the default Simple CA architecture.

Therefore, to correct this setup, we need to do the following:

• Stop the puppetserver.service

sudo systemctl stop puppetserver.service

• Delete the SSL and CA directories, which include all old CA-related files and the certificates signed by the old self-signed root certificate.

sudo rm -rf /etc/puppetlabs/puppet/ssl
sudo rm -rf /etc/puppetlabs/puppetserver/ca

If you have signed a lot of certificates already, I would recommend backing up these directories to another place before deleting them. Just in case you mess up and don’t know how to fix it, you will be able to recover then.

• Regenerate the CA (self-signed root certificate) and the primary server’s cert:

sudo /opt/puppetlabs/bin/puppetserver ca setup

Output:

Generation succeeded. Find your files in /etc/puppetlabs/puppetserver/ca

• Verify that your CA and the primary server’s cert are generated:

sudo ls -l /etc/puppetlabs/puppet/ssl/
sudo ls -l /etc/puppetlabs/puppetserver/ca/

• Start puppetserver.service back

sudo systemctl start puppetserver.service

Ok. Now we have a Puppet Server with the Intermediate CA set up.

Configuring Agents

For now, if you launch a new server, you can perform the signing certificate process as usual. However, for the servers with old certificates that were signed by the old CA, they won’t work at the moment and may show “Error: certificate verify failed…”, for example:

To fix this, we need to delete the old certificates on these agents and send a new request for a new certificate to the Puppet server so that the Puppet server signs this certificate with the new CA.

On agents, we do the following steps:

• Stop puppet.service:

sudo systemctl stop puppet.service

# OR - Choose one of these ways

sudo puppet resource service puppet ensure=stopped

• Find the Puppet SSL directory on the Agent, simply run:

sudo /opt/puppetlabs/puppet/bin/puppet config print ssldir --section agent

Output (here is an example; your SSL directory may be different depending on your setup):

/etc/puppetlabs/puppet/ssl

• We need to delete this SSL directory, run:

sudo rm -rf /etc/puppetlabs/puppet/ssl

• Re-generate a new certificate and wait for the Puppet Server to sign. Run:

sudo /opt/puppetlabs/bin/puppet agent -t

Output may look like:

Info: Refreshed CRL: 1F:D9:02:B7:3B:0A:91:26:80:11:36:D1:69:89:63:87:7B:EB:A3:2A:31:CE:46:A7:4E:C9:BE:31:EA:2A:1B:0D
Info: Creating a new RSA SSL key for web-01.srv.local
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for web-01.srv.local
Info: Certificate Request fingerprint (SHA256): 49:E4:3B:22:EF:5F:A0:A3:B3:9E:FE:BA:AD:DC:69:7B:83:50:F7:81:33:39:4C:B8:AE:33:FE:C2:EB:06:BA:C6
Info: Certificate for web-01.srv.local has not been signed yet
Couldn't fetch certificate from CA server; you might still need to sign this agent's certificate (web-01.srv.local).
Exiting now because the waitforcert setting is set to 0.

• Log in to the Puppet Server and sign the new certificates sent from agents. Firstly, list all waiting certificates by running:

sudo /opt/puppetlabs/bin/puppetserver ca list

It will show all the available certificates waiting to be signed. Then, we run (replace the certname below with your agent’s certname):

sudo /opt/puppetlabs/bin/puppetserver ca sign --certname <node-name>.example.com

# OR sign all in case you make sure there is no compromised certificate appears

sudo /opt/puppetlabs/bin/puppetserver ca sign --all 

Should sign each certname showing up through puppetserver ca list

• After that, we go back to the agents and run sudo /opt/puppetlabs/bin/puppet agent -t again to test, and remember to start the puppet.service back.

sudo systemctl start puppet.service

# OR - Choose one of these ways

sudo puppet resource service puppet ensure=running

That’s all we need to do. Now we have a more secure way to protect the system.