Managing Sensitive Data in Puppet Using Hiera-eyaml

Just another setup to improve the security of Puppet. When it comes to Puppet, Hiera is an integral part of it. It’s the place where you store the configuration data in key-value pairs, and look up what data a module requires for a node during catalog compilation

Configuration data can be plain-text data or sensitive data. However, you may wonder how to store sensitive data (such as passwords, SSH keys, tokens, etc.) properly in Hiera, where you can deploy this kind of data to Puppet agents securely? Hiera-eyaml is the answer for this.

According to https://github.com/voxpupuli/hiera-eyaml:

Hiera-eyaml is a backend for Hiera that provides per-value encryption of sensitive data within yaml files to be used by Puppet.

Prerequisite

• Have Puppet Server installed. In this blog, I’m using Puppet 8.

Noted: This blog continues the Puppet Series. I have a Puppet project in https://gitlab.com/binhdt2611/puppet-demo to demonstrate each blog I write. My Puppet uses r10k to deploy this project to environments located in the /etc/puppetlabs/code/environments/ directory on the puppet-master server. You can have a look at Setup Puppet 8 on Ubuntu 24.04 – Configuration Management for a Scaling Enterprise and Mastering Puppet: Implementing Roles and Profiles Effectively in Reality to set up if you don’t know how to set up Puppet 8 Server/Agent

Installing Hiera-eyaml

Puppet Server, since version 5.2.0, has hiera-eyaml included already. So, if you followed me in the previous blogs to install Puppet 8. On your puppet-master server, you can find eymal in /opt/puppetlabs/puppet/bin/, simply run:

sudo /opt/puppetlabs/puppet/bin/eyaml --help

Output:

(Alternative) If you are running Puppet older than version 5.2.0, please check their hiera-eyaml documentation for alternative installation methods.

The most common way is just to run (should run as ‘root’):

/opt/puppetlabs/bin/puppetserver gem install hiera-eyaml

Create public and private keys

Next, we need to create a set of public/private keys that Hiera will use to encrypt/decrypt YAML data.

On the puppet-master server (the server with Puppetserver installed), simply run:

sudo /opt/puppetlabs/puppet/bin/eyaml createkeys

Output:

The set of key pairs is created in ./keys. We can check it by running ls -l keys. It should show:

We need to store these keys securely in a safe place and should not publish them to the repository. Also, we should move these key files to a suggested directory, which normally is /etc/puppetlabs/puppet/eyaml or /var/lib/puppet/keys. We will use the /etc/puppetlabs/puppet/eyaml directory as I’m using Ubuntu.

On the puppet-master server, simply run the following command to set up:

sudo mkdir -p /etc/puppetlabs/puppet/eyaml
sudo mv keys/*.pem /etc/puppetlabs/puppet/eyaml/
sudo chown -R puppet:puppet /etc/puppetlabs/puppet/eyaml
sudo chmod -R 0500 /etc/puppetlabs/puppet/eyaml
sudo chmod 0400 /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
sudo chmod 0400 /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
sudo ls -lha /etc/puppetlabs/puppet/eyaml

Configure hiera-eyaml for environments

Now, we have a pair of pub/priv keys placed in the /etc/puppetlabs/puppet/eyaml directory. We need to configure hiera.yaml file to let Puppet know which path it should search for these keys when it looks up encrypted data, for example, in any data/example-encrypted.eyaml file.

Puppet loads hiera.yaml file in 3 different locations:

For production, it’s recommended to configure Hiera in the environment layer. That’s where we can utilise a version control system to store our Puppet Code Project, and use r10k to deploy this code project into the corresponding environments. As I mentioned earlier, I’m using https://gitlab.com/binhdt2611/puppet-demo as my code project, so under the project root path, I have hiera.yaml already configured. Just update it to support hiera-eyaml:

---
version: 5

defaults:
  datadir: data
  data_hash: yaml_data

hierarchy:
  - name: "Per-node data (yaml version) overrides all config defined in file below"
    path: "nodes/%{::trusted.certname}.yaml"
  
  # Other YAML hierachy levels .....
  
  # Here is an example of enabling hiera-eyaml for "eyaml" files
  - name: "Encrypted data for per-node and shared"
    paths:
      - "secrets/%{::trusted.certname}.eyaml"
      - "secrets/shared.eyaml"
    lookup_key: eyaml_lookup_key
    options:
      pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
      pkcs7_public_key:  /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem

  # Other YAML hierachy levels .....
  
  - name: "Other YAML hierarchy levels"
    paths:
      - "common.yaml"

In this example, any encrypted data is defined in data/secrets/%{::trusted.certname}.eyaml will override data defined in data/secrets/shared.eyaml

Now we need to commit and push all these changes to your remote repository and log in puppet-master to deploy by r10k. I’ve set a cron job to run r10k to deploy changes every 5 minutes to /etc/puppetlabs/code/environments/{production,<any-other-branches>} on puppet-master server for me.

If you want to manage Hiera configuration with Puppet, we also have a hiera-module developed by the community. You can refer to their documentation to learn more. To me, I just prefer to change hiera.yaml file manually and manage it by git instead, as once we configure, this is the part we rarely touch.

Testing

Although we can test encrypting/decrypting by eyaml command directly on the puppet-master. However, as a best practice, we should install hiera-eyaml gem in our local machine, and just copy the content of the /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem file to a file in our local working directory. That is to avoid typing eyaml command on the server with a supplied password where it can be revealed accidentally if your puppet-master is compromised, and someone may be able to see your password through history command.

Run sudo cat /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem to show the content or download this public key to your local machine.

Under the project root path (is the puppet-demo project in my case) in your local working directory, We create a .eyaml/ directory and put public_key.pkcs7.pem (or create this file and paste the content you copied into this) in this directory. Also, create .eyaml/config.yaml with content

--- 
pkcs7_public_key: '/path/to/your/project/.eyaml/public_key.pkcs7.pem'

This is to tell which public key eyaml should look for.

Remember to add .eyaml/ to your .gitignore file under the project root path so that we don’t commit this folder.

Once you’ve done that, it’s ready to use eyaml command to encrypt the sensitive data that you pass to it, and you will receive an encrypted block then. For example, I’m encrypting a password “PleaseChangeMe”, run:

eyaml encrypt -s PleaseChangeMe

Output:

string: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAW....
....IZIAWUDBAEqBBD1dZwLsOZzm6nyLDe7df7XgBAbRn+unnFygfc2UGEJkafs]

OR

block: >
  ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBAD
  AFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEANFBWX8J7Qd08zY/9WqCsmmzZt/
  ....
  ....
  T6/iknepHcW3Pu1TA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBD1dZwLsO
  Zzm6nyLDe7df7XgBAbRn+unnFygfc2UGEJkafs]

Next, you can copy the encrypted data from either string or block, and still under the project root path, create the data/secrets/shared.eyaml file. And add the content, for example:

---
# Choose one of two ways:
# If you use "string", paste the copied content like this
profiles::mysql::mysql_user: ENC[PKCS7,MIIBeQYJKoZIhv.....gfc2UGEJkafs]

# If you use "block", paste the copied content like this
profiles::mysql::mysql_user: >
  ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBAD
  AFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEANFBWX8J7Qd08zY/9WqCsmmzZt/
  ....
  ....
  T6/iknepHcW3Pu1TA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBD1dZwLsO
  Zzm6nyLDe7df7XgBAbRn+unnFygfc2UGEJkafs]

Ok. We’re done. Now you just need to commit your changes and push them to the VSC system (mine is GitLab) to let r10k deploy again.

We can now test the puppet lookup command to see whether it returns the correct password.

# Note: Because I created a "enable_hiera_eyaml" branch and pushed changes 
# to this branch to test. 
# Later I'll merge this branch into "production", that's when we can just run:
# 
#   $ sudo /opt/puppetlabs/puppet/bin/puppet lookup profiles::mysql::mysql_user
# 
sudo /opt/puppetlabs/puppet/bin/puppet lookup profiles::mysql::mysql_user --environment enable_hiera_eyaml

Output:

That’s all for this blog.

References

• https://www.puppet.com/blog/puppet-eyaml-hiera-data
• https://github.com/voxpupuli/hiera-eyaml
• https://help.puppet.com/core/8/Content/PuppetCore/hiera_eyaml.htm
• https://help.puppet.com/core/8/Content/PuppetCore/hiera_config_yaml_location.htm