Project signing and verification

Project signing is a new feature developed for Red Hat Ansible Automation Platform that came out in the latest 2.3 release. It enables users to sign project-based content (think playbooks, workflows, inventories, etc.) and verify whether or not that content has remained secure. It also features a new CLI tool, ansible-sign. This blog post will explain how it works, illustrate how to implement it, and highlight a few scenarios.

Why we need signing capabilities

Organizations need to make sure their automation is tested and performing the intended tasks.  However, what if someone deploys untested automation, or worse yet, someone intentionally tries to automate something nefarious?  It might not even be intentional, but can simply be an organization using a community collection whose author removes a feature that they were using.

When organizations start adopting automation at the enterprise level, there may be hundreds to thousands of tasks being performed every hour across thousands of infrastructure nodes. How do you make sure the automation content that is being executed can be trusted? How do you know your automation is doing what you think it is? Is your organization pulling content from various sources outside of your company? Can all of those sources be trusted with equal confidence?

This feature helps ensure your end-to-end supply chain remains untampered from source of truth to running an automation job using that content. Put another way, it stops man in the middle type attacks and provides assurances around quality and conformance.

How it works

Project signing enables users to sign content and verify whether or not that content has remained secure. To accomplish this, we use asynchronous keys, one private key at the point of signing, and a public key to verify that the content has not been tampered with. 

First, a project will have the option to be signed. When the user syncs a signed project, the controller pulls in the changes, compares the public key stored in the controller with the private key that the checksum manifest was signed with (this prevents tampering with the checksum manifest itself), then re-calculates checksums of each file in the manifest to ensure that it matches. The user commits the change, the updated checksum manifest, and the signature to the repo. If the project is signed, the content signature will be verified after the project sync. If the signature is not valid, the given job will not run. Currently, GPG is the only supported means of signing and validation.

What is a GPG key?

GPG keys are a means of encrypting and decrypting data. Private keys are used to decrypt messages that are encrypted using the public key. They can also be used for signing messages or data. This is a technique to prove that you own the key. Remember not to share this one, it’s private!

Public keys are the second half of a GPG key. They are used to encrypt messages or data for the owner of the private key. This key can be shared, as it can only be used to encrypt messages or data for the private key owner.

Project signing uses GPG detached signature generation and validation (using python-GnuPG) to verify content in this context. GnuPG utilizes a two-key cryptography approach to project signing. Asynchronous keys are used to validate the integrity of a project’s content. A private key is used at the point of signing and a public key is used to verify that the content has not been manipulated in any way.

For more information regarding the generation of GPG keys, please see: https://www.redhat.com/sysadmin/creating-gpg-keypairs

What’s a checksum?

A checksum is the result of an algorithm called a cryptographic hash function. It uniquely produces a string based on the content of a given file. You can think of it as a unique fingerprint for a specific file. If any of the content is modified, the checksum will change. The checksum manifest contains the checksums for every file in the repo. This is how the checksum/digest is generated.

Let’s walk through the process:

Now that we understand how project signing works, let’s discuss how you can implement it.

First, you will need to generate a GPG key. You can verify that you have a valid GPG keypair and in your default GnuPG keyring, with the following command:

$ gpg --list-secret-keys

If this command produces no output, or one line of output that states trustdb was created, then you do not have a secret key in your default keyring. Please refer to How to Create GPG Keypairs for a step-by-step guide to GPG keypair generation and alternatively go through the project signing self-paced lab to see an example. Once you have your GPG key, you are ready to move forward and add it to the UI.

You will run the above command again, and then grab your key’s fingerprint from the CLI. You will use these commands to accomplish this:

$ gpg --list-keys
$ gpg --export --armour <key fingerprint> > my_public_key.asc 

Note that you will want to copy everything from:

-----BEGIN PGP PUBLIC KEY BLOCK-----
To 
-----END PGP PUBLIC KEY BLOCK----- 

Alternatively, this can be saved to a .txt file and uploaded later. 

In the Ansible UI, click “Credentials” from the left navigation menu, and then click “Add”. Once there, you can either paste the block you copied earlier or browse and select the GPG key file. Be sure to save once that has been accomplished.

You can now select your GPG key in the “Create New Project” field. That will look something like this:

Now that you have a GPG public key credential saved, content verification will automatically take place on any project to which that credential is added. Note that this is automatic, and if a GPG credential is selected in a project, project verification will occur. If you do not want that to happen, that credential will need to be removed from the given project. 

Ansible-sign:

The ansible-sign utility is a library and auxiliary CLI tool for project signing. This is the piece that utilizes checksum manifest generation and validation, specifically sha256, and GPG detached signature generation and validation for content. Let’s take a look at how we install and use it!

Installing ansible-sign:

Ansible-sign can be installed using pip or the Ansible Automation Platform subscription. To run this installation, use $ pip install ansible-sign or use dnf install ansible-sign on an Ansible Automation Platform subscribed system. To verify if the installation was successful, run $ ansible-sign -–version. You should see an output similar to this: ansible-sign 0.1.

Once you’ve confirmed you have installed ansible-sign, you’re ready to start signing!

Signing content:

To begin signing, you will need an Ansible project directory. For more examples of project directory structures, please visit: Ansible Documentation. Below is a simple project directory we will use to illustrate how to sign content using the ansible-sign utility. Please note that ansible-sign project commands always take the project root directory as their last argument. Use . to indicate your current directory. 

$ cd sample-project/
$ tree -a .
.
├── inventory
└── playbooks
    ├── get_uptime.yml
    └── hello.yml
1 directory, 3 files

You will first need to create a MANIFEST.in file. This should reside in the project root directory. This file will tell ansible-sign which files it should protect. For more information regarding MANIFEST.in directives, please see the Python Packaging User Guide. For this example, your MANIFEST.in file should look like this:

include inventory
recursive-include playbooks *.yml

Once the project repository has been set up and the GPG keys have been created, you are ready to begin signing. The below command can be run to generate and sign a checksum manifest.

$ ansible-sign project gpg-sign /path/to/project

If successful, you should see the following output:

[OK   ] GPG signing successful!
[NOTE ] Checksum manifest: ./.ansible-sign/sha256sum.txt
[NOTE ] GPG summary: signature created

If you now review your file structure, you should see that two sha256 files have been added:

$ tree -a .
.
├── .ansible-sign
│   ├── sha256sum.txt
│   └── sha256sum.txt.sig
├── inventory
├── MANIFEST.in
└── playbooks
    ├── get_uptime.yml
    └── hello.yml

Can it be automated?

In environments with highly-trusted continuous integration (CI) environments, such as OpenShift, it is possible to automate the signing process. For example, you could store your GPG private key in a CI platform as a secret, and import that into GnuPG in the CI environment. You could then run through the signing workflow within the normal CI workflow/container/environment.

Common Use Cases:

• CI/CD pipelines
  • https://www.redhat.com/en/topics/devops/what-cicd-pipeline
• Preventing man-in-the-middle attacks
• Standards enforcement

What happens if project signing fails?

There are multiple scenarios that could cause project signing to fail. We have provided helpful exit codes to shed light on what may have occurred to cause project signing to fail. Below table explains all the possibilities.

Scenarios:

Summary:

Project signing is an exciting new feature coming in Ansible Automation Platform 2.3, designed to help keep your content secure when using it within automation controller. It lays the foundation for further signing enhancements.

We have also designed a self-paced lab that can take you through the step-by-step process of project signing and verification.

This complements the other signing capabilities we have already in place for Collections and automation execution environments using private automation hub. See this blog for further details.

Where to go next:

• Ansible Automation Platform 2.3 release blog
• Automation controller project signing and verification documentation
• Ansible-sign Utility:
  • https://github.com/ansible/ansible-sign
• Ansible-sign Utility Documentation:
  • https://ansible.github.io/ansible-sign/rundown.html
• Collection signing in private automation hub blog