blog_hashicorp-vault

Automation is an essential part of modern IT. In this blog I focus on Ansible credential plugins integration via Hashicorp Vault, an API addressable secrets engine which will make life easier for anyone wishing to handle secrets management and automation better. In order to automate effectively, modern systems require multiple secrets: certificates, database credentials, keys for external services, operating systems, networking. Understanding who is accessing secret credentials and when is difficult and often platform-specific and to manage key rotation, secure storage and detailed audit logging across a heterogeneous toolset is almost impossible. Red Hat Ansible Tower solves many of these issues on its own, but its integration with enterprise secret management solutions means it can utilize secrets on demand without human interaction.

In terms of secrets management, I will demonstrate how some of the risks associated with an automation service account can be mitigated by replacing password authentication with ssh certificate based authentication. In the context of automation, a service account is used to provide authorised access into endpoints from a central location.  Best practices around security state that, shared accounts could pose a risk. While Red Hat Ansible Tower has the ability to obfuscate passwords, private keys, etc. It doesn't address risks associated with key generation that involve an operator (human). With Red Hat Ansible Tower integrated with Hashicorp Vault we are able to rotate service credentials regularly and on demand.

 

Secure Credentials

While secrets and tokens through methods such as kerberos have implemented secure credentials, Hashicorp Vault provides a seamless workflow to manage ssh certificates as an authentication method.

Some of the benefits I see with certificate-based authentication are:

  1. Certificate verification and time validity is verified by the CA certificate on the endpoint.
  2. No need to reset password on accounts every 90 days (or less).
  3. You can trace the usage of a certificate to a specific automation.
  4. No risk of ‘shared account’ falling into the wrong hands because no human intervention required.

 

Leveraging HashiCorp Vaults secrets engine for automation

Deploy Hashicorp Vault

The first thing we need in any PKI infrastructure is a certificate authority which HashiCorp Vault has built into it. I’ve documented the overall Vault setup procedure that I use here. Without going through the details it's worth noting the procedure will build a PKI infrastructure that can sign ssh client certificates and grant an automation account access for a maximum of 15 minutes.

Vault also provides account auditing and logging, like when was the last time I asked for a credential, so it is perfect for creating an audit trail of when an automation account is checked out and used to authenticate.

Vault also lets you use roles to manage who can access user credentials from the Vault so it's easy to separate basic user access from root level access by implementing policies.

In my typical LINUX setup I use Active Directory to supply user accounts and POSIX information, but the authentication is done using signed ssh certificates. What this means is that accounts are managed centrally and that integrating ssh certificate based authentication integrates with existing access management practices.

 

Setting up the Ansible Tower Credential

Now, we want to give Ansible Tower the ability to request authentication credentials on demand.

The following example illustrates how to configure a Machine Credential to sign a public key using HashiCorp Vault:

First, we need to create a credential that has the authority to request certificate signing. This is the HashiCorp Vault credential. The token I am using belongs to my user, ‘richard’ who has already been created inside the Vault and is restricted by the plain-richard policy as per what is described in the Vault installation document.

Credential 1: ‘richard-vault’

pasted image 0-8

We can then test it by clicking the test button, and using the contents of the public ssh-key created in the Vault steps earlier, combined with the ssh agent location, role and principal name.

pasted image 0-9

Now we need to a new machine credential that will be used for ssh access.

Create a new credential of type machine and populate the following fields.

Screen Shot 2019-11-13 at 10.47.31 AM

pasted image 0-10

For the next step, we will be combining the two credentials using API commands.

In this step I worked out what the ID of my two credentials is by issuing the following commands using the tower-cli tool.

tower-cli credential list
== =========================== =============== 
id            name             credential_type 
== =========================== =============== 
12 richard-ssh-public-unsigned               1
11 richard-vault                            20
== =========================== =============== 

And then use this api call to link the machine credential to the HashiCorp Vault credential:

curl -sik  "https://ansible.bravo.com.au/api/v2/credentials/12/input_sources/" \
    --user admin:Fo0B@r \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{"source_credential": 11, "input_field_name": "ssh_public_key_data", "metadata": {"public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8kJkcODjq80ryg6WpKIj+Ytj7z9CP+dtHaeaQ/eFR/69txr1fbbxrsTALVOJKSkcTwlG6JBacKcBXs4jacwfbRjRr7q2K4FRgZ4hygYtM9d/PhG2l7nH5sFzhy4o4sc8kdnXfcWeNdZFH/ySqfvvdexDt+nmeCSlixk5i9wzAhOayDo2EiboN38zbkfC7FiE/gQGlFVeqou4m5dbLXpXz5N/JhwzEbmWTQCckHdlYflro9TLTMVZn1/FN2mjr9ZzuqFzDk4IS8HuBkvbo+9GxcM6qGNGv1h/Af40uetRQcanzEu0AsT98Yo9KSeQ+tsX4dOJq7WEV+LwfactJ3WJj vault@ansible.redhat.local", "secret_path": "ssh", "role": "regular", "valid_principals": "richard"}}'

The result from the command above results in the ‘Signed SSH Certificate’ field populated with the Vault credential.

pasted image 0-11

The next step in Ansible Tower is to associate the machine credential with a Job Template. When the Job Template is run, Ansible Tower will use the provided HashiCorp URL and token to sign the unsigned public key data using the HashiCorp Vault SSH Secrets API. Ansible Tower will generate an id_rsa and id_rsa-cert.pub on the fly and apply them using ssh-add.

pasted image 0-12

Running the template I can see the ssh-add process in action from the tower logs. In this example my logs are being sent to elasticsearch which makes things easier to find.

unnamed-4

Now whenever I use the richard-ssh-public-unsigned credential in Ansible Tower, what is actually happening behind the scenes is that Vault is signing my public key.

Here is a manual representation of what is happening behind the scenes, as if I were doing things manually.

Note: I am using a JSON payload here to customize the key contents.

$ export VAULT_ADDR="http://127.0.0.1:8200"
$ vault write \
    -field=signed_key \
    ssh/sign/regular \
    valid_principals="richard" \
    public_key=@$HOME/.ssh/id_rsa.pub \
    > $HOME/.ssh/cert-signed.pub

Now the ssh certificate has been created, we can view the contents like this;

$ ssh-keygen -Lf $HOME/.ssh/cert-signed.pub

Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:bvXl4AuO8TTPYPoKSLCgQ5BicxU0dSolrz3z4pNGuQ0
Signing CA: RSA SHA256:gZjh7y2CMtE7fmYqnSe0ArkYZdtQ50TW+R8xbs1BxyY
Key ID: "plain-richard"
Serial: 10382493167060316007
Valid: from 2019-10-25T12:10:49 to 2019-10-25T12:25:49
Principals: 
        richard
Critical Options: (none)
Extensions: 
        permit-pty

You can test out whether the generated key has access by issuing an ssh command like this;

ssh -i ~/.ssh/id_rsa -i ~/.ssh/cert-signed.pub richard@3.104.104.20

Note: the user richard is actually an account that exists within my active directory domain. It is the ssh daemon which is performing the key verification and then passing the rest of the authentication process through to PAM.

$ id -a
uid=575601162(richard) gid=575600513(domain users) groups=575600513(domain users),575601111(ansible users),575601116(ansible admins

In my next blog, I will implement the same solution using CyberArk Conjur, another great solution for enterprise secure credentials retrieval and updates.


About the author

Richard Hailstone is an Associate Principal Specialist Solutions Architect at Red Hat. He helps Enterprise businesses innovate with open source technologies. He spends the majority of his days working with technical teams on harnessing the power of automation through Ansible. He has over 22 years’ industry experience that has spanned the dot com era, the emergence of virtualisation, containers and hybrid cloud, and has worn a variety of hats including development, operations, engineering and architecture roles. Aside from spending his time coding with new technologies, Richard spends as much time as possible outdoors with his wife and three children, finding new renovation projects, or adventuring through new mountain bike trails.

Read full bio