Using Kerberos for Windows in Ansible Automation Platform 2

Kerberos is often the preferred authentication method for managing Windows servers in a domain environment. Red Hat Ansible Automation Platform has allowed customers to leverage Kerberos authentication for a number of years now. So why revisit this subject? 

Ansible Automation Platform 2 was released in July 2021 and was a major re-architecture of the platform. One of the fundamental changes was the introduction of automation execution environments  - the use of containers to consistently package, distribute and execute Ansible Playbooks. Without going into the weeds, automation execution environments consist of a RHEL base image, Ansible Core and any dependencies required to execute our Ansible automation - these are typically Ansible Content Collections and Python libraries. 

The move to containers means that we sometimes need to consider that localhost is now a container. There is an excellent blog post that goes into the details of how localhost isn’t what it seems when it comes to automation execution environments.

With all of this in mind, let’s go through a guided example of how to configure Kerberos authentication in Ansible Automation Platform 2, how to test the configuration and how to configure automation controller to use Kerberos.

Example configuration

In order to use Kerberos with Ansible, we will need a Kerberos configuration file. The contents of the configuration file will vary depending on factors such as DNS configuration, KDC discovery and number of domains. In this example, we have a single domain which is DEMOLAB.LOCAL which is not discoverable to our automation controller server.

Here is our sample Kerberos client configuration. There is a similar example documented in the automation controller admin guide.

[libdefaults]
	rdns = false
	default_realm = DEMOLAB.LOCAL

[realms]
 DEMOLAB.LOCAL = {
  kdc = ms-ad.demolab.local
  admin_server = ms-ad.demolab.local
 }
 

Now we need to consider automation execution environments and remember that localhost isn’t what it seems! When automation controller runs an Ansible Playbook, it will invoke a container image which won’t have access to the underlying filesystem of the controller node. We need to check that the execution environment has access to the Kerberos client configuration. There are two options to achieve this.

1. We can map the Kerberos client configuration into the running execution environment from automation controller.
2. We can customize and build our own execution environment using Ansible Builder and add the krb5.conf into the image.

For this example, I am going to map the Kerberos client configuration into the running execution environment as I don’t need to make any other changes. 

In order to map the file into the execution environment, let’s write the configuration to the Kerberos client configuration directory on our automation controller server called /etc/krb5.conf.d/DEMOLAB.LOCAL.conf

# cat /etc/krb5.conf.d/DEMOLAB.LOCAL.conf 

[libdefaults]
	rdns = false
	default_realm = DEMOLAB.LOCAL

[realms]
 DEMOLAB.LOCAL = {
  kdc = ms-ad.demolab.local
  admin_server = ms-ad.demolab.local
 }
 

Testing from the command line

We are ready to check the configuration from the command line. First, let’s confirm what execution environments we have available to test with. On the automation controller server we can switch to the awx user and list the execution environment images. Here we can see the ee-supported-rhel8 execution environment is available which is the image I want to test with.

$ su - awx
$ podman images
REPOSITORY                                                        	TAG     	IMAGE ID  	CREATED  	SIZE
registry.redhat.io/ansible-automation-platform-23/ee-supported-rhel8  latest  	024856bccc5f  4 weeks ago  1.66 GB

If you need to test with another image then use “podman pull” to copy the relevant image from a container registry.

Now it’s time to see if we can map the configuration directory into the execution environment, authenticate and obtain a kerberos ticket. The following Podman command will start our execution environment container and map /etc/krb5.conf.d/ into the running container. We’ll also be given an interactive shell inside the container so that we can test kinit. One final note on this, Kerberos uses a credential cache to hold valid Kerberos credentials. A reminder again, localhost isn’t what it seems here so we don’t have access to the filesystem and the Kerberos cache. We’ll create a temporary Kerberos credential cache to allow us to test the configuration by setting the KRB5CCNAME variable. The winrm connection plugin uses a similar mechanism. 

$ podman run --rm  -v "/etc/krb5.conf.d:/etc/krb5.conf.d:O" -it registry.redhat.io/ansible-automation-platform-23/ee-supported-rhel8:latest /bin/bash

bash-4.4# export KRB5CCNAME=`mktemp`
bash-4.4# echo $KRB5CCNAME
/tmp/tmp.ZzBXbtGiV1
bash-4.4# kinit svc-ansible@DEMOLAB.LOCAL
Password for svc-ansible@DEMOLAB.LOCAL:
bash-4.4# klist
Ticket cache: FILE:/tmp/tmp.ZzBXbtGiV1
Default principal: svc-ansible@DEMOLAB.LOCAL

Valid starting 	Expires        	Service principal
04/04/23 14:01:37  04/05/23 00:01:37  krbtgt/DEMOLAB.LOCAL@DEMOLAB.LOCAL
    renew until 04/11/23 14:01:33

As a further test, we can validate if the ticket works for authenticating against a specific server. After successfully running kinit in the execution environment, we can use kvno to authenticate with a target host. In this example let’s check that we can use the ticket to authenticate with windows2.demolab.local.

bash-4.4# kvno http/windows2.demolab.local
http/windows2.demolab.local@DEMOLAB.LOCAL: kvno = 1
bash-4.4# klist
Ticket cache: FILE:/tmp/tmp.pe2PBReLm5
Default principal: svc-ansible@DEMOLAB.LOCAL

Valid starting 	Expires        	Service principal
05/10/23 15:26:35  05/11/23 01:26:35  krbtgt/DEMOLAB.LOCAL@DEMOLAB.LOCAL
    renew until 05/17/23 15:26:33
05/10/23 15:26:49  05/11/23 01:26:35  http/windows2.demolab.local@DEMOLAB.LOCAL
    renew until 05/17/23 15:26:3

The test looks good! Time to move the configuration into automation controller! 

Configuring automation controller

We need to add a Machine credential in automation controller for our Windows account. This should be the user principal name (UPN) in the format ‘username@DOMAIN.COM’. To add a credential in automation controller, navigate to Credentials in the left hand menu, press Add and select the credential type as Machine. Enter the domain user and password as follows:

Next, we can map the Kerberos client configuration into the execution environment. An administrator will need to configure Settings -> Job Settings  and edit the “path to expose to isolated jobs”, adding the Kerberos directory. Note how this is the same format we used when testing from the CLI. The new configuration should look like this:

Time to test! We can use a simple “ping” playbook to validate the configuration. We can also increase the logging to include WinRM connection messages. Edit the job template and set the verbosity to level 5.

When we launch the job template we should see the detailed connection messages.

We can see that the temporary Kerberos credential cache is created, the account is able to authenticate and we obtained a ticket. The playbook should complete successfully.

Troubleshooting

You might encounter some error messages when configuring Kerberos within an execution environment. Here are some errors you might see and possible solutions.

• Included profile directory could not be read while initializing Kerberos 5 library - This might indicate that there is a configuration file in your /etc/krb5.conf.d directory that is trying to include additional directories which the execution environment doesn’t have access to. Check any configuration files in /etc/krb5.conf.d that might have an includedir option.

• Invalid UID in persistent keyring name while getting default ccache - Remember to set a temporary credential cache directory as described above.

• Cannot find KDC for realm "<DOMAIN>" while getting initial credentials - There can be many reasons for this error. Check that the Kerberos configuration looks correct and that a valid KDC is listed in the configuration. If you’re relying on DNS to lookup KDC’s, then ensure dns_lookup_kdc is set to true. 

• Server not found in Kerberos database - this can often be related to DNS issues or an incorrectly configured Ansible inventory. Check that the name of the host in the inventory matches the name of the server in DNS. Also, validate that Ansible is attempting to connect to the host using the FQDN by using the WinRM debug logs. Here is an example where the ansible_host variable is set for a Windows server and we are attempting to connect to the IP instead of hostname.

Next Steps

Hopefully this example was helpful in understanding some of the changes in Ansible Automation Platform 2 and how to get started with Kerberos authentication for managing Windows servers. Windows continues to be a first class citizen for Ansible and there are more resources available to help you on your journey.

• Ansible for Windows - learn how Ansible can be used to manage Windows infrastructure and common use-cases.
• Trial subscription - Are you ready to go? Get your own trial subscription for unlimited access to all the components of Ansible Automation Platform.