SSH Connection Upgrades coming in Ansible 1.5

sshconnection_blog-1

Ansible features a very finely tuned and efficient SSH implementation that we've been working on (believe it or not), on and off, for almost two years. It can work with passwords, keys, any user account, sudo with no password, sudo passwords (or not), and all sorts of other bits. It parallelizes very well, many users using it to contact ginormous amounts of systems in a single pass. It has optimizations to deal with SSH on legacy platforms. It understands all the required prompts and can let you know when a password is wrong. It works with host key checking on or off, and can still prompt you to add new hosts (or not) even while running in multiple forks. All good stuff and things that take a long time to get right!

The current implementation works by transferring remote module files (code) to temp directories on remote systems, arguments already baked in, and cleaning up after itself -- so you never have to install any software on remote systems to manage them, and there is very little payload and nothing to bootstrap to get going, and it leaves nothing behind but log entries. (A lot of people assume management over SSH means we run shell commands and scrape the results -- rather, it's actually more of an abstract transport layer). And it supports using features like ControlPersist so OpenSSH doesn't have to reconnect each time it talks to a host. Good stuff.

While comprehensive and efficient, some people still wanted it to go a bit faster. Note that, truth be told, connections and transports aren't where you spend most of your time in systems management. It's package updates and installation and waiting on services to come online. However, it's nice to have the underlying systems plumbing be smooth, just so that all reduces down to statistical noise. One of the features in Ansible we added to Ansible previously was accelerate mode, which uses SSH to securely bootstrap a temporary socket-connected daemon for improved connection throughput. However, we were blown away recently when community member Jerome Wagner submitted a contribution that proved, in many cases, accelerate mode isn't even needed.

What Jerome did was streamline the way Ansible transfers content such that files don't need to be transferred for basic module execution, which eliminates lots of extra steps and bandwidth. It simply sends them over the channel and executes them, so no chmod is needed, nor no 'rm'. The end result is the number of SSH operations, when no file transfer is required, is cut in about half. It's really awesome and a great example of how some of the best contributions you can get to a project don't have to be super large.The results are pretty impressive. For an arbitrary test workload involving lots of module executions:

Original SSH transport with Control Persist - 1 minute, 1 second
Original SSH transport without Control Persist - 1 minute, 10 seconds
Paramiko - 1 minute, 4 seconds
Accelerate Mode - 30 seconds
"SSH_Alt" (Jerome's patch set) with Control Persist - 25 seconds

As you can see from the above, the "SSH_Alt" patchset is actually faster than accelerate mode in the above case! Truthfully, accelerate mode is a little better at file transfer still, although in realistic workloads, ssh_alt will almost always win, and has the benefit of sending all traffic over the more standard (and more secure, with less moving parts) custom connection.For those that aren't aware what "paramiko" is, it's a custom pure-python implementation of an SSH client that we use in some cases. We keep Paramiko around because the SSH client on Enterprise Linux 6 control machines is still downlevel and doesn't support ControlPersist, and paramiko is faster in those cases. Thus Ansible is very smart and will select paramiko instead of OpenSSH by default on those systems. (We look forward to the day where Control Persist on SSH is available everywhere, and people should get their Red Hat TAMs on the phone if they agree!)For completeness, here's the results of an arbitrary file transfer test:

Original SSH transport with Control Persist - 2 minutes, 42 seconds
Original SSH transport without Control Persist - 4 minutes, 9 seconds
Paramiko - 2 minutes, 12 seconds
Accelerate Mode - 1 minute, 6 seconds
"SSH_Alt" (Jerome's patch set) with Control Persist - 2 minutes

In the above example, accelerate mode gets faster file transfer across because it's not doing nearly as much encryption, and this is why accelerate mode isn't the default right now. If you are going to be doing something like downloading large ISOs, you should look into something like bittorrent anyway, and for package updates, you should be setting up a package mirror. And we always want to make choices like that a conscious choice. As of Ansible 1.5 (and currently on the development branch), the "SSH Alternative" implementation replaces the default SSH implementation by default, though you have to add a "pipelining=True" parameter to the [ssh_connection] section of ansible.cfg to enable this feature, and should note that if using sudo operations, this requires disabling 'requiretty' in your sudoers file. By enabling this, your playbooks, if using OpenSSH, as you most definitely are on Ubuntu or OS X, are going to automagically get a lot zippier.

The next time someone tells you SSH is slow, point them to this blog post, because they are wrong.

Thanks for some awesome work, once again, Jerome! Ansible 1.5 is slated for release in early March, 2014.

The Inside Playbook

SSH Connection Upgrades coming in Ansible 1.5

Related News

Michael DeHaan

Categories