Orchestration, You Keep Using That Word

April 28, 2014 by Michael DeHaan

Pokey the Horse on a Moog synthesizer, for no good reason


“Orchestration” is a commonly used word in IT automation. Unfortunately, it is often used to mean many completely different things, reducing the importance of understanding the key concepts. At Ansible, the team has at times said we need a better word, but stopped short of naming it. So let’s look at the word “Orchestration” and some of the ways it is often used, and talk about what Ansible really does – which is really everything people may call orchestration, and then some.

Orchestration as a Command Blaster

At the most basic level, command blasting means blindly running a “/usr/sbin/reboot” command against tens, hundreds, or thousands of servers. It’s quick and dirty, but there’s nothing to say you couldn’t have used something like a basic parallel SSH script to get this done. This has been done with old school scripting for years.

We do that too, but we can also apply resource models in this way (make sure the user looks like this on all these hosts, etc), which takes that up a level.  This is not new - I think Func may have applied it first, but it's modules models were less declarative than Ansible, and the idea was then used all over the place.

Orchestration as a Configuration Trigger

In many cases triggering another management program on many hosts is another way orchestration is used. For instance, if you use a single-purpose deployment tool like Fabric or Capistrano to trigger some legacy configuration management, that might be called “orchestrating X with Y”. In reality, it’s usually just replacing a non-scaling (or hard to order) server solution X with a stand-alone implementation of Y. You’ll lose centralized logging, but you might be able to fix the next problem we’ll mention easier. Ansible can easily do this too.

Orchestration as Config Ordering

At a slightly higher level, suppose you are using some configuration management software. The idea that you have to deploy a database update prior to enabling new webserver code is an example of common ordering that can be called orchestration. This is basic, and Ansible can easily define these orders in a playbook - but it can also go beyond this pattern to provide more advanced orchestration.

Multi-Tier, Non-System Specific Orchestration -- Orchestration like an Orchestra

This is what Ansible really excels at - and is often lacking in other solutions. Suppose you are a composer, and you have an orchestra in mind and a basic tune you want the orchestra to play. The word orchestration really means taking a basic melody and chords and adapting it so that a large ensemble of instruments can play it. It is about writing the individual scores for all of the instruments.   IT orchestration is a difficult thing to do well - to know how the strings should harmonize with the brass, how the percussion will support the tempo, how the woodwinds will tie it all together.

So, to make it all happen, an orchestra must be conducted. It’s not simple enough to say “woodwinds play, then brass” – as is common in some basic configuration systems. Wouldn’t it be sad if the brass could only play once? Instead, you must be able to say, “woodwinds play this movement, then brass plays THAT movement, and then woodwinds play that movement that references the melody of the previous”.  And sometimes both play at the same time.  

This is what Ansible was written to do: Manage complex multi-tier deployments.

We don’t just stop at configuration management or application deployment. To really do orchestration, you must do all of those well AND be an outstanding workflow engine. Real world app stacks involve lots of different classes of systems all working in concert.

Orchestration in the Real World

Suppose I need to deploy a web application stack, but to do that, I need to deploy the database server first.  And I need to include on the database server all the IP addresses of the web servers that can connect to it - and also while looping over each server, add that server to a monitored pool of hosts that involves running something on the monitoring server.  

Suppose I then also need to update a given amount of my infrastructure (say, 10%) at a time, only taking certain hosts out of load balancing at a time. This is the key principle behind achieving zero downtime rolling updates in Ansible. See also this continous deployment guide here.

By including a process that first pulls nodes out of monitoring, then load balancing, updating those hosts, and then putting them back in load balancing, and then monitoring, a seamless rolling update can be achieved regardless of infrastructure size. And that’s the kind of thing you can do with a half page of automation code in Ansible.

Further, it’s easy to introduce tests. If tests are called before a system is added back to a load balanced pool, it can be possible to not add failing servers back to production.

IT orchestration is also about doing things that are not just talking to servers – if we just orchestrated servers, it would be a sad, sad, world. Ansible can also talk to REST APIs (such as using the ‘uri’ module), send chat messages, and pretty much do anything.  Increasingly, our applications are not just server centric, but they are services centric.

One frequent scenario that may arise is that each server might need to speak to a rate-limited service.  With the “serial” keyword in Ansible, it’s possible to define steps where only a finite number of systems may bombard a service in a given amount of time.

Another scenario is we might want to wait for a particular port to become open before proceeding, or even wait a given number of seconds – or for human input.

Still more related examples of orchestrating services include provisioning new cloud resources, such as deploying compute nodes or adjusting autoscaling policies.   Node provisioning and cloud configuration often has to be done prior to even beginning to think about configuration and apps.

Orchestration is About Mapping Out Workflow

What we believe orchestration is, then, is not just the simple blasting out of commands, or the ordering of what system runs a tool before another, but the definition of a flowchart.

While Ansible isn’t a complete flowchart system (there are no gotos and it progresses top down), what it can do is model an arbitrary (and potentially very advanced) IT process, rather than just a basic description of a state. Playbooks can be written for very specific events, and different playbooks used at different times.

Many GUI-based expensive enterprise tools try to do the same thing, but can be very hard to use, and can’t be used in DevOps style (text in source control) workflows.

Ansible is that system, because it’s simply text based, but also strictly order based, and the steps along the way that it executed can be arbitrarily limited, easily controlled with conditionals, and steps can easily be done on one host on behalf of another.

User switching (via su or sudo or direct login) for different steps is also trivial and implemented as a first class concept.

Ultimately while we’d like to think systems are modeled by a graph or something pure and theoretical, they really aren’t. We aren’t always just saying, “this system should be the perfect model of X”. Often we have a lot of steps for an arbitrary workflow, and Ansible allows you to do both – define the declarative model if you want, but also define a process of steps – and also things in between.

Real IT orchestration is about enabling application of that model, but also about modelling of workflow. Our future is about making that workflow increasingly easier to use, more accessible, and including more tools in the toolbox to make all manner of IT automation tasks increasingly trivial.

Ansible can be used with any configuration management system you have. Ansible can be used with other deployment systems, or scripts you might already have.  You could also just use Ansible for everything. But either way, it’s an AMAZING conductor that probably has a great place in your environment whatever you are doing. Even if you have a lot of legacy IT automation that you feel you can’t get out from under, give Ansible a try for orchestration. As Atlassian points out, we can play nicely with others

Related News

Deploying Highly Available OpenShift Origin Clusters | Installing and Building Docker With Ansible | Listen To Your Servers Talk | Fixing Heartbleed With Ansible | Ansible Me A Sandwich



Configuration Management


Michael DeHaan

Ansible project founder.

rss-icon  RSS Feed