Immutable Systems and Ansible - Building and Deploying AMIs to AutoScaling Groups

September 15, 2014 by James Martin

As discussed  in the previous  blog post, there are multiple ways to use AWS AutoScaling groups with Ansible.

Previously we showed how you can use Ansible Tower's provisioning callback mechanism to provision freshly provisioned AutoScale instances as soon as they come online.

In this blog post,  you will learn how to use Ansible to build and deploy AMIs to AutoScale groups.   

Advantages of deploying pre-built AMIs

  • Using pre-built images eliminates on-demand configuration in favor of build time construction, reducing the amount of time it takes for an instance to become "ready" during a scaling event.

  • In the event that a web resource you normally install from is unavailable, you will still be able to role out new instances

  • Deployments using this system can potentially save bandwidth costs as well as install time.

  • Allows you to treat your instances as expirable, unchangeable things, terminating them when they fail in lieu of troubleshooting.

Disadvantages of deploying pre-built AMIs

  • Keeping track of all the images you are generating may become a burden.

  • Increased orchestration complexity when updating infrastructure.

  • Opaqueness of the image -- "what's in this thing I'm deploying?"

This might be done to achieve faster application rollouts and may increase deployment reliability, but also changes how you deploy and update your cloud infrastructure.

Building AMIs

Ansible can easily build an AMI for your application using the ec2_ami module.

The ec2_ami will take a running instance and create an AMI from it. You first want to have an Ansible role(s) you'd like to apply to an instance. Once your role development is complete, you can use it to define how you'd like your AMI to appear.

The typical AMI-building playbook will:

  1. Launch a temporary instance.

  2. Apply a set of roles to that instance.

  3. Run the ec2_ami module against that instance.

  4. Terminate the temporary instance.

This playbook is a reference example of how to build a custom AMI. Once that custom AMI is complete, the AMI it generates can be used as input to one of the following approaches.

Some users may also like to use tools like Aminator and Packer, both featuring Ansible provisioners, instead of the above module.  This is largely up to the reader and all are good options.

AutoScale Update Policy

Rolling out instances uses the ec2 module in Ansible, which is generally straightforward.  Rather than specifying a base OS image, you would then specify the IDs of images constructed above.   Once deployed though, how do you move between one version of the application and another when following an “immutable systems” type approach?  Ansible has strong rolling update support for classical infrastructure, but can it apply here?  It turns out it can.

Amazon has the ability to do a rolling update to instances in a AutoScale group, but only has exposed that feature to AutoScale groups created & managed with CloudFormation.  As a result, our playbooks will use a bit of CloudFormation, which ansible has a module for.

When using an ASG update policy, if the ASG's associated Launch Configuration has changed, the instances in the ASG will be replaced serially to match the new Launch Configuration.

Since an AMI ID is part of a Launch Configuration, we can leverage this update policy feature to deploy instances into a ASG with a newly specified AMI, and do so in a rolling fashion.

If you take a look at this playbook and its respective, you can see the following happening:

  1. VPC, Subnets, Security groups are created with native Ansible ec2* modules.

  2. In a separate role, the Launch Configuration and the AutoScale group are created with a CloudFormation template that is launched with the CloudFormation module in order to leverage the CloudFormation-only update policy described above.

This workflow can be launched with:

ansible-playbook cfn_update_policy.yml

Rolling AMI deploy with pure Ansible

You don't have to use CloudFormation's Update Policies to leverage a rolling replacement approach for intsances in an ASG -- you can also do it with native Ansible modules.

The ec2_asg module in development branch of Ansible (scheduled for 1.8) supports this and may provide a nice alternative  to users not wanting to contend with CloudFormation.   This workflow would be as follows:

  1. New AMI is generated

  2. New launch configuration name is generated with new AMI ID.

  3. Autoscale Group is assigned the new launch configuration

  4. A list of instances not associated with the current launch configuration is collected

  5. The desired capacity and minimum size of the ASG are altered to account for the batch size of the rolling termination.

  6. The list of instances we collected through before is looped through by the batch size, and each instance is terminated. After an instance terminated, we wait for the replacement to come online.   The batch size controls how many instances to remove/replace at once.

  7. After all instances are replaced, we set the ASG to its normal operating desired_capacity, min_size, and max_size.  This workflow can be launched with:

ansible-playbook rolling_ami.yml -vv -e "deploy=yes"

or optionally through Ansible Tower:

Screen Shot 2014-09-10 at 7.14.24 PM.png

Blue - Violet

The previous 2 options focused on a rolling approach.  The next two are going to focus on parallel deployments.  The concept behind the blue violet approach is as follows:

  1. One provisioned autoscaling group is running and behind a ELB

  2. The new autoscaling group and launch configuration are provisioned side-by-side, behind the same ELB.

  3. Ansible waits for new instances to become available in the AutoScale group.

  4. Role terminates old autoscaling group and associated instances.

The main draw back to this approach is that you effectively have to have double the number of deployed instances for a small amount of time while you are deploying your new AMI to the new ASG.  Of course, your application must be tolerable of 2 different versions running side by side.  The plus side is that is much faster than performing a rolling approach.


This workflow can be launched with:

ansible-playbook blue.yml -vv -e "deploy=yes"

and when you're ready for the violet to take over

ansible-playbook violet.yml -vv -e "deploy=yes terminate_relative=yes"

Blue - Green deployments

The approach behind a blue-green deployment is outlined here:

It entails provisioning a new Autoscale group and ELB and cutting the traffic over to the new cluster.

The problem with this approach is two fold: you must pre-warm ELB before the cut-over.  Unfortunately this is a manual process which requires contacting AWS support. You'd also have to make DNS change so that your website points to the CNAME of the new ELB.  The DNS change is not instant--it could take days for DNS changes to propagate amongst the world's DNS infrastructure.

One potential workaround would be to use a tool such as socat on the old cluster nodes to direct incoming traffic to the new ELB while DNS is propagating the updated record.

Over time, the connections to the old clusters should diminish, and then the cluster can be manually terminated.


In the past blog article, we shared some tips about how to use Ansible with an autoscaling configuration where Tower live configures systems.   In this post, we’ve showed multiple ways to use Ansible and Tower in an immutable systems approach.   Ultimately, there are a lot of ways “to cloud”, and what choices make sense for you are going to be based on your application and team workflow.  We hope these posts give you some powerful tips to get started if you’re interested in Ansible, Ansible Tower, and EC2 Autoscaling.


IT Automation


James Martin

James Martin, Red Hat practice lead for Red Hat Ansible, has always had a passion for automation in the 22 years he's worked in IT. He has worked in various roles from systems administrator, to systems engineer, to consultant. James came to Red Hat through its acquisition of Ansible in 2015 and now leads of team of consultants focused on automation.

rss-icon  RSS Feed

Ansible Tower by Red Hat
Learn About Ansible Tower