OpenStack Kolla: Dependency Management Done Right With Docker and Ansible

October 27, 2015 by Steven Dake


Kolla provides production-ready containers and deployment tools for operating OpenStack clouds that are scalable, fast, reliable, and upgradable, using community best practices.  Kolla entered the OpenStack Big Tent during the Liberty cycle by submitting Kolla to OpenStack technical committee oversight --  enabling the Kolla project and its contributors to have access to community resources such as marketing, technical resources, bi-yearly conference space and voting rights in the OpenStack Technical Committee election.

During the creation of the Kolla mission statement, we agreed as a community not to permit the selection of technology choices in our mission statement. Still: we knew we would choose Docker as our container runtime technology, and Ansible as our orchestration system. We made these choices not only because “that’s what all the cool kids are doing’ -- but also because they solve real technical problems for our problem domain.  Docker solves our image management process and Ansible solves our multi-node deployment process.  We could have chosen other technologies to solve these problems, but both Docker and Ansible do something orders of magnitude better than competitors: a complete and absolute focus on simplicity coupled with a high degree of capability.

Since a fundamental factor in outcome of the quality of an engineered system is simplicity, choosing simple dependencies was absolutely critical for us. Kolla as it stands today is very simple. If we had chosen complex dependencies, the entire foundation of Kolla would have been at risk. By choosing Docker, Ansible, and Jinja2 we simplified our dependencies and optimized our quality.

A key factor in the outcome of the growth of an open source system is the rate at which new or inexperienced developers can learn the existing implementation to refactor it into the next implementation. Kolla is designed from the ground up to offer a scalable development experience. There are two basic patterns that must be learned in order to implement deployment support for an OpenStack Big Tent project. The effort involved in learning these two Kolla-specific patterns is on the order of three to five half-day blocks of time commitment. Ansible, Docker, and Jinja2 are also designed to offer a scalable development experience, meaning they can be learned in the same three to five half day blocks of time commitment. Other technical choices would not have afforded Kolla a scalable development experience and enable us to deliver on our ambitious goal of deploying the Big Tent.

Kolla uses Docker to store the file-system contents of images and provide an execution environment for those containers. One key advantage of Docker is that a registry system provides a central repository of images which can be used to instantiate a deployment. By choosing Docker, we obtain the superpower of immutability with the Docker registry. Kolla has approximately ninety containers. Sixty would likely be running; the remainder are base containers used as intermediate build steps shared by children containers. One problem we ran into and the first pattern Kolla generated with Docker is that we needed to implement both RPM and DEB based container distributions for full system call interface compatibility. To solve that problem, we turned to another fantastically simple dependency called jinja2. Jinja2 is a templating library that provides conditionals among other features. We use Jinja2 in our Dockerfiles to turn the Dockerfile DSL into a semi-imperative DSL. An example Heat Engine Dockerfile.j2 is shown below:


As can be seen, we can put all Heat engine related code in one Dockerfile.j2, but conditionally make choices about what code is run within the docker build operation based upon what type of container we are building. This permits us to maintain 90 containers for an intersection of RDO, OSP, and source distributions with CentOS, Debian, Fedora, Oracle Linux, RHEL, and Ubuntu container operating systems. Repetition is encouraged in Dockerfiles if true system call interface compatibility is desired, but with our choice of a Jinja2 templating system as a dependency, we remove the DRY principle violation.

The second pattern to learn is the orchestration process for a typical OpenStack service, which is as follows:

  1. Ansible processes main.yml
    1. Ansible processes config.yml
      1. Customization augmentation files merged with opinionated jinja2 templates
      2. JSON configuration file copied
      3. merged config file copied to location container can access via host bind-mount
    2. Ansible processes bootstrap.yml
      1. Create service database
      2. Create service database user
      3. Start bootstrap container
      4. Clean up bootstrap container
    3. Ansible processes start.yml
      1. Start the container using the docker module with appropriate options

By following these two patterns, it is very straightforward in ten hours to create a completely new deployable service based upon an upstream OpenStack server project. This engineering scalability is mandatory for deploying the Big Tent, where approximately ten new projects were added during the Liberty development cycle. This engineering scalability exists because Docker, Ansible, and Jinja2 are simple yet powerful dependencies.

Because Kolla made the right choices about dependencies, Kolla itself is a far better system. Kolla can deploy one-hundred node OpenStack Liberty clusters in approximately thirty minutes with full 3N high availability, Ceph storage, and deployment of HAProxy, Keepalived, Memcached, MariaDB + Galera, RabbitMQ, Keystone, Glance, Nova, Neutron, Cinder (CEPH Only), Swift, Murano, Heat, and Horizon. It can do all this work in a tidy 15,000 lines of code and is already “gold plated” at version 1.0.0 with minimal technical debt.

There is room for improvement by adding upgrade capability and more Big Tent services. We also have a long list of new work that can be done including machine bootstrapping with Ironic, a ReST management API, a configuration CLI, a status reporting GUI, and other interesting ideas the community has suggested. What is clear is that our core mission in the OpenStack Kolla project - the deployment of OpenStack in Docker containers using Ansible - is well executed due in part to well- considered dependency management.

Kolla at OpenStack Summit

If you're at OpenStack Summit in Tokyo, and you want to learn more about Kolla (and how it uses Ansible), be sure to attend this session:




Steven Dake

Steven Dake is bringing container technology to OpenStack on behalf of Cisco Systems, Inc. Steve serves as Kolla PTL for the Mitaka cycle, and served as PTL for the Liberty, Kilo, and Juno cycles. Steve also contributes to the design and implementation of Kolla. Prior to gaining an interest in container technology, Steve initiated and led the development of OpenStack Heat with a small development team while at Red Hat, Inc.

rss-icon  RSS Feed

Ansible Tower by Red Hat
Ansible In-Depth Whitepaper
Ansible Tower by Red Hat
Learn About Ansible Tower