Ansible and ChatGPT: Putting it to the test

January 23, 2023 by Craig Brandt

Image: DALL-E

Artificial intelligence (AI) is revolutionizing how we work and play in exciting ways. At first glance, AI tools, such as ChatGPT, seem to provide all the correct answers. But once we delve deeper and implement the suggestions, it often isn’t as effortless as it appears. This is especially true when generating code.

In this blog, we wanted to put ChatGPT to the test and see how it fares with developing  Ansible Playbooks and share our results. We’ll also cover the experience and feedback from developers across domains.

We’ll also provide more information on our upcoming automation AI superpower, Project Wisdom.

First, let’s briefly discuss what ChatGPT is and how it works.

 

What is ChatGPT?

“We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests.”

OpenAI ChatGPT release announcement

 

ChatGPT is a chatbot developed by OpenAI and built on their GPT (Generative Pre-trained Transformer) 3.5 large language model.

Large language models (LLM) are trained on massive amounts of data to predict the next word in a sentence. GPT 3.5 enables ChatGPT to write paragraphs and pages of content,  interact conversationally, and provide human-like answers.

ChatGPT uses two techniques to improve its responses: supervised and reinforcement learning.

More information on ChatGPT can be found on the OpenAI ChatGPT release blog.

Next, let's put ChatGPT to the test using a typical scenario we see with our customers.

 

Generating Ansible content with ChatGPT

We’ll ask ChatGPT to write an Ansible Playbook to create three AWS EC2 instances. We chose this example as it’s an everyday use case with many samples available for ChatGPT to derive a recommendation.

Our test environment consists of:

We kept the question simple and broad - “Write an Ansible Playbook to create 3 AWS ec2 instances”. 

ChatGPT - “Create 3 AWS ec2 instances” code output. 

The “good”

ChatGPT did a great job understanding what I asked for. The generated Ansible Playbook included my specific request to create three EC2 instances, as indicated in the count: 3 variable. ChatGPT also kept the variable names consistent. For example, the ec2 variable parameters are referenced in the sequential tasks.

It also provided a concise, easy-to-understand explanation of the variables and playbook.

ChatGPT - “Create 3 AWS ec2 instances” playbook explanation. 

The “not so good”

Several times, I asked ChatGPT the same question and received a different answer. This could be acceptable when working individually, but consistent coding standards are crucial when working at scale with multiple team members or departments.

Note

Ansible Lint, now fully supported with the launch of Red Hat Ansible Automation Platform 2.3, provides profile support enabling automators to apply your organization’s coding standards in real time using the Ansible Visual Studio Code extension. Please visit the Creating Custom Rules for Ansible Lint blog for more information.

I’ve configured my test environment with all the necessary AWS prerequisites. We’re ready to test!

❯ ansible-playbook playbooks/ec2_instances.yml
…output omitted…
ERROR! couldn't resolve module/action 'ec2'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/Users/craig/demos/ansible/public_demos/local_dev/chatgpt-blog/playbooks/cloudops.yml': line 7, column 5, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  tasks:
  - name: Launch EC2 instances
    ^ here
…output omitted…

Ansible-playbook output. 

On my initial run, I received the following error:

ERROR! couldn't resolve module/action 'ec2'. This often indicates a misspelling, missing collection, or incorrect module path.

The ec2 module was deprecated and removed in the upstream community Ansible package version 4.0.0. This module was replaced with the e2_instance module based on boto3.  ChatGPT, unfortunately, recommended a deprecated Ansible module removed in 2021.

Note

Starting in Ansible Core 2.10, we moved to an improved architecture with Ansible Content Collections, the recommended method for developing new Ansible content.

Let’s update the first task with the new amazon.aws.ec2_instance module, and use the Ansible Visual Studio Code extension to lint it for best practices.

…output omitted…
  vars:
    region: us-east-1
    instance_type: t2.micro
    image:
      id: ami-0dba2cb6798deb6d8
    key_name: mykey
    security_group: mysecuritygroup
    count: 3
  
  tasks:
  - name: Create EC2 instances
    amazon.aws.ec2_instance:
      key_name: "{{ key_name }}"
      security_group: "{{ security_group }}"
      instance_type: "{{ instance_type }}"
      image: "{{ image }}"
      region: "{{ region }}"
      count: "{{ count }}"
      wait: true
      vpc_subnet_id: subnet-1234abcd
      network:
        assign_public_ip: true
    register: ec2
…output omitted…

Updated “Create EC2 instances” task.

 Let’s review the changes to the first task:

  • The image variable was changed to a YAML dictionary containing the id value.
  • We used the Fully Qualified Collection Name (FQCN) module name - amazon.aws.ec2_instance
  • The network module parameter was changed to a YAML dictionary.
  • The boolean module parameters, wait and assign_public_ip, were changed from yes to true.

The second time’s the charm! Let’s re-run the playbook with the updated task.

❯ ansible-playbook playbooks/ec2_instances.yml
...output omitted...
TASK [Create EC2 instances] *******************************************
changed: [localhost]

TASK [Wait for instances to be in running state] ***********************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'public_ip'. 'dict object' has no attribute 'public_ip'\n\nThe error appears to be in '/home/craig/demos/ansible/public_demos/local_dev/chatgpt-blog/playbooks/ec2_instances_fixed.yml': line 29, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    register: ec2\n  - name: Wait for instances to be in running state\n    ^ here\n"}
...output omitted...

Ansible-playbook output. 

This time, the first task ran successfully. However, the second task, [Wait for instances to be in running state], produced the following error:

fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'public_ip.'

Why the error? The amazon.aws.ec2_instance module now saves the output to public_ip_address instead of public_ip. Let’s update the variable name in the task and re-run the playbook.

❯ ansible-playbook playbooks/ec2_instances.yml
...output omitted...
TASK [Wait for instances to be in running state]***********************
msg: 'value of state must be one of: absent, drained, present, started, stopped, got: running'
...output omitted...

Ansible-playbook output. 

This time it returned a different error. The [Wait for instances to be in running state] task uses the wait_for Ansible module. Unfortunately, ChatGPT’s suggested playbook uses an incorrect value for the state parameter - running

I’ve updated the state parameter to present and linted the task using the Ansible Visual Studio Code extension. We’re nearly there! Here’s the updated task:

 - name: Wait for instances to be in running state
    ansible.builtin.wait_for:
      state: present
      host: "{{ item.public_ip_address }}"
      port: 22
    with_items: "{{ ec2.instances }}"

Updated “Wait for instances to be in running state” task.

Let’s review the changes:

  • We used the module’s Fully Qualified Collection Name (FQCN) - ansible.builtin.wait_for.
  • The state parameter was changed from running to present.
  • The item.public_ip variable was changed to item.public_ip_address.

Let’s rerun our updated playbook.

❯ ansible-playbook playbooks/ec2_instances.yml
...output omitted...
TASK [Wait for instances to be in running state] **********************
ok: [localhost] => (item={'ami_launch_index': 0, 'image_id': 'ami-02e0bb36c61bb9715', 'instance_id': 'i-0ee1d103e7734a979'
...output omitted...
TASK [Print instance information] *************************************
ok: [localhost] => (item={'ami_launch_index': 2, 'image_id': 'ami-02e0bb36c61bb9715', 'instance_id': 'i-02ce52702ef9d8143'
...output omitted...

Ansible-playbook output. 

It worked! We’ve updated the playbook run using the latest, supported version of Ansible and applied the best practices using Ansible Lint.

Here’s the final playbook version.

---
- name: Create EC2 instances
  hosts: localhost
  gather_facts: false
  vars:
    region: us-east-1
    instance_type: t2.micro
    image:
      id: ami-0dba2cb6798deb6d8
    key_name: mykey
    security_group: mysecuritygroup
    count: 3
  tasks:
    - name: Create EC2 instances
      amazon.aws.ec2_instance:
        key_name: "{{ key_name }}"
        security_group: "{{ security_group }}"
        instance_type: "{{ instance_type }}"
        image: "{{ image }}"
        region: "{{ region }}"
        count: "{{ count }}"
        wait: true
        vpc_subnet_id: subnet-1234abcd
        network:
         assign_public_ip: true
      register: ec2
    - name: Wait for instances to be in running state
      ansible.builtin.wait_for:
        state: present
        host: "{{ item.public_ip_address }}"
        port: 22
      with_items: "{{ ec2.instances }}"
    - name: Print instance information
      ansible.builtin.debug:
        var: item
      with_items: "{{ ec2.instances }}"

Final Ansible Playbook version.

Note

It would be possible to use the Ansible Automation Platform compatibility execution environment, available to our subscribed customers, and automation content navigator to run automation content with Ansible Engine 2.9 compatibility. However, we strongly recommend creating new Ansible content using the improved format to future-proof your automation.

 

Summarizing the experience

ChatGPT’s conversational style enabled us to provide our request in simple, easy-to-understand language, and the Ansible Playbook it generated reduced our time to get started. The playbook, however, needed several modifications to get it working with the latest Ansible version.

ChatGPT’s recommendation also didn’t consider multiple factors, such as best practices, future compatibility, and, more importantly, context.

 

Context. The missing ingredient

Organizational context is the secret ingredient in scalable, enterprise-grade automation content. At this stage, ChatGPT and similar AI tools can’t provide a recommendation specific to your organization’s environment or products. 

The usefulness of the generated content relies heavily on your understanding and approach to the problem you’re trying to solve.

 

Developer community sentiment

“The scary part was just how confidently incorrect it was.”

Hacker News

 

AI is already helping new and experienced programmers develop code. However, many developers who’ve tested ChatGPT and similar AI tools share our opinion - It won’t do their job for them quite yet.

In fact, Stack Overflow implemented a temporary ban on the text generated using ChatGPT due to the influx of incorrect answers.

 

“The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce.”

Stack Overflow

 

Posts on forums, such as Hacker News, echo the sentiment. AI reduces development time, but AI-generated code needs to be reviewed, and without a clear understanding of what you want to achieve, it generally won’t provide the answer you’re looking for.

This is where we envision our forthcoming exciting AI project, Project Wisdom, moving beyond generic Ansible content generation to becoming a trusted Ansible content advisor.

 

The future: Project Wisdom

Project Wisdom, developed in close collaboration with IBM Research, is the future we envision to give you Ansible AI superpowers.

Red Hat is the world’s leading provider of enterprise open source solutions, and we believe in collaborating openly with customers and the community. Project Wisdom builds on this commitment and leverages our experience in creating enterprise products the open source way, trusted by more than 90% of Fortune 500 companies.

We envisage Project Wisdom moving beyond generic recommendations provided by multiple AI-based tools available today to an enterprise-grade solution giving our customers the competitive automation edge.

 

Attribution

We plan for Project Wisdom to recognize Ansible content contributions by sharing information about the source data used to train its models. If recommendations are similar to the training data used, Project Wisdom will reference the potential source, its author, and the content license. 

This acknowledgment aims to foster and strengthen the collaborative relationships that make open source the transformative platform for innovation.

 

Content with context

We intend for Project Wisdom to generate Ansible content with context. Imagine having Ansible Playbooks instantaneously generated based on your unique environment and automation content. We aim to provide the capability by offering customers private models trained to their specific needs.

 

Unified experience

We plan to deliver this in a unified experience with Project Wisdom seamlessly integrated into our Ansible developer content tooling. 

Project Wisdom is designed with Ansible in mind, and we want it to be your trusted Ansible content advisor.

Please stay tuned for more exciting announcements and sign-up to test Project Wisdom.

 

Closing words from the ChatGPT expert

We thought it best to ask the prominent expert on ChatGPT to write our closing to this blog: ChatGPT itself.

 

Where to go next

Share:

Topics:
Ansible, Ansible Automation, Ansible Wisdom


 

Craig Brandt

Craig Brandt is a Principal Technical Marketing Manager for Ansible Automation Platform. Prior to this position, Craig served as a Solution Architect representing Red Hat at the IBM Services Integration Hub. He focused on large, complex deals that covered EMEA, LATAM and Canada regions. He brings over 16 years of experience in the IT field that covers automation, containerisation, management, operations, development and solution design

Categories

See All


rss-icon  RSS Feed

AF 2022 - Blog static promo