Event-Driven Ansible - ChatOps - From Chat to Action

Like any well-rounded individual, in times of intense concentration, you will find me talking to myself in search of some hidden knowledge that I might have received in a dream, or perhaps quoting something from a fantasy novel about wizards and creatures in an attempt to fix a problem. Unfortunately, wearing a robe and shouting “Repairo Network!” while pointing my pen toward the device has yet to help in any situation. 

At the 2023 AnsibleFest, as part of the main stage demonstration, I used the magic of Event-Driven Ansible to integrate ChatOps in our fictional infrastructure drama. ChatOps is not new, but I think it's a pretty cool way to make changes or interact with your infrastructure. 

We know that Event-Driven Ansible requires a source for events, a list of conditions which we call rulesets, and ultimately an action to match those conditions, which makes it perfect to use as a chatbot-type system.

For me to have a heart-to-heart with my beloved network, I will need to configure my chat as a source of events for Event-Driven Ansible, and to do this, I will use the webhook source plugin, which is part of the ansible.eda collection. Many enterprise chat platforms provide a method to send and receive webhooks to trigger words in the chat, which is perfect for what we want to do.  

Event-Driven Ansible ChatOps mechanism

• We will use an outgoing webhook from the chat application to Event-Driven Ansible.
• Event-Driven Ansible will match the payload against its defined conditions.
• Once a condition is met, the corresponding action will take place.
• Feedback notification will be sent from the action as an incoming webhook to the chat.

Now that the fundamentals are done, let's look at how we could use this. 

I have a network configured with three interconnected switches, got my chat system up and have Event-Driven Ansible listening with my rulebooks activated! 

Event-Driven Ansible NetOps - ChatOps use case

If you have read one of the previous blogs about using network telemetry with Event-Driven Ansible, you would see that we are streaming telemetry from the network into Kafka and from there using Event-Driven Ansible to process events. In a large-scale environment, we would typically use a centralized tool like CloudVision from Arista, or Cisco ThousandEyes to gather the telemetry and forward them to Event-Driven Ansible.

> Ding ding, incoming message!

For our first example, we will simulate a BGP failure and take all the steps we would typically perform to troubleshoot by codifying  them in an Ansible Playbook that we can use as an action. Below is an example of a failure in BGP, followed by my rulebook:

ceos3(config-if-Et3)#show ip bgp summary vrf all

BGP summary information for VRF default
Router identifier 3.3.3.3, local AS number 65003
Neighbor Status Codes: m - Under maintenance
  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc
  10.0.1.2         4  65002             24        24    0    0 00:17:24 Estab 4  4
  10.0.1.1         4  65001             22        23    0    0 00:01:20 Idle(NoIf)        

BGP failure output from Leaf02

The below rulebook is allowing us to listen to our Kafka topic for messages from our switches. 

---
- name: Switch State Event from Arista

  hosts: all
  sources:
   - ansible.eda.kafka:
       host: broker
       port: 9092
       topic: network

  rules:
   - name: BGP change detected
     condition: event.fields.session_state == "IDLE"
     action:
       run_playbook:
        name: bgp_troubleshooting.yml
        post_events: true

   - name: Notify Chat
     condition: event.status is defined
     action:
        run_playbook:
          name: notify_chat.yml

rulebook/switch_status.yml  Ansible Rulebook

Our first condition evaluates the event payloads coming from our Kafka topic and we are looking for an “IDLE” state in the BGP status. Once this condition has been met, we will run a troubleshooting playbook. What I want to point out is the post_events: true in the action. 

Posting events with Event-Driven Ansible

Post_events allows us to provide feedback from the playbook back into the rulebook. If you are running troubleshooting on an event, we would perhaps want to know what the outcome is of specific tasks, so we could decide on the next course of action or gather all the relevant information to create an incident ticket.

In my example, I have a second condition which is looking for a fact that gets set whenever this troubleshooting is done. Once it is done, the notify_chat.yml playbook will trigger and my company chat will be notified of the issue and the results of the first phase of troubleshooting.

---
 - name: Send notification message via Mattermost
      community.general.mattermost:
       url: http://chat.mydomain.com:8065
       api_key: 6xyqnbqdpfg89eneqg13146azc 
       attachments:
         - text: "!!!!!! ALERT !!!!!!"
           color: '#ff00dd'
           title: BGP ERROR
           fields:
            - title: Issue
              value: "BGP {{ ansible_eda.event.fields.session_state }} Error!"
              short: true
            - title: Details
              value: "Neighbor Address: {{ ansible_eda.event.tags.neighbor_address }} from switch {{ ansible_eda.event.name }}"
              short: true
      delegate_to: localhost

playbooks/notify_chat.yml Ansible Playbook

Mattermost chat output

> BRB, got to automate!

The previous example used Event-Driven Ansible to respond to an event by running a troubleshooting playbook and then notifying the team of the details. 

But what if I want to whisper sweet nothings to Event-Driven Ansible and have it do some work for me?

Let’s use the chat to initiate a configuration backup of one or more of our switches. We will configure an outgoing webhook based on a trigger word and point it to my Event-Driven Ansible endpoint. 

We’ll use a trigger word in the chat like “backup” that will be used in the event condition, but what I want is to be able to say “backup leaf02.mydomain.com” and have Event-Driven Ansible backup the correct switch. So, in my rulebook I will define a condition looking for that “backup” trigger, and I will then action a playbook that will process the text payload so I can get the hostname of the switch. 

---
- name: Listen for events on a webhook
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000

  rules:

   - name: Filter payload for hostnames
      condition: event.payload.text is match("backup")
      action:
      run_playbook:
       name: filter_regex.yml
       post_events: true

    - name: Backup specified switch
      condition: event.switch_devices is defined
      action:  
      run_playbook:
       Name: switch_backup.yml

rulebook/chatops.yml rulebook

---
- name: Regex Processing for hostnames
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    eda_input: "{{ ansible_eda.event.payload.text }}"  # access event payload
    regex_pattern: '\b(\w+\.mydomain\.com)\b'          # can supply via vars.yml

  tasks:

    - name: Extract switch name from text
      set_fact:
       cacheable: yes
       switch_devices: "{{ eda_input | regex_findall(regex_pattern) }}"
       regex_proc: done

playbook/filter_regex.yml playbook

This “filter_regex.yml” playbook will process the event payload and look for the hostname of the switch. We then set it as a fact so it is persistent. What is super cool is that Event-Driven Ansible can access facts that have been set in playbooks, and in conjunction with post_events, we are able to feed this back into the rulebook for processing. To access this fact, we can simply use event.switch_devices.

The below playbook is triggered to backup the desired switch when the event.switch_devices is defined:

---
- name: Backup Switch
  hosts: "{{ ansible_eda.event.switch_devices }}"
  gather_facts: false
  connection: httpapi
  vars:
   ansible_httpapi_port: 443
   ansible_httpapi_use_ssl: true
   ansible_httpapi_validate_certs: false

  tasks:

  - name: configurable backup path
    arista.eos.eos_config:
     backup: yes
     backup_options:
     filename: "{{ ansible_eda.event.switch_devices }}.cfg"

playbook/switch_backup.yml playbook

There you have it! We have been able to use incoming and outgoing webhooks as sources for Event-Driven Ansible in a ChatOps environment, allowing us to drive automation with the power of the typed word! 

Call to Action

• Watch the action in latest demo
• Check out our NetOps and Event-Driven Ansible blog
• Give the Self-paced labs a try 
• Check out the Getting Started with Event-Driven Ansible training
• Grab these code examples on the GitHub repo
• Learn more about Event-Driven Ansible 
• Join the Ansible Community Forum to discuss this blog post