The Network CLI is Dead, Long Live XML! (just kidding, it’s an Ansible+NETCONF+YANG Deep Dive)

Now that I've startled you, no, the network CLI isn’t going away anytime soon, nor are people going to start manipulating XML directly for their network configuration data. What I do want to help you understand is how Ansible can now be used as an interface into automating the pushing and pulling of configuration data (via NETCONF) in a structured means (via YANG data models) without having to truly learn about either of these complex concepts. All you have to understand is how to use the Ansible Content Collection as shown below, obfuscating all technical implementation details that have burdened network operators and engineers for years.

Setting the stage

Before we even start talking about NETCONF and YANG, our overall goal is for the network to leverage configuration data in a structured manner. This makes network automation much more predictable and reliable when ensuring operation state. NETCONF and YANG are the low-level pieces of the puzzle, but we are making it easier to do via well known Ansible means and methodologies.

What we believe as Ansible developers is that NETCONF and YANG aren't (and shouldn't) be quintessential or ultimate goals for network automation engineers. You should not need to memorize complex YANG data models or NETCONF RPC commands. It should be an implementation detail that’s already solved elsewhere, such as Ansible. It’s more about the goal of solving the problem of treating network configuration data as platform agnostic key-value pairs in a standardized schema. NETCONF and YANG fit the bill here, and then use Ansible as the primary interface in for users.

Here’s the problem though: NETCONF and YANG are incredibly complex and somewhat unapproachable concepts to network engineers, therefore the implementation is still somewhat rare and unpopular. If we can take NETCONF and YANG and make Ansible as the presentation layer for users, this becomes much more approachable to network engineers that may not be able to (or have the desire to) become python programmers.

So let’s do this in an Ansible “friendly” way, using Ansible content such as Ansible modules, roles and plugins wrapped up into a Collection. You don’t have to worry about the YANG model itself or the actual XML that’s being passed. It can all be represented by YAML.

Types of YANG Implementations

Although the YANG schema is indeed a standard, how the data itself is represented may vary depending on the implementation method. Therefore, there are YANG models that are defined and maintained by specific network vendors, and models that are defined and maintained by the community that are vendor neutral.

• Vendor defined:
  • The data represented in the YANG model vary based on vendor implementation and the released YANG models are published and maintained by vendors themselves. E.g.: https://github.com/YangModels/yang/tree/master/vendor/
• Vendor neutral:
  • Standard YANG model defined by IETF, IEEE, etc. https://github.com/YangModels/yang/tree/master/standard/ietf/RFC
  • OpenConfig, which is an informal working group of network operators that promotes a vendor-neutral model. 
    https://github.com/openconfig/public/tree/master/release/models.

Therefore, it can become extremely difficult to manage multiple YANG implementations depending on the platform being automated. The goal is to leverage Ansible to standardize and normalize the management of the YANG data models on behalf of the user.

How does NETCONF factor in?

The NETCONF protocol is an IETF standard for managing configuration as well as retrieving state data from a NETCONF enabled device. It uses SSH for transport to send RPC requests and receives the responses in XML format. The protocol defines standard RPC endpoints like edit-config, get, get-config, copy-config, delete-config, lock, unlock and many others to manage the remote device. Ansible has supported NETCONF as a standard connection method for quite some time, and is now provided by the ansible.netcommon Collection for Ansible 2.9.10 and later.

YANG to NETCONF mapping

The above figure represents how the hierarchy of the YANG model maps to that of the NETCONF XML RPC. The <edit-config> and the <running> tags represent the RPC request to edit the running datastore configuration given in XML payload onto the device. 

The data hierarchy in the payload starting with interfaces tag followed by interface tag is defined in the YANG model wherein interfaces is defined as a container node, followed by the interface child node as a list (since there are multiple interfaces to be configured).

Thus by parsing the YANG model, the Network Management System (NMS) can either generate and/or validate the NETCONF XML RPC structure to be sent to the device in a programmatic approach.

With this context, let's now move on to talk about the Ansible YANG Content Collection. While developing this Collection, we adhered to the following requirements:

• Functions with all variants of YANG models
• Abstracts YANG related complexities from the user via Ansible modules
• Uses structured data in JSON format as input/output
• Fetches YANG models from a network appliance at runtime (if supported)
• Renders skeleton structured data (JSON, XML, YANG tree) for given YANG model

Based on these guidelines, let’s go through the modules supported in this Collection.

The community.yang.fetch module

This module fetches the YANG model and its dependent yang models from the device using the NETCONF get schemas capability, if supported, and optionally stores the YANG files locally on disk.

The task to get the list of supported YANG models on the remote appliance:

- name: Fetch the list of supported yang model names
  community.yang.fetch:

Example: https://gist.github.com/ganeshrn/f45d34602058aa5eeacc188b14f05206

The output of this task runs against Cisco IOS XR version 6.1.3 and can be referred to the following gist file.

To fetch the given YANG model provided by name option along with its dependencies, refer to the following example:

- name: Fetch given yang model and it’s dependent from remote host
  community.yang.fetch:
    name: Cisco-IOS-XR-ifmgr-cfg
    dir:"{{playbook_dir}}/{{inventory_hostname}}/yang_files"
  register: result

Example: https://gist.github.com/ganeshrn/b03ebddce9579ea8749a7889d638a2a5

The fetched YANG models are stored at the location given by dir option.

ls iosxr01/yang_files/
Cisco-IOS-XR-ifmgr-cfg.yang     Cisco-IOS-XR-types.yang

Fetch all the YANG models supported by the remote host by assigning the all value to the name option:

- name: Fetch all the yang models and store it in dir location
  community.yang.fetch:
    name: all
    dir:"{{playbook_dir}}/{{inventory_hostname}}/yang_files"
  register: result

Example: https://gist.github.com/ganeshrn/d82648f74cd217b724cf0ecb7ddeaa94

The YANG models are fetched and stored at the location given by dir option.

The community.yang.get module

This module uses the NETCONF get RPC to fetch configuration data from the remote host and render it in JSON format (as per RFC 7951 which defines JSON encoding of data modelled with YANG).

To fetch a subset of running configuration, render it in JSON format and save it in file use the below tasks:

- name: get interface configuration using cisco iosxr yang model
  community.yang.get:
    filter: |
           <interface-configurations xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-ifmgr-cfg">
             <interface-configuration>
               <interface-name>GigabitEthernet0/0/0/0</interface-name>
             </interface-configuration>
           </interface-configurations>
    file: "{{ playbook_dir }}/{{inventory_hostname}}/yang_files/Cisco-IOS-XR-ifmgr-cfg.yang"
    search_path: "{{ playbook_dir }}/{{inventory_hostname}}/yang_files"
  register: result- name: copy json config to file
  copy:
    content: "{{ result['json_data'] | to_nice_json }}"
    dest: "{{playbook_dir}}/{{inventory_hostname}}/config/interfaces.json"

Example: https://gist.github.com/ganeshrn/3fe905aef0556fbfa7b8c2e466d1c0c5

The filter option in community.yang.get task refers to the subset of the configuration that should be fetched from the remote host; it currently supports XML format and will soon support JSON format as well. The filter structure can be derived from output of community.yang.generete_spec task, which I’ll explain later. The file option corresponds to the YANG model that the configuration adheres to. The search_path points to the directory location in which all the dependent YANG files are stored. The copy task copies the retrieved configuration in JSON format into a file path provided by the dest option.

After the task is run, the sample contents of interfaces.json file is:

$cat iosxr01/config/interfaces.json
{
   "Cisco-IOS-XR-ifmgr-cfg:interface-configurations": {
       "interface-configuration": [
           {
               "active": "act",
               "description": "manually configured",
               "interface-name": "GigabitEthernet0/0/0/0"
           }
       ]
   }
}

The community.yang.configure module

This module takes the JSON configuration as input (as per RFC 7951, which defines JSON encoding of data modelled with YANG), pre-validates the config with the corresponding YANG model, and converts input JSON configuration to XML payload to be pushed on the remote host using the netconf_config module.

To demonstrate this task, we will first modify the interface config fetched using the community.yang.get task earlier:

$cat iosxr01/config/interfaces.json
{
   "Cisco-IOS-XR-ifmgr-cfg:interface-configurations": {
       "interface-configuration": [
           {
               "active": "act",
               "description": "configured using Ansible YANG collection",
               "interface-name": "GigabitEthernet0/0/0/0"
           }
       ]
   }
}

After that push, use the configure module:

- name: configure interface using structured data in JSON format
 community.yang.configure:
   config: "{{ lookup('file', './iosxr01/config/interfaces.json') | to_json }}"
   file: "{{ playbook_dir }}/{{inventory_hostname}}/yang_files/Cisco-IOS-XR-ifmgr-cfg.yang"
   search_path: "{{ playbook_dir }}/{{inventory_hostname}}/yang_files"
 register: result

Example: https://gist.github.com/ganeshrn/89137bf3303aaea3e8907e1b638f0eba

The config option accepts the value in JSON format and the file lookup plugin is used to read the content of the file updated. The file option corresponds to the YANG model that the configuration adheres to. The search_path points to the directory location in which all the dependent YANG files are stored. This task reads the YANG file and does pre-validation on the value of the config option to check for correctness of the configuration that is to be pushed on to the device based on the constraints defined in the YANG file.

The combination of community.yang.get and community.yang.configure is particularly useful in brownfield deployments to retrieve the current running configuration on the device, update it to the required values and push back onto the device.

The community.yang.generate_spec module

Handcrafting JSON and/or XML data manually is not straightforward, nor an activity that is realistic to an end-user. The user should be able to easily generate the configuration data structure using the given YANG model, and generate the corresponding JSON, XML schema and the YANG tree representation (as per RFC 8340) of the model. This is particularly useful if the given hierarchy is not configured on the device or it is a greenfield deployment.

To generate the reference JSON, XML, and YANG tree schema, run the following task:

- name: generate spec from cisco iosxr interface config data and store it in file
 community.yang.generate_spec:
   file: "{{ playbook_dir }}/{{inventory_hostname}}/yang_files/Cisco-IOS-XR-ifmgr-cfg.yang"
   search_path: "{{ playbook_dir }}/{{inventory_hostname}}/yang_files"
   json_schema:
     path: "{{ playbook_dir }}/{{ inventory_hostname }}/spec/Cisco-IOS-XR-ifmgr-cfg.json"
   xml_schema:
     path: "{{ playbook_dir }}/{{ inventory_hostname }}/spec/Cisco-IOS-XR-ifmgr-cfg.xml"
     defaults: True
   tree_schema:
     path: "{{ playbook_dir }}/{{ inventory_hostname }}/spec/Cisco-IOS-XR-ifmgr-cfg.tree"

Example: https://gist.github.com/ganeshrn/f52b984a106247e6e3e843be4a3cc7c2

The file option corresponds to the YANG model for which the JSON, XML and YANG tree schema should be generated. The search_path points to the directory location in which all the dependent YANG files are stored. The optional path option within the json_schema, xml_schema and tree_schema options identifies the file where the generated schema are to be stored. When defaults option is set to true either via json_schema or xml_schema schemas are generated by adding the default values for data as mentioned in the YANG model. 

After running the task, the sample contents within the files are:

• Cisco-IOS-XR-ifmgr-cfg.json
• Cisco-IOS-XR-ifmgr-cfg.xml
• Cisco-IOS-XR-ifmgr-cfg.tree

Resources and Getting Started

• Download, install and try the YANG Collection by installing it from Ansible Galaxy: https://galaxy.ansible.com/community/yang
• If you are further interested get involved in the project by contributing directly to the YANG Collection repository, refer to the upstream GitHub repository: https://github.com/ansible-collections/community.yang 
• Attend the Ansible fest 2020 breakout session on “Network Automation using Ansible and YANG”
• Attend the Advanced Track at Network Automation Summit on October 5 and listen to Ganesh do his Deep Dive on Ansible Collections for Network Automation