network-automation-blog

A collection of posts on technologies used in the area of network automation and programmability.

View on GitHub

This post intends to talk about using TextFSM, a tool that is used to parse textual data to create structured data objects.

Introduction

The CLI on a network device is a good friend of a network engineer. It gives us the information we need in a no-nonsense manner, usually in the form of a table or some other structure that makes sense when you read it.

Taken from https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_ospf/command/iro-cr-book/ospf-s1.html#wp1717098229

Router# show ip ospf database
OSPF Router with id(192.168.239.66) (Process ID 300)
                 Displaying Router Link States(Area 0.0.0.0)
  Link ID       ADV Router    Age        Seq#     Checksum  Link count
172.16.21.6   172.16.21.6    1731    0x80002CFB    0x69BC       8
172.16.21.5   172.16.21.5    1112    0x800009D2    0xA2B8       5
172.16.1.2    172.16.1.2     1662    0x80000A98    0x4CB6       9
172.16.1.1    172.16.1.1     1115    0x800009B6    0x5F2C       1
172.16.1.5    172.16.1.5     1691    0x80002BC     0x2A1A       5
172.16.65.6   172.16.65.6    1395    0x80001947    0xEEE1       4
172.16.241.5  172.16.241.5   1161    0x8000007C    0x7C70       1
172.16.27.6   172.16.27.6    1723    0x80000548    0x8641       4
172.16.70.6   172.16.70.6    1485    0x80000B97    0xEB84       6
                Displaying Net Link States(Area 0.0.0.0)
  Link ID       ADV Router      Age        Seq#        Checksum
172.16.1.3  192.168.239.66     1245    0x800000EC      0x82E
                Displaying Summary Net Link States(Area 0.0.0.0)
  Link ID       ADV Router       Age        Seq#        Checksum
172.16.240.0   172.16.241.5    1152      0x80000077      0x7A05
172.16.241.0   172.16.241.5    1152      0x80000070      0xAEB7
172.16.244.0   172.16.241.5    1152      0x80000071      0x95CB

Above we can see the OSPF database on a Cisco router, telling us about connected routes in the same area as the router, routes as advertised from the designated router, and route summaries at the end. While this makes sense to us, a computer does not understand the semantics behind this output. Using TextFSM, we can describe the manner using which information can be extracted from this output.

What is TextFSM and how to use it

Going by the TextFSM GitHub repository wiki,

TextFSM is a Python module that implements a template based state machine for parsing semi-formatted text.

This is a pretty loaded sentence, so let us break it down a bit.

Great, so putting this together, we can deduce that TextFSM is a tool that allows you to write templates to describe the data to be parsed, gives developers the flexibility to define the different states the template can parse in, and works on semi-formatted data that have an apparent textual format in which they are represented.

TextFSM uses RegEx to parse data.

TextFSM template definitions

A TextFSM template has 2 components:

Values

These are the variables that need to be captured in the parsing process. TextFSM works in a manner where it builds a table of records that are parsed from semi-formatted text data, so declaring the values can be considered the same as declaring the columns of the table. A value is declared in the following way:

Value [option[,option...]] name regex

NOTE: The parts enclosed in square brackets [] mean that this part is OPTIONAL.

State definitions

After defining the values that will be recorded in a row, we need to tell TextFSM about the parsing states. Each state consists of some parsing rules that tell TextFSM what to do when it encounters a certain type of input. These rules are written using RegEx. Each state definition must be separated by a blank line. A state is defined as such:

stateName
  ^rule
  ^rule

There are 3 reserved states:

TextFSM allows you to define custom states of your own.

Rules

In a state, you can define one or more rules that tell TextFSM what to do when it encounters certain input lines. TextFSM takes in incoming strings and compares them with the rules using RegEx patterns. If the input line matches with the rule, then the action specified in the rule is executed. Once the action completes, TextFSM moves to the next input line and starts comparing it with the rules starting from the beginning of the state.

Rules are written as such:

^regex [-> action]

The rule starts with a carat symbol ^. This is followed with a RegEx pattern that is used to match with the input line. To capture a value when the rule is matched with the input line, a syntax of the form ${ValueName} is used. To specify the end of the RegEx pattern, $$ is used.

Actions

A rule can define what action needs to be executed when an input line matches with the RegEx pattern in the rule. An action is identified with the ` -> ` character after the RegEx pattern. There are 3 types of actions:

  1. Line action: Actions that apply to an input string match.
  2. Record action: Actions that apply to collected values.
  3. State transition: Action to change to a different state.

The default action that takes place when a rule matches with the input line is Next.NoRecord, which means go to next input line, start attempting to match with rules from beginning of current state, and don’t write a new row record.

Types of line actions:

Record options are optional actions that can be specified after a line action, separated by a fullstop ".". Types of record actions:

The fullstop . separator is mandatory only if both line and record actions are specified. If one or both are left as the implicit default then the dot is omitted. So, Next, Next.NoRecord and NoRecord do exactly the same thing.

A state transition allows the rule matching to be moved to another state from the current one. This state must be reserved, or defined in the template. In a rule that executes a state transition:

  1. First all actions (line and record) are executed.
  2. Then the next line is read.
  3. Finally the current state changes to the new state, and rule matching with the newly read input line continues in the new state.

If a rule uses Continue as a record option, it is not possible to change state in this rule.

An Error action is also allowed by TextFSM, in case you would like the template to fail if something unexpected shows up in the input lines. Error stops all line processing, discards everything collected so far, and throws an Exception.

^regex -> Error [word|"string"]

The word option for the error allows the throwing of a custom fail message.

Example TextFSM template

A great example of a TextFSM template is one to parse the output from the command show ip ospf database on a Cisco router, as we saw an output for this above. Posting it again for convenience (edited it a bit).

Taken from https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_ospf/command/iro-cr-book/ospf-s1.html#wp1717098229

Router# show ip ospf database
OSPF Router with id(192.168.239.66) (Process ID 300)
                 Displaying Router Link States(Area 0.0.0.0)
  Link ID       ADV Router    Age        Seq#     Checksum  Link count
172.16.21.6   172.16.21.6    1731    0x80002CFB    0x69BC       8
172.16.21.5   172.16.21.5    1112    0x800009D2    0xA2B8       5
172.16.1.2    172.16.1.2     1662    0x80000A98    0x4CB6       9
172.16.1.1    172.16.1.1     1115    0x800009B6    0x5F2C       1
172.16.1.5    172.16.1.5     1691    0x80002BC     0x2A1A       5
172.16.65.6   172.16.65.6    1395    0x80001947    0xEEE1       4
172.16.241.5  172.16.241.5   1161    0x8000007C    0x7C70       1
172.16.27.6   172.16.27.6    1723    0x80000548    0x8641       4
172.16.70.6   172.16.70.6    1485    0x80000B97    0xEB84       6
                Displaying Net Link States(Area 0.0.0.0)
  Link ID       ADV Router      Age        Seq#        Checksum
172.16.1.3  192.168.239.66     1245    0x800000EC      0x82E
                Displaying Summary Net Link States(Area 0.0.0.0)
  Link ID       ADV Router       Age        Seq#        Checksum
172.16.240.0   172.16.241.5    1152      0x80000077      0x7A05
172.16.241.0   172.16.241.5    1152      0x80000070      0xAEB7
172.16.244.0   172.16.241.5    1152      0x80000071      0x95CB

The TextFSM template to parse this would be as such:

# Taken from ntc-templates, https://github.com/networktocode/ntc-templates/blob/master/templates/cisco_ios_show_ip_ospf_database.textfsm

Value Filldown ROUTER_ID (\d+\.\d+\.\d+\.\d+)
Value Filldown PROCESS_ID (\d+)
Value Filldown AREA (\d+\.\d+\.\d+\.\d+|\d+)
Value LINK_ID (\d+\.\d+\.\d+\.\d+)
Value ADV_ROUTER (\d+\.\d+\.\d+\.\d+)
Value AGE (\d+)
Value LINK_COUNT (\d+)
Value TAG (\d+)

Start
  ^.*\(${ROUTER_ID}\) \(.* ${PROCESS_ID}\)
  ^.*\(Area ${AREA}\)
  ^${LINK_ID}\s+${ADV_ROUTER}\s+${AGE}\s+\S+\s+\S+\s+${LINK_COUNT} -> Record
  ^${LINK_ID}\s+${ADV_ROUTER}\s+${AGE}\s+\S+\s+\S+ -> Record
  ^\s+Type-5 AS External Link States -> Tag
  # Capture time-stamp if vty line has command time-stamping turned on
  ^Load\s+for\s+
  ^Time\s+source\s+is

Tag
  ^Link ID\s+ADV Router\s+Age\s+Seq#\s+Checksum\s+Tag -> Next.Clearall
  ^${LINK_ID}\s+${ADV_ROUTER}\s+${AGE}\s+\S+\s+\S+\s+${TAG} -> Next
  ^\s -> Start

EOF

This template would have something of this sort to visualize the data collected.

| ROUTER_ID      | PROCESS_ID | AREA    | LINK_ID     | ADV_ROUTER  | AGE  | LINK_COUNT | TAG |
|----------------|------------|---------|-------------|-------------|------|------------|-----|
| 192.168.239.66 | 300        | 0.0.0.0 | 172.16.21.6 | 172.16.21.6 | 1731 | 8          |     |
| 192.168.239.66 | 300        | 0.0.0.0 | 172.16.21.5 | 172.16.21.5 | 1112 | 5          |     |
| 192.168.239.66 | 300        | 0.0.0.0 | 172.16.1.2  | 172.16.1.2  | 1662 | 9          |     |
... (Abridged)

Using TextFSM with Python

Let’s say, using netmiko, you send a show command to your network device to get back some string output. Using a TextFSM template to parse it, and the textfsm Python module, you can get a structured output for the data to be parsed.

import textfsm
from netmiko import ConnectHandler

cisco_dev = {
  'device_type': 'cisco_ios',
  'host':   '10.10.10.10',
  'username': 'test',
  'password': 'password',
  'port' : 8022,          # optional, defaults to 22
  'secret': 'secret',     # optional, defaults to ''
}

net_connect = ConnectHandler(**cisco_dev)
output = net_connect.send_command("show ip ospf database")

template = open('template_name.textfsm') # open textfsm file with .textfsm extension, or .tpl also works.

re_table = textfsm.TextFSM(template) # initialize textfsm object
fsm_results = re_table.ParseText(output) # parse output text with textfsm object.

"""
ParseText output returns back a list of tuples.
First tuple is the header, every subsequent tuple is a row.
Let us make a dict with relevant key value pairs out of this!
"""

results = list()
for item in fsm_results:
    results.append(dict(zip(re_table.header, item)))

return results

And you get back a list of dictionaries like this:

[
  {
    "ROUTER_ID": "192.168.239.66",
    "PROCESS_ID": "300",
    "AREA": "0.0.0.0",
    "LINK_ID": "172.16.21.6",
    "ADV_ROUTER": "172.16.21.6",
    "AGE": "1731",
    "LINK_COUNT": "8",
    "TAG": "",
  },
  {
    "ROUTER_ID": "192.168.239.66",
    "PROCESS_ID": "300",
    "AREA": "0.0.0.0",
    "LINK_ID": "172.16.21.5",
    "ADV_ROUTER": "172.16.21.5",
    "AGE": "1112",
    "LINK_COUNT": "5",
    "TAG": "",
  },
  ...
  # ABRIDGED
]