In this one we’re going to look at some of the more complex processing in TTP. Often when you want to customize the data then you have two options in reality. The first option is to do the processing within Python i.e. outside of TTP itself. This is a flexible option because you have structured data then you can just use all the various data manipulation techniques found in Python. The second option is to use a macro within TTP and do the manipulation of the data there instead.
Clearly which option you use can come down to personal preference or in some cases it is essential to make changes to the output within TTP e.g. to stop values being overwritten or to ensure the data is processed correctly.
In this tutorial, we are going to look at the macro function within TTP and learn how to use it. We’re going to use a custom macro (it’s actually a Python function under the hood) to change some data. For this I am going to build a parser for Nokia SROS for the command “show bgp summary”. The source CLI and initial setup is below. We start with importing our modules pprint and ttp. We then put the CLI output into a multi-line string called data_to_parse. If you’ve looked at my previous posts then this will already be familiar.
import pprint
from ttp import ttp
data_to_parse="""
A:R1# show router bgp summary
===============================================================================
BGP Router ID:10.22.0.25 AS:64400 Local AS:64400
===============================================================================
BGP Admin State : Up BGP Oper State : Up
Total Peer Groups : 1 Total Peers : 3
Total BGP Paths : 820 Total Path Memory : 10122
Total IPv4 Remote Rts : 0 Total IPv4 Rem. Active Rts : 0
Total IPv6 Remote Rts : 0 Total IPv6 Rem. Active Rts : 0
Total Supressed Rts : 0 Total Hist. Rts : 0
Total Decay Rts : 0
Total VPN Peer Groups : 5 Total VPN Peers : 5
Total VPN Local Rts : 69
Total VPN-IPv4 Rem. Rts : 303 Total VPN-IPv4 Rem. Act. Rts: 32
Total VPN-IPv6 Rem. Rts : 0 Total VPN-IPv6 Rem. Act. Rts: 0
Total L2-VPN Rem. Rts : 0 Total L2VPN Rem. Act. Rts : 0
Total VPN Supp. Rts : 0 Total VPN Hist. Rts : 0
Total VPN Decay Rts : 0
Total MVPN-IPv4 Rem Rts : 0 Total MVPN-IPv4 Rem Act Rts : 0
Total MDT-SAFI Rem Rts : 0 Total MDT-SAFI Rem Act Rts : 0
Total MSPW Rem Rts : 0 Total MSPW Rem Act Rts : 0
Total FlowIpv4 Rem Rts : 0 Total FlowIpv4 Rem Act Rts : 0
Total IPv4 Backup Rts : 0 Total IPv6 Backup Rts : 0
===============================================================================
BGP Summary
===============================================================================
Neighbor
AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
10.22.1.8
64400 5420893 0 39d23h23m 101/32/69 (VpnIPv4)
426383 0
10.17.2.11
64400 5322002 0 35d20h09m 101/0/69 (VpnIPv4)
425237 0
10.17.4.11
64401 5391552 0 27d20h00m 101/0/69 (VpnIPv4)
425918 0
10.104.34.92
2200* 0 0 14d00h13m Connect
1 0
10.124.34.121
2200* 0 0 00h38m03s Active
1 0
-------------------------------------------------------------------------------
"""
If you look at the CLI Output it’s got some permutations which need to be managed. The first one is the AS Number has an asterisk after it if the BGP session is not established. We may or may not care about that but for the purposes of the tutorial there is a requirement to get rid of that. The second one is that unlike a Cisco CLI the word “established” is not shown in the output. We can see neighbors like 10.22.1.8 have an established BGP Session but it’s not explicitly called out. Conversely, the ones which are not established like 10.104.34.92 do not explicitly state they are not established either. We can only infer from the status of Connect and Active that these sessions are not up. Being precise it would be good to infer if a neighbor session is Established or not from the source data. That will be our second requirement.
Summary of our additional requirements are:
- Remove the asterisk from the remote ASN which has the non-established neighbors
- Infer into the data whether the session is Established or not.
Let’s start by building the template as normal. If we look at the CLI output we can easily capture the first part. We just need to place variables against BGP_Router_ID Local AS, BGP Admin State, BGP Oper State and the Total BGP Paths. We could also pull out more information if we wanted to but those are the most significant fields in the top half of the CLI ouput. As usual we are using results per template to ensure we can have this into a dictionary and we also have an arbitrary template name called nokia_bgp_summ. You’ll notice I have used record against the ASN variable. The record function in TTP is like keeping that value stored for other parts of the template. The way TTP works is it processes one line at a time so it does not retain every piece of information in memory. I want to have the ASN appear in my data for each BGP neighbor. Using record means that key/value is not lost and is available to other templates.
<template name="nokia_bgp_summ" results="per_template">
<group name="{{ BGP_Router_ID }}.BGP_Neighbor.{{ BGP_Neighbor }}" method="table">
BGP Router ID:{{ BGP_Router_ID }} AS:{{ ASN | record("ASN") }} Local AS:{{ Local_ASN }}
BGP Admin State : {{ Admin_State }} BGP Oper State : {{ Oper_State }}
Total BGP Paths : {{ Total_BGP_Paths }}
{{ BGP_Neighbor | IP }}
<group name="_" method="table" set="ASN" macro="process_further">
{{ Remote_ASN }} {{ Pkts_Received }} {{ In_Q }} {{ Peer_Status }} {{ State }}/{{ Recieved }}/{{ Sent }} ({{ Address_Family }})
{{ Remote_ASN | split('*') }} {{ Pkts_Received }} {{ In_Q }} {{ Peer_Status }} {{ State }}
{{ ignore(r"\\s+") }} {{ Pkts_Sent }} {{ Out_Q }}
</group>
</group>
If we look at the second part of the CLI output this is where the actual BGP neighbors are displayed. To pick out that line we are using {{ BGP_Neighbor | IP }}. Using, pipe| IP we will ensure that we match an IP address. The next part is where it gets interesting. To process each BGP neighbor, I create a nested group to ensure the results . Probably the most striking aspect is using the underscore _ in the group name to tell TTP to flatten those results. This will give me a set of data with the BGP Neighbor and all the associated values which have been scraped from the CLI. I have set=”ASN” which will mean that the locally configured ASN will be included in the results for every neighbor.
{{ ignore (r”\\s+”) }} is used to tell TTP to ignore the spaces preceding the Packets Sent field. Otherwise I will need to ensure the correct number of spaces in my template.
{{ Remote_ASN | split(‘*’) }} is used to remove the asterisk from the data. This leaves behind just the ASN number e.g. ASN 2200 instead of ASN 2200*. However, the split function is actually a python inbuilt which creates a list from the result. I don’t want a list in my data since I am working with dictionaries so this is where I need the macro to clean that up.
macro=”process_further” is used to called a macro for this group which allows us to do some further processing with the data.
Lets start to look at macros and how we use them:
<macro>
def process_further(data):
print(data)
return data
</macro>Let’s start with a macro like this which actually just prints the data to the screen, line by line as it processed. By doing this you’re seeing what TTP is seeing and you can also see the results it is finding for this specific group.
It should be obvious that the macro is really just a standard Python function. You can do anything in here which you can do within Python. The key question is can I use the Python Debugger in here so I can look at the data and find out the right syntax to do certain things. The answer is YES and it’s awesome because you can now see exactly what is being captured by TTP.
<macro>
def process_further(data):
print(data)
import ipdb;
ipdb.set_trace()
return data
</macro>If you run the script you will find yourself inside the Python Debugger. From here your only limitation is your knowledge of working with Python datasets like dictionaries and lists.
Let me run through the code within my macro in more detail but this is more a python lesson than a TTP one.
This piece of code is going to get rid of that list left over from the consequence of doing that split function on the Remote ASN variable. At the moment the data is like this [‘2200’, ‘ ‘] so I need to get rid of the 2nd empty field and covert this variable into a pure dictionary format.
My If statement just checks if the Remote ASN field is a list because the ones without the asterisk field will not be a list since there was no split function needed on those.
The next line uses “pop” to remove the 2nd index (note it’s index[1] based on Python numbering). That gets rid of that empty field.
The remaining line converts the value from a list to a dictionary and that completes all the actions needed. There was a reasonable amount of effort to remove that asterisk and get the data the way I wanted it. Perhaps I could have just lived with it.
if type(data.get('Remote_ASN')) == list:
data['Remote_ASN'].pop(1)
data.update(dict(zip(['Remote_ASN'], data['Remote_ASN']))) The next part of the macro is dealing with whether the BGP Neighbor is Established or not. We want to infer this into our data. The first thing we do is create two new dictionary variables bgp_est and bgp_notest to contain the data we want to include. We then look at the State key because if the first character is a digit then we know this means it’s established whereas Active, Connect or Idle states are all words/letters. It’s a very simple way of finding out and hopefully it covers the bases. We then update the data with the appropriate keys (either bgp_est or bgp_notest)
bgp_est={'Peer_Status': 'Established'}
bgp_notest={'Peer_Status': 'Not_Established'}
peer_status=data.get("State")
if type(peer_status) == str:
if peer_status[:1].isdigit() is True:
data.update(bgp_est)
else:
data.update(bgp_notest) Let’s now see the full macro:
<macro>
def process_further(data):
# Remove the list caused by using split function
if type(data.get('Remote_ASN')) == list:
data['Remote_ASN'].pop(1)
data.update(dict(zip(['Remote_ASN'], data['Remote_ASN'])))
# Additional key to confirm if neighbor is Established
bgp_est={'Peer_Status': 'Established'}
bgp_notest={'Peer_Status': 'Not_Established'}
peer_status=data.get("State")
if type(peer_status) == str:
if peer_status[:1].isdigit() is True:
data.update(bgp_est)
else:
data.update(bgp_notest)
return data
</macro>Like every function, you need to “return” the output hence the return data is needed at the end of the function.
In summary, you’ve seen that macros in TTP are very useful for debugging and also for changing the data being captured. In order to use them you need to have some background experience of working with dictionaries, lists, how to manipulate data and the Python Debugger. If you don’t have that level of knowledge then I would recommend starting that learning in Python first before using Macros in TTP. It’s nothing to be afraid of if you are already used to doing this within Python but a potentially steep curve if you’re not used to it. The full script is below as well as a code screenshot.
import pprint
from ttp import ttp
data_to_parse="""
A:R1# show router bgp summary
===============================================================================
BGP Router ID:10.22.0.25 AS:64400 Local AS:64400
===============================================================================
BGP Admin State : Up BGP Oper State : Up
Total Peer Groups : 1 Total Peers : 3
Total BGP Paths : 820 Total Path Memory : 10122
Total IPv4 Remote Rts : 0 Total IPv4 Rem. Active Rts : 0
Total IPv6 Remote Rts : 0 Total IPv6 Rem. Active Rts : 0
Total Supressed Rts : 0 Total Hist. Rts : 0
Total Decay Rts : 0
Total VPN Peer Groups : 5 Total VPN Peers : 5
Total VPN Local Rts : 69
Total VPN-IPv4 Rem. Rts : 303 Total VPN-IPv4 Rem. Act. Rts: 32
Total VPN-IPv6 Rem. Rts : 0 Total VPN-IPv6 Rem. Act. Rts: 0
Total L2-VPN Rem. Rts : 0 Total L2VPN Rem. Act. Rts : 0
Total VPN Supp. Rts : 0 Total VPN Hist. Rts : 0
Total VPN Decay Rts : 0
Total MVPN-IPv4 Rem Rts : 0 Total MVPN-IPv4 Rem Act Rts : 0
Total MDT-SAFI Rem Rts : 0 Total MDT-SAFI Rem Act Rts : 0
Total MSPW Rem Rts : 0 Total MSPW Rem Act Rts : 0
Total FlowIpv4 Rem Rts : 0 Total FlowIpv4 Rem Act Rts : 0
Total IPv4 Backup Rts : 0 Total IPv6 Backup Rts : 0
===============================================================================
BGP Summary
===============================================================================
Neighbor
AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
10.22.1.8
64400 5420893 0 39d23h23m 101/32/69 (VpnIPv4)
426383 0
10.17.2.11
64400 5322002 0 35d20h09m 101/0/69 (VpnIPv4)
425237 0
10.17.4.11
64401 5391552 0 27d20h00m 101/0/69 (VpnIPv4)
425918 0
10.104.34.92
2200* 0 0 14d00h13m Connect
1 0
10.124.34.121
2200* 0 0 00h38m03s Active
1 0
-------------------------------------------------------------------------------
"""
nokia_bgp_summ = """
<template name="nokia_bgp_summ" results="per_template">
<group name="{{ BGP_Router_ID }}.BGP_Neighbor.{{ BGP_Neighbor }}" method="table">
BGP Router ID:{{ BGP_Router_ID }} AS:{{ ASN | record("ASN") }} Local AS:{{ Local_ASN }}
BGP Admin State : {{ Admin_State }} BGP Oper State : {{ Oper_State }}
Total BGP Paths : {{ Total_BGP_Paths }}
{{ BGP_Neighbor | IP }}
<group name="_" method="table" set="ASN" macro="process_further">
{{ Remote_ASN }} {{ Pkts_Received }} {{ In_Q }} {{ Peer_Status }} {{ State }}/{{ Recieved }}/{{ Sent }} ({{ Address_Family }})
{{ Remote_ASN | split('*') }} {{ Pkts_Received }} {{ In_Q }} {{ Peer_Status }} {{ State }}
{{ ignore(r"\\s+") }} {{ Pkts_Sent }} {{ Out_Q }}
</group>
</group>
<macro>
def process_further(data):
# Remove the list caused by using split function
if type(data.get('Remote_ASN')) == list:
data['Remote_ASN'].pop(1)
data.update(dict(zip(['Remote_ASN'], data['Remote_ASN'])))
# Additional key to confirm if neighbor is Established
bgp_est={'Peer_Status': 'Established'}
bgp_notest={'Peer_Status': 'Not_Established'}
peer_status=data.get("State")
if type(peer_status) == str:
if peer_status[:1].isdigit() is True:
data.update(bgp_est)
else:
data.update(bgp_notest)
return data
</macro>
</template>
"""
parser = ttp(template=nokia_bgp_summ)
parser.add_input(data_to_parse, template_name="nokia_bgp_summ")
parser.parse()
res = parser.result(structure="dictionary")
pprint.pprint(res, width=100)
Bite Size Automation 