Stage Two – (cont) Walk-through of the Parser script
Having defined the REGEX we need then the next step is to use this standard code within the script to extract the data.
The first section is like this and is standard:
for line in out.splitlines():
if line:
line = line.strip()
else:
continueWe are using the splitlines method on out which is where our raw CLI output is stored. The Line variable is a string from the for loop which is iterating over every single line in the output. We then do a strip which removes all the whitespaces. I guess the right way to explain this is it’s reading it every line and then removing the whitespaces to tidy things up.
The else statement is a match if line holds no value (i.e. the for loop has finished).
The next bit is where we are going to start using those compiled REGEXs to match CLI output and place it within a dictionary.
m = p1.match(line)
if m:
group = m.groupdict()
interface=group['interface']
result_dict[interface]={}
result_dict[interface]['protocol']={}
continueThe standard seems to be variable “m” which I assume is short for match. If you look at the first line we are pulling in variable p1 and mapping our line into it. Let’s take a look at what is inside the variable line when we have our first match.
(Pdb) line
'GigabitEthernet1'
(Pdb) type(line)
<class 'str'>
(Pdb) m
<_sre.SRE_Match object; span=(0, 16), match='GigabitEthernet1'>That is the REGEX within p1 is being checked against the contents of that line. If there is a match then m will contain a match like above. Otherwise m will be empty. The conditional if will be activated if m has a match or it will move past this section onto the next one.
A quick diversion: The python debugger can be used like this below and also along side this is an example of my debug script which I use to test with. Note that I can now use parse instead of execute so long as I have done the “make json” step to ensure my parser script will be imported by pyATS. You need something like this script to work out how the script is working or not working as the case maybe.
python3 -m pdb <your script which runs the nbar parser>.py --testbed-file <your testbed file>.yaml# Import our libraries
from genie.conf import Genie
# Create a testbed object for the network
testbed = Genie.init("testbed_iosxe.yaml")
for device in testbed.devices:
# Connect to the device
testbed.devices[device].connect()
output = testbed.devices[device].parse("show ip nbar protocol-discovery protocol")
print(output)
We need to comeback to the walk-through of the parser script.
If you remember my capture group for p1 was called <interface>. The groupdict function maps the contents into a dictionary using m as the input. Therefore to get the interface value all I need is group[‘interface’] which will equal GigabitEthernet1 or whatever interface it matches on.
Using pdb we can see the contents of the variable group
(Pdb) group
{'interface': 'GigabitEthernet1'}Now I am also starting to build my schema since I want to have interface as the first key in the dictionary, followed by protocol
result_dict[interface]={}
result_dict[interface]['protocol']={}
At this point result_dict would return the following below:
(Pdb) result_dict
{'GigabitEthernet1': {'protocol': {}}}Basically, we have just added this to our dictionary. Note we have the continue keyword which will mean that we will go back to the top of the for loop and process the next line. It’s simple but effective because I can control what I want to do here.
In some cases, I will not want to that, instead I will want to read in the next set of data because my regex pattern may start to match other lines unintentionally. For example, like in the following REGEX match patterns where I have multiple lines which look for a digit and if that goes out of sequence then we end up with the wrong data being mapped into the wrong fields.
If you’re unsure of how the sequencing of reading in the lines and REGEX matches is going then I would suggest switching on a pdb debug when you try to parse the command so you can see how each line is being read in. Depending on what you want to happen then you may need continue or not as each complied REGEX string is checked .
Having grabbed the interface and pulled that into our dictionary then we move down to the next block.
m = p2.match(line)
if m:
group = m.groupdict()
protocol=group['protocol']
result_dict[interface]['protocol'][protocol]={}
result_dict[interface]['protocol'][protocol].update({'IN Packet Count': int(group['In_Packet_Count'])})
result_dict[interface]['protocol'][protocol].update({'OUT Packet Count': int(group['Out_Packet_Count'])})
continue
Basically, more of the same going on. This time we are looking for protocol. Look at how the capture group of protocol is being mapped into a dictionary. Due to protocol being a key in the dictionary then we need create an empty one {} or we will receive a KeyError.
The update method is used to insert those In_Packet_Count and Out_Packet_Count values into the dictionary. What you end up with is the protocol in this case “unknown” as a legitimate entry and the values associated with that. I also want these values to always be digits. If I don’t have int then I could have the wrong data captured in the worst case and this will all play a part in the schema later where I will tell the schema to enforce an integer for that field. Using the default type of string str is too loose.
(Pdb) result_dict
{'GigabitEthernet1': {'protocol': {'unknown': {'IN Packet Count': 406, 'OUT Packet Count': 8}}}}Note I have the continue keyword which will make this loop back through. The remaining lines need to be captured sequentially hence they will not have the continue keyword in the code block.
m = p3.match(line)
if m:
group = m.groupdict()
result_dict[interface]['protocol'][protocol].update({'IN Byte Count': int(group['In_Byte_Count'])})
result_dict[interface]['protocol'][protocol].update({'OUT Byte Count': int(group['Out_Byte_Count'])})
m = p4.match(line)
if m:
group = m.groupdict()
result_dict[interface]['protocol'][protocol].update({'IN 5min Bit Rate (bps)': int(group['In_Bitrate'])})
result_dict[interface]['protocol'][protocol].update({'OUT 5min Bit Rate (bps)': int(group['Out_Bitrate'])})
m = p5.match(line)
if m:
group = m.groupdict()
result_dict[interface]['protocol'][protocol].update({'IN 5min Max Bit Rate (bps)': int(group['In_Bitrate_Max'])})
result_dict[interface]['protocol'][protocol].update({'OUT 5min Max Bit Rate (bps)': int(group['Out_Bitrate_Max'])})
If the sequencing and ordering is confusing then I would suggest using pdb as that will help you understand how each line is being processed. Also I highly recommend working in pdb to populate the right keys that you need. That way every step can be checked and you can see what data is being captured line by line.
Once the for loop is completed then the only thing remaining is to return the end dictionary. Except, I have an issue that the schema seems to requires a starting key so I need to make sure that is added. I added it at the end but you could do it at the start and I will probably do that in future. Results matter the most though and not so much about how you get the result.
result_dict = {'interface': result_dict} This just means the dictionary will have ‘interface’ as the starting key (like below)
{
'interface': {
'GigabitEthernet1': {The final step is to return the complete dictionary but at this point the schema will be checked which we haven’t declared yet so say stay in pdb to make sure your dictionary is correct.
return result_dictFull contents of the dictionary are listed below:
{
'interface': {
'GigabitEthernet1': {
'protocol': {
'unknown': {
'IN Packet Count': 110,
'OUT Packet Count': 8,
'IN Byte Count': 3000,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 3000,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 3000,
'OUT 5min Max Bit Rate (bps)': 0
},
'dns': {
'IN Packet Count': 43,
'OUT Packet Count': 0,
'IN Byte Count': 3000,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 3000,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 3000,
'OUT 5min Max Bit Rate (bps)': 0
},
'ssh': {
'IN Packet Count': 16,
'OUT Packet Count': 11,
'IN Byte Count': 0,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 0,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 0,
'OUT 5min Max Bit Rate (bps)': 0
},
'eigrp': {
'IN Packet Count': 28,
'OUT Packet Count': 28,
'IN Byte Count': 0,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 0,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 0,
'OUT 5min Max Bit Rate (bps)': 0
},
'ldp': {
'IN Packet Count': 23,
'OUT Packet Count': 23,
'IN Byte Count': 0,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 0,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 0,
'OUT 5min Max Bit Rate (bps)': 0
},
'ospf': {
'IN Packet Count': 14,
'OUT Packet Count': 14,
'IN Byte Count': 0,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 0,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 0,
'OUT 5min Max Bit Rate (bps)': 0
},
'Total': {
'IN Packet Count': 234,
'OUT Packet Count': 84,
'IN Byte Count': 6000,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 6000,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 6000,
'OUT 5min Max Bit Rate (bps)': 0
}
}
},
'GigabitEthernet2': {
'protocol': {
'unknown': {
'IN Packet Count': 108,
'OUT Packet Count': 7,
'IN Byte Count': 3000,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 3000,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 3000,
'OUT 5min Max Bit Rate (bps)': 0
},
'dns': {
'IN Packet Count': 43,
'OUT Packet Count': 0,
'IN Byte Count': 3000,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 3000,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 3000,
'OUT 5min Max Bit Rate (bps)': 0
},
'eigrp': {
'IN Packet Count': 28,
'OUT Packet Count': 28,
'IN Byte Count': 0,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 0,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 0,
'OUT 5min Max Bit Rate (bps)': 0
},
'ldp': {
'IN Packet Count': 23,
'OUT Packet Count': 23,
'IN Byte Count': 0,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 0,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 0,
'OUT 5min Max Bit Rate (bps)': 0
},
'ospf': {
'IN Packet Count': 14,
'OUT Packet Count': 14,
'IN Byte Count': 0,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 0,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 0,
'OUT 5min Max Bit Rate (bps)': 0
},
'Total': {
'IN Packet Count': 216,
'OUT Packet Count': 72,
'IN Byte Count': 6000,
'OUT Byte Count': 0,
'IN 5min Bit Rate (bps)': 6000,
'OUT 5min Bit Rate (bps)': 0,
'IN 5min Max Bit Rate (bps)': 6000,
'OUT 5min Max Bit Rate (bps)': 0
}
}
}
}In the next part we look at the schema and how to define that to finish the parser script before we move onto the Testing and finally submission.
Bite Size Automation 