sâmbătă, 3 martie 2018

LnetD v0.1 - network inventory based on ISIS

Hi again, Finally manage to find time and put all the code in an web application. There are two project that i want to thank for offering all the tools required to build this. https://github.com/afourmy/eNMS/ https://github.com/afourmy/flask-gentelella Now back to the topology map and inventory based on ISIS.


  •  Login screen


  • Dashboard


  • Inventory


  • D3js Topology and Map

As always code on github: https://github.com/cpmarvin/lnetd

miercuri, 7 februarie 2018

Dijkstra - Networkx, d3.js weathermap based on ISIS topology

Based on previous posts the code is extended with couple of new features.


First the parallel link algorithm is enhanced using http://webiks.com/d3-js-force-layout-straight-parallel-links/ calculation, please see the link for more details. I have parallel links support and 'half links' to support displaying in/out link utilisation.

Same as before I'm using py ez to parse the isis database from a juniper device. This creates a json with all the links in the topology. The json also contains information about current 5 minutes utilisation and snmp ifindex for each link.

The weathermap function is based on snmp data collected by telegraf and stored in influxdb. The py ez script will populate the json with information from influxdb. Basically for each link we have the IP address, based on this we query influxdb to get the snmp ifindex. Then I do a second query to get the last 5 minutes util and the link capacity.

The resulting json has the following information:

Source, target, local ip, remote ip, util, capacity, local_ip_remote_ip, ifindex.
[{"r_ip": "x.x.x.x", "l_int": 49, "capacity": 10000, "target": "PE1-CPT", "metric": "145", "l_ip_r_ip": ["x.x.x.x", "y.y.y.y"], "util": 1491081683, "source": "PE1-GSW", "r_int": 0, "l_ip": "y.y.y.y"}

After the d3js topology is displayed, clicking on an link will create a plotly graph with last 24h utilisation. This is done via ajax query to the api. The request argument is host + ifindex. The api will query influxdb and get the snmp information.

The last piece of the puzzle is the dijkstra algorithm to find shortest path between nodes.

This is achieved using networkx python library and works on top of flask. The js code sends a request with all links array, source and destination. The api will return all links id for all ecmp paths between nodes. Based on link Id d3js will colour the links with dotted lines.

Demo time: 

Topology:



Plotly graph on link click:




Dijkstra's spf between two nodes:






Code on https://github.com/cpmarvin/lnetd

Leave a message if you have a hard time replicating this. This project started as a learning exercise and the code is not very clean.

luni, 9 octombrie 2017

Segment Routing TE

Recently the company that i work(changed job again ) for has roll out segment routing. This is still limited to 7 countries.

Next step for us is to look into PCEP-SR/BGP-LU/and SR-TE ( this post ). We don't have loots of rsvp-te tunnels, ~90 or so, 70% used for igp-shortcuts uneaqul load balancing and rest for services ( some customers want to go via a specific subsea cable , for some reason :) ).

We don't have a controller , and probably never will... and it's going to be a while while the segment features are present in all 3 vendors( that we use).

Anywho this is about SR-TE ... what it supposed to do and current implementation on JNP.

SR-TE is defined in:
https://tools.ietf.org/html/draft-ietf-idr-segment-routing-te-policy-00 ;

Mapping traffic over this is defined :
https://tools.ietf.org/html/draft-filsfils-spring-segment-routing-policy-01;

JNP implementation of SR-TE is not there , so end of blogpost.... What they do have at this stage , is a static SR feature.
This means instead of learning the SR policy via BGP ( and help me crete a proper post ) the current implementation is static for now...

In terms of mapping traffic onto the SR ( draft-filsfils-spring-segment-routing-policy-01) this is fine , at least the use case below.


Topology



vmx8->10.11.11.11/32 follows the spf ; xrv2->xrv1
Label traffic is SR-based ; top-most label 800001

root@vmx8> show route 10.11.11.11

inet.0: 22 destinations, 23 routes (22 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

10.11.11.11/32     *[BGP/170] 00:00:00, MED 0, localpref 100, from 10.1.1.1
                      AS path: I, validation-state: unverified
                    > to 10.2.8.2 via ge-0/0/0.0, Push 800001

inet.3: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.1.1.1/32        *[L-ISIS/14] 00:03:29, metric 20
                    > to 10.2.8.2 via ge-0/0/0.0, Push 800001

root@vmx8> traceroute 10.11.11.11
traceroute to 10.11.11.11 (10.11.11.11), 30 hops max, 52 byte packets
 1  10.2.8.2 (10.2.8.2)  116.892 ms  2.880 ms  2.667 ms
     MPLS Label=800001 CoS=0 TTL=1 S=1
 2  10.1.2.1 (10.1.2.1)  4.990 ms *  4.668 ms

The goal is to create an SR-TE policy so that traffic flows vmx8->xrv2-xrv5>xrv3->xrv1. Ideally this would have been created by BGP SR-TE,
but that is not working in the current code ... so let's do it static.

set protocols source-packet-routing segment-list sr-te_to_xrv1 xrv5 label 800005
set protocols source-packet-routing segment-list sr-te_to_xrv1 xrv3 label 800003
set protocols source-packet-routing segment-list sr-te_to_xrv1 xrv1 label 800001

set protocols source-packet-routing source-routing-path to_xrv1 to 10.1.1.1
set protocols source-packet-routing source-routing-path to_xrv1 color 0
set protocols source-packet-routing source-routing-path to_xrv1 primary sr-te_to_xrv1

What this does is , create the SR-TE with a label stack of 3, the next hop label is not needed.

root@vmx8> show spring-traffic-engineering lsp         
To              State     LSPname
10.1.1.1-0<c>   Up        to_xrv1

the -0<c> is really important. This is the color of the tunnel... this as this as SLA agreement. In fact let create another one with different color. The 10.1.1.1 is the end-point , matching the next-hop of the ipv4 unicast route.

This time is xrv2-xrv5-xrv2-xrv1

set protocols source-packet-routing segment-list sr-te_to_xrv1#2 xrv5 label 800005
set protocols source-packet-routing segment-list sr-te_to_xrv1#2 xrv2 label 800002
set protocols source-packet-routing segment-list sr-te_to_xrv1#2 xrv1 label 800001

set protocols source-packet-routing source-routing-path to_xrv1#2 to 10.1.1.1
set protocols source-packet-routing source-routing-path to_xrv1#2 color 10
set protocols source-packet-routing source-routing-path to_xrv1#2 primary sr-te_to_xrv1#2

As expected :

root@vmx8> show spring-traffic-engineering lsp
To              State     LSPname
10.1.1.1-0<c>   Up        to_xrv1
10.1.1.1-10<c>  Up        to_xrv1#2

root@vmx8> show route table inetcolor.0

inetcolor.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.1.1.1-0<c>/64             
                   *[SPRING-TE/8] 00:08:58, metric 1, metric2 20
                    > to 10.2.8.2 via ge-0/0/0.0, Push 800001, Push 800003, Push 800005(top)
10.1.1.1-10<c>/64             
                   *[SPRING-TE/8] 00:08:58, metric 1, metric2 20
                    > to 10.2.8.2 via ge-0/0/0.0, Push 800001, Push 800002, Push 800005(top)


Now , one of the method to inject traffic into the SR-TE is to color a destination prefix. So let's do just that. I'm using yabgp to insert
the bgp route.

the json it's below:

{
    "attr": {
    "1": 0,
    "2": [[2, []]],
    "3": "10.1.1.1",
    "5": 200,
        "8": ["NO_ADVERTISE"],
        "16": [[779, 0]]
    },
    "nlri":["10.11.11.11/32"]
}

Test 1:
Advertise 10.11.11.11/32 with next-hop 10.1.1.1 and with extended community 779 (BGP_EXT_COM_COLOR = 0x030b) with value 0.

root@vmx8> show route 10.11.11.11

inet.0: 22 destinations, 24 routes (22 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

10.11.11.11/32     *[BGP/15] 00:00:06, localpref 200, from 10.97.97.2
                      AS path: I, validation-state: unverified
                    > to 10.2.8.2 via ge-0/0/0.0, Push 800005
                    [BGP/170] 00:01:56, MED 0, localpref 100, from 10.1.1.1
                      AS path: I, validation-state: unverified
                    > to 10.2.8.2 via ge-0/0/0.0, Push 800001

root@vmx8> ...32 extensive | match "color|Protocol next hop"               
                Protocol next hop: 10.1.1.1-0<c>
                Communities: no-advertise color:0:0

As expected route is received with corect color , and the next hop is the SR-TE policy.

Traceroute :

root@vmx8> traceroute 10.11.11.11
traceroute to 10.11.11.11 (10.11.11.11), 30 hops max, 52 byte packets
 1  10.2.8.2 (10.2.8.2)  99.071 ms  215.040 ms  6.419 ms
     MPLS Label=800005 CoS=0 TTL=1 S=0
     MPLS Label=800003 CoS=0 TTL=1 S=0
     MPLS Label=800001 CoS=0 TTL=1 S=1
 2  10.2.5.5 (10.2.5.5)  5.144 ms  4.763 ms  4.597 ms
     MPLS Label=800003 CoS=0 TTL=1 S=0
     MPLS Label=800001 CoS=0 TTL=2 S=1
 3  10.3.5.3 (10.3.5.3)  4.751 ms  4.642 ms  4.752 ms
     MPLS Label=800001 CoS=0 TTL=1 S=1
 4  10.1.3.1 (10.1.3.1)  4.762 ms *  44.309 ms

Test 2:
Advertise 10.11.11.11/32 with next-hop 10.1.1.1 and with extended community 779 (BGP_EXT_COM_COLOR = 0x030b) with value 10.


{
    "attr": {
    "1": 0,
    "2": [[2, []]],
    "3": "10.1.1.1",
    "5": 200,
        "8": ["NO_ADVERTISE"],
        "16": [[779, 10]]
    },
    "nlri":["10.11.11.11/32"]
}

root@vmx8> ...extensive | match "Communities|Protocol next hop"             
                Protocol next hop: 10.1.1.1-10<c>
                Communities: no-advertise color:0:10

root@vmx8> traceroute 10.11.11.11
traceroute to 10.11.11.11 (10.11.11.11), 30 hops max, 52 byte packets
 1  10.2.8.2 (10.2.8.2)  6.217 ms  5.156 ms  4.686 ms
     MPLS Label=800005 CoS=0 TTL=1 S=0
     MPLS Label=800002 CoS=0 TTL=1 S=0
     MPLS Label=800001 CoS=0 TTL=1 S=1
 2  10.2.5.5 (10.2.5.5)  4.722 ms  4.470 ms  4.569 ms
     MPLS Label=800002 CoS=0 TTL=1 S=0
     MPLS Label=800001 CoS=0 TTL=2 S=1
 3  10.2.5.2 (10.2.5.2)  4.681 ms  4.500 ms  4.715 ms
     MPLS Label=800001 CoS=0 TTL=1 S=1
 4  10.1.2.1 (10.1.2.1)  4.806 ms *  9.379 ms

duminică, 2 aprilie 2017

d3js network weathermap

I've always wanted something like this.

We use grafana and telegram for snmp statistics and the only "legacy" part was Cacti and that because of the weathermap plugin. Don't get me wrong , i really like that plugin it's just that well i had some free time in the recent weeks so here it goes.

This is based on this. : The javascript does couple of api calls to a web server that in turn queries influxdb for snmp counters for each router/interface.Really simple web-api server build with flask.

I will put the scripts on github soon (https://github.com/cpmarvin/flask_api_influxdb). So high level diagram :


Features:
  •  draw links btw nodes and color them according to %util, static refBW ( ...for now ) 
  • show util graph for last 24 hours on link click 
  • based on past 1 to 24 hours recolor links based on util at that point 
  • refresh in 5 minutes 

 Initial view: 


Look in the past: 


Show graph for last 24h: 


Code for the webpage on github : here.

PS: 


I'm not a programmer so this can be written better. 
Next step is integrate this in grafana with a custom plugin. Help needed!!

joi, 16 martie 2017

Network Diagram D3js based on ISIS Database


Let's build one based on ISIS database.

https://github.com/cpmarvin/isis_db_parser

First you need something to get the isis database from a device( ie: junos) so lets use py-ez

clone rep from github:
root@lab:/usr/local/src# git clone https://github.com/cpmarvin/isis_db_parser.git
cd /usr/local/src/
python isis_get.py

This will log to the router , get the db and provide a dic for links with some information about metric and local/remote ip and local/remote interface index ( Junos specific). In my lab this is what i get ( this ( use print df1 )

     source        target metric          l_ip l_int          r_ip r_int
0  vrr1.pgt      bdr1.lab    500  81.91.194.52   335  81.91.194.53   345
1  vrr1.pgt      bdr1.lab    500  81.91.194.50   333  81.91.194.51   437
4  bdr1.lab  AGG1.INX.LAB     10  192.168.1.29   393  192.168.1.30   333

and the dict:
[{'r_ip': '81.91.194.53', 'l_int': '335', 'target': 'bdr1.lab', 'metric': '500', 'source': 'vrr1.pgt', 'r_int': '345', 'l_ip': '81.91.194.52'}, {'r_ip': '81.91.194.51', 'l_int': '333', 'target': 'bdr1.lab', 'metric': '500', 'source': 'vrr1.pgt', 'r_int': '437', 'l_ip': '81.91.194.50'}, {'r_ip': '192.168.1.30', 'l_int': '393', 'target': 'AGG1.INX.LAB', 'metric': '10', 'source': 'bdr1.lab', 'r_int': '333', 'l_ip': '192.168.1.29'}]


Once you have this you can use the d3js script ( https://github.com/cpmarvin/d3js )  , and build the diagram :

Features:
- zoom
- neighbour highlight
- filtering based on regexp node name
- show all link label
Small network diagram :
Neighbours highlight:



miercuri, 24 august 2016

Open Source - NetFlow Analysis Tool






So ... how do you get network visibility using opensource. Easy ... well sort of





Tools and versions used :

- pmacctd 1.6.1-git (20160823-00)
- kafka 0.10.0.1
- influxdb 0.13
- grafana 3.1
- kafka-influxdb ( git )
- custom plugin for pmacct

Fastnemon is work in progress.

The results :

- Overview : Network Total/Transit/Peers





- Transit breakdown by router/interface/transit_peer_as/source_as


 - Total by router/interface/peer_as




- Total by dst_net/interface/peer_as

















This are the main ones i use everyday, the pmacct aggregation below

-src_as,dst_as,peer_src_as,peer_dst_as,peer_src_ip,as_path,dst_net,src_net,dst_mask,src_mask,in_iface,std_comm,med,proto,src_as_path

And a nice overview dashboard for  managment to look at