openwisp / netdiff Goto Github PK

Actually the diff function compares every link in the old and new topology to search the unmodified ones.
If we have two link, that are simply inverted like: ('192.168.0.1', '192.168.1.1') and ('192.168.1.1', 192.168.0.1'), the program will mark them as different and will break up the diff of the graph.

The fix is in the pull request for batman-adv

Mock HTTP in tests

Batman-adv also returns a format that mocks the OLSR txtinfo.

They did this for compatibility reasons in the past.

We are currently using it in ninux, so adding support in BatmanParser would speed up the testing process in the ninux community.

Improve the diff method so that it can do two operations:

• calculate differences but ignoring changes changes to metrics/costs
• calculate differences counting metric/cost changes as differences

This wireguard-tools script helps explaining the output of the dump command and how to parse it:
https://github.com/WireGuard/wireguard-tools/blob/master/contrib/json/wg-json

I think we should implement something similar in python and generate the network graph.

We can pass an argument which lets us choose whether the primary ID of the nodes should be the IP address or the public key, for consistency with the other parsers I would use the IP as default.

The first argument (data) of each parser can represent many things:

• topology data
• a file path
• a URL

Unfortunately this can cause different type of issues, including security issues in which an attacker can access a file on a remote machine that uses netdiff to accept network topology data through a POST request.

It would be better to implement stricter initialization arguments, eg: data, path, url.

Shall we make the result of the difffunction consistent with the NetJSON NetworkGraph object?

Now it returns something like:

{
    "removed": [
         [
            "172.16.40.4", 
            "172.16.40.3"
        ]
    ], 
    "added": [
        [
            "172.16.40.1", 
            "172.16.40.2"
        ]
    ]
}

To make it consistent we could change it into:

{
    "removed": {
        "type": "NetworkGraph",
        "protocol": "OLSR",
        "version": "0.6.6",
        "metric": "ETX",
        "nodes": [
            {
                "id": "10.150.0.3"
            },
            {
                "id": "10.150.0.2"
            }
        ],
        "links": [
            {
                "source": "10.150.0.3",
                "target": "10.150.0.2",
                "weight": 1.0
            }
        ]
    }
    "added": {
        "type": "NetworkGraph",
        "protocol": "OLSR",
        "version": "0.6.6",
        "metric": "ETX",
        "nodes": [
            {
                "id": "172.16.40.1"
            },
            {
                "id": "172.16.40.2"
            }
        ],
        "links": [
            {
                "source":  "172.16.40.1", 
                "target": "172.16.40.2",
                "weight": 1.0
            }
        ]
    }
}

In case there are no changes we should prefer null to an empty NetworkGraph object for brevity.
The result would look like the following:

{
    "removed": null,
    "added": null
}

We need to detect changes in label, properties and local_addresses. At the moment if the information of a node or a link changes netdiff won't recognize it as a change but this baffles even myself.

Update: this shows up in openwisp-network-topology.
When the data of the nodes or links in the network topology change (eg: addresses, names of nodes, other info), the changes are not reflected because netdiff doesn't handle this case, we should fix it.

Add a ZeroTierParser that can parse the information of ZeroTier peers and generate a network graph.

• Utilize zerotier-cli peers -j command to obtain the ZeroTier peers information as shown below:

[
{
 "address": "62f865ae71",
 "isBonded": false,
 "latency": 382,
 "paths": [
  {
   "active": true,
   "address": "50.7.252.138/9993",
   "expired": false,
   "lastReceive": 1645034513389,
   "lastSend": 1645034513008,
   "preferred": true,
   "trustedPathId": 0
  }
 ],
 "role": "PLANET",
 "version": "-1.-1.-1",
 "versionMajor": -1,
 "versionMinor": -1,
 "versionRev": -1
},
{
 "address": "778cde7190",
 "isBonded": false,
 "latency": 344,
 "paths": [
  {
   "active": true,
   "address": "103.195.103.66/9993",
   "expired": false,
   "lastReceive": 1645034513362,
   "lastSend": 1645034513008,
   "preferred": true,
   "trustedPathId": 0
  }
 ],
 "role": "PLANET",
 "version": "-1.-1.-1",
 "versionMajor": -1,
 "versionMinor": -1,
 "versionRev": -1
},
{
 "address": "cafe04eba9",
 "isBonded": false,
 "latency": 253,
 "paths": [
  {
   "active": true,
   "address": "84.17.53.155/9993",
   "expired": false,
   "lastReceive": 1645034508255,
   "lastSend": 1645034513008,
   "preferred": true,
   "trustedPathId": 0
  }
 ],
 "role": "PLANET",
 "version": "-1.-1.-1",
 "versionMajor": -1,
 "versionMinor": -1,
 "versionRev": -1
},
{
 "address": "cafe9efeb9",
 "isBonded": false,
 "latency": 247,
 "paths": [
  {
   "active": true,
   "address": "104.194.8.134/9993",
   "expired": false,
   "lastReceive": 1645034513259,
   "lastSend": 1645034513008,
   "preferred": true,
   "trustedPathId": 0
  }
 ],
 "role": "PLANET",
 "version": "-1.-1.-1",
 "versionMajor": -1,
 "versionMinor": -1,
 "versionRev": -1
}
]

• Extract the role and latency fields to set the cost of links.

The current NetJSON output is outdated.

The weight attribute was renamed to cost.

This aims to extract details from the output of different formats of OpenVPN. An example output can be shown as follows:

OpenVPN​ CLIENT LIST
Updated​ , ​ Mon​ ​ Mar​ ​ 27​ ​ 18​ : ​ 47​ : ​ 04​ ​ 2017
Common​ ​ Name​ , ​ Real​ ​ Address​ , ​ Bytes​ ​ Received​ , ​ Bytes​ ​ Sent​ , ​ Connected​ ​ Since
claud43nodo1​ , ​ 80.181​ . ​ 191.1​ : ​ 50268​ , ​ 53875636​ , ​ 192011412​ , ​ Sun​ ​ Mar​ ​ 26​ ​ 09​ : ​ 06​ : ​ 20​ ​ 2017
LeMonacelle​ , ​ 2.226​ . ​ 154.66​ : ​ 44846​ , ​ 5164751​ , ​ 19264301​ , ​ Mon​ ​ Mar​ ​ 27​ ​ 16​ : ​ 10​ : ​ 59​ ​ 2017
Syskrack​ , ​ 79.26​ . ​ 49.2​ : ​ 60616​ , ​ 56928467​ , ​ 189355476​ , ​ Sun​ ​ Mar​ ​ 26​ ​ 09​ : ​ 06​ : ​ 19​ ​ 2017
Kali2​ , ​ 2.226​ . ​ 154.66​ : ​ 55438​ , ​ 52681920​ , ​ 193489019​ , ​ Sun​ ​ Mar​ ​ 26​ ​ 09​ : ​ 06​ : ​ 20​ ​ 2017
mirko​ , ​ 79.36​ . ​ 196.24​ : ​ 65039​ , ​ 1057697​ , ​ 245684199​ , ​ Sun​ ​ Mar​ ​ 26​ ​ 09​ : ​ 06​ : ​ 20​ ​ 2017
buda​ , ​ 79.12​ . ​ 108.6​ : ​ 62026​ , ​ 1061815​ , ​ 245676448​ , ​ Sun​ ​ Mar​ ​ 26​ ​ 09​ : ​ 06​ : ​ 23​ ​ 2017
pomezia​ , ​ 95.251​ . ​ 243.132​ : ​ 50275​ , ​ 1058494​ , ​ 245684089​ , ​ Sun​ ​ Mar​ ​ 26​ ​ 09​ : ​ 06​ : ​ 21​ ​ 2017
ROUTING TABLE
Virtual​ ​ Address​ , ​ Common​ ​ Name​ , ​ Real​ ​ Address​ , ​ Last​ ​ Ref
aa​ : ​ f7​ : ​ ef​ : ​ 8f​ : ​ 55​ : ​ 52​ , ​ Syskrack​ , ​ 79.26​ . ​ 49.2​ : ​ 60616​ , ​ Mon​ ​ Mar​ ​ 27​ ​ 18​ : ​ 45​ : ​ 29​ ​ 2017
8a​ : ​ 0b​ : ​ ac​ : ​ 35​ : ​ 42​ : ​ c6​ , ​ LeMonacelle​ , ​ 2.226​ . ​ 154.66​ : ​ 44846​ , ​ Mon​ ​ Mar​ ​ 27​ ​ 16​ : ​ 11​ : ​ 01​ ​ 2017
a6​ : ​ 26​ : ​ 32​ : ​ 97​ : ​ 8b​ : ​ 6c​ , ​ claud43nodo1​ , ​ 80.181​ . ​ 191.1​ : ​ 50268​ , ​ Sun​ ​ Mar​ ​ 26​ ​ 23​ : ​ 14​ : ​ 10​ ​ 2017
c4​ : ​ 6e​ : ​ 1f​ : ​ ff​ : ​ 02​ : ​ 23​ , ​ Kali2​ , ​ 2.226​ . ​ 154.66​ : ​ 55438​ , ​ Mon​ ​ Mar​ ​ 27​ ​ 18​ : ​ 45​ : ​ 29​ ​ 2017
GLOBAL STATS
Max​ bcast​ / ​ mcast queue length​ , ​ 11
END

Upgrade networkx dependency. This is probably not easy but should be doable.

In Ninux there are several network "islands" interconnected via BGP. Many islands announce in their OLSR networks as HNAs the subnets learned from BGP.

This has some consequences:

1. same HNA subnets might appear as being announced from two very distant nodes
2. the devices in an island might appear as belonging to the HNA announced by another island, which, coupled with the situation explained in issue #3 might lead to compute, erroneously, the presence of links between distant islands

An ugly way to mitigate 2. might be not considering for IP to HNA belonging the nodes that are announcing a number of HNAs that exceeds a given threshold.

Please, can you write a a netdiff parser for fetching routing info from bird daemon version 2?

There's one problem with displaying changes in weight. When the weight of a link changes (which we must suppose happens continously) it doesn't make much sense to put this change in the "added" section, nor in the "removed", we have to have another section called "changed".

Suppose the only difference is that one link has changed its weight, this would be the result:

{
    "removed": null,
    "added": null,
    "changed": {
        "type": "NetworkGraph",
        "protocol": "OLSR",
        "version": "0.6.6",
        "metric": "ETX",
        "nodes": [
            {
                "id": "172.16.40.1"
            },
            {
                "id": "172.16.40.2"
            }
        ],
        "links": [
            {
                "source":  "172.16.40.1", 
                "target": "172.16.40.2",
                "weight": 3.2
            }
        ]
    }
}

This also reimplements #7

A node might be referenced one time with an id and another time with one of its aliases.

In such a case netdiff would consider the topology changed.

The local_addresses attribute was recently introduced in NetJSON NetworkGraph (see netjson/netjson#15), we can use this feature to work it consistently on all the parsers of the protocols that use aliases.

test_weight fails randomly, couldn't understand the cause yet.

It seems that in some rare cases the allowed_ips may be None, but the library doesn't handle this right now:

AttributeError: 'NoneType' object has no attribute 'split'
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "django/views/generic/base.py", line 84, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "openwisp_network_topology/api/views.py", line 137, in post
    topology.receive(request.data)
  File "openwisp_network_topology/base/topology.py", line 364, in receive
    self.update(data)
  File "openwisp_network_topology/base/topology.py", line 295, in update
    diff = self.diff(data)
  File "openwisp_network_topology/base/topology.py", line 138, in diff
    latest = self.get_topology_data(data)
  File "openwisp_network_topology/base/topology.py", line 119, in get_topology_data
    latest = self.parser_class(data=data, url=self.url, timeout=TIMEOUT)
  File "netdiff/parsers/base.py", line 72, in __init__
    self.original_data = self.to_python(data)
  File "netdiff/parsers/wireguard.py", line 20, in to_python
    return self._wg_dump_to_python(e.data)
  File "netdiff/parsers/wireguard.py", line 28, in _wg_dump_to_python
    parsed_lines = self._parse_lines(lines)
  File "netdiff/parsers/wireguard.py", line 74, in _parse_lines
    allowed_ips=allowed_ips.split(','),

Add CI build on github actions.

It would be great if the OpenVPN could add human readable labels for nodes (taken from hostname).

But first we need to address #35.

When parsing a NetJSON output with NetJsonParser, the output of the json method does not include the label, but it would be better to include it.

In the ninux Rome network, that uses OLSR, there are some wireless links built in the following way:

• on one side of the wireless link a device that does NOT talk OLSR, with IP address IP_X
• on the other side of the wireless link a device talking OLSR that announces as HNA a subnet that includes IP_X

The consequence is that IP_X does not appear explicitly in the OLSR topology returned by the txtinfo and jsoninfo olsrd plug-ins.

This means that, to have a complete topology, either:

• the application using netdiff should verify the matching between the IP addresses of the known devices and HNAs
• the application using netdiff should pass to netdiff the list of know devices, so that netdiff can do the matching between IP addresses and HNAs
• netdiff should return explicitly the list of all the hosts that could belong to the HNA. For example 254 hosts for a /24 HNA.

Each routing protocol has a JSON schema for its topology.

We are trying to define a more generic JSON here: https://github.com/interop-dev/json-for-networks/blob/master/examples/network-graph.json

It would be great if each netdiff parser could parse both structures.

Interested in completing the task for GCI-2018

The _netjson_networkgraph function is used to generate a NetJSON NetworkGraph.

https://github.com/openwisp/netdiff/blob/master/netdiff/utils.py#L127

Right now it is more like an internal function. It would be nice providing it to the module users so that they don't have to reinvent the wheel so much.

For example, in openwisp-network-topology, the json method of the Topology model is already reinventing it:

https://github.com/openwisp/openwisp-network-topology/blob/fac83331cee7194aef4ba92f0a5fa067d3597aac/openwisp_network_topology/base/topology.py#L139-L169

netdiff can return NetJSON as output, this is quite handy to work in tandem with https://github.com/interop-dev/netjsongraph.js

The limitation is that netdiff is not able to include in the NetJSON output information that is specific to each routing protocol, although NetJSON NetworkGraph allows custom properties for nodes and links and these properties are visualized by netjsongraph.js.

Overcoming this limitation is useful in order to make it possible to start using netjsongraphjs and netdiff together to display detailed topology information.

Allow supporting the latest networkx version.

Maybe it will be possible add when we use OLSRD to create file for netjsongraph use file latlon.js which contain xy coordinates node for exmaple:
....
Node('195.165.33.135',52.026421,15.629709,0,'0.0.0.0','NODE-1');
PLink('195.165.33.254','195.165.33.1',1.000,1.000,0.100,52.015381,15.597080,0,52.015518,15.597071,0);
....

and add this info to "properties": and when crate graph and exit xy coordinates when we clik on graph green line in display properties this link add information distance: between nodes . We can calculate use xy coordinates. It is usefull information about link. In our site weh hav node where links between nodes are form 1.5 km do 9 km

we can add one more option in script where user will define path to latlon.js file

olsrlatlon('/var/run/latlon.js')

or other

Hi,

I have try use netdiff to create netjsonGraph from my local network base on OLSRD 1 (version olsrd v0.9)
I have create simple file with following:

import os
import six
import networkx

from netdiff import OlsrParser
from netdiff import diff

olsr = OlsrParser('telnet://127.0.0.1:9090')
......

when I have run script I have following errors:
Traceback (most recent call last):
File "pmk.py", line 9, in
olsr = OlsrParser('telnet://127.0.0.1:9090')
File "/data/src/netdiff-master/netdiff/parsers/base.py", line 46, in init
self.original_data = self.to_python(data)
File "/data/src/netdiff-master/netdiff/parsers/olsr.py", line 20, in to_python
return self._txtinfo_to_jsoninfo(e.data)
File "/data/src/netdiff-master/netdiff/parsers/olsr.py", line 89, in _txtinfo_to_jsoninfo
raise ParserError('Unrecognized format')

But when I switch to use textinfo plugin on port 2006 I have result with create output ready to use with netjson.js

How to create script to get date via jsoninfo plugin istead textinfo plugin ????

reserved for GCI

Sometimes OpenVPN will report nodes which are named UNDEF (which stands for undefined).
We should ignore these nodes. Don't forget to write a test for this change :-)

Why this UNDEF shows up in the first place?

Mock Telnet in tests

If the topology file or URL is not reachable we should raise an adhoc subclass of NetdiffException.

The reason for this is to help client libraries understanding easily when the topology file is not reachable, otherwise they would have to catch the different type of exceptions (HTTP, telnet, file).

If two graphs having the same nodes but different links are compared, the diff is empty while a difference in the added link section is expected.

The recent addition for bmx6 parsing is using the old output, so it doesn't work with newer versions of bmx6.

Here is the output from b6m-jsonp: http://pastebin.com/KmfGnTs7

Maybe it would be better to use the common name as the ID for the openvpn parser, because the ID based on real address + port can change often if multiple nodes are on the same public IP address, generating duplicated nodes in the topology.

Curious - will this tool output "dot" formatted files for using a tool like GraphViz for plotting the graph ?

https://en.wikipedia.org/wiki/DOT_(graph_description_language)

If not, this may be a really neat format to export to so that a whole bunch of tools can consume, and visualize the networks laid out here.

Maybe some of this code can be adapted and reused:
https://github.com/ninuxorg/nodeshot/blob/0.9.x/nodeshot/scripts/read_topology_hna.py#L62

Replace nose with nose2 (testing library).

Error messages like 'metric cannot be None except when protocol is "static"' do not make much sense when adding support to formats like OpenVPN.

NetJSON NetworkGraph is useful as a generic json graph format, so we should be less picky about these petty details.

We should remove this check and update the tests, ensuring test coverage does not decrease.

At the moment having links with same target and source is not possible (the second is discarded by networkX).

Rename NetParserException in ParserError and NetParserJsonException in ParserJsonError for simplicity.

The Net prefix is redundant and doesn't add any value.

The Error suffix clearly conveys an error and also better follows PEP8: https://www.python.org/dev/peps/pep-0008/#exception-names

AlsoNetJsonException should be renamed to NetJsonError.

We should make a clear distinction between an Exception and an Error.

Reserved for GCI.

See instructions in: 6b72dab#r26618814

Drop support for python 2.

Also test python 3.6 and 3.7, drop python 3.5.