openwisp / openwisp-network-topology Goto Github PK

The INSTALLED_APPS does contain some requirements that are not there out of the box, like allauth mention them in docs.

Similar to openwisp/openwisp-controller#199.

@atb00ker can you do this as part of the clean up work in this module pelase?
If yes, please assign it and put it into the board.

Add support for ZeroTier parser to generate network topology.

Dependencies

• openwisp/netdiff#106

@atb00ker, can you please take care of this when introducing swappable and merging? a0c5d59

Integrate topology visualizer in the admin in some way. A good way would be a button that displays an overlay like we do in openwisp-controller for the preview feature, but you may take a shortcut and just link the visualizer in the frontend: it depends from how much you want to learn:

1. just adding a link to the visualizer in the frontend is a shortcut that will be fast to implement and won't teach you much
2. visualizer in overlay will take a few days longer but will teach you stuff you probably don't know about django's internals

You may also consider to do both in a very agile way, first you quickly implement 1 and then you go ahead and implement 2, this approach can help you learn more about incremental development and fast iteration!

The choice is up to you @rohithasrk.

I got two exceptions which seem related. I'm not sure how to replicate this.

AttributeError: 'int' object has no attribute 'label'

AttributeError: 'int' object has no attribute 'label'
  File "django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 505, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 465, in handle_exception
    self.raise_uncaught_exception(exc)
  File "rest_framework/views.py", line 476, in raise_uncaught_exception
    raise exc
  File "rest_framework/views.py", line 502, in dispatch
    response = handler(request, *args, **kwargs)
  File "openwisp_network_topology/api/views.py", line 70, in post
    topology.receive(request.data)
  File "openwisp_network_topology/base/topology.py", line 352, in receive
    self.update(data)
  File "openwisp_network_topology/base/topology.py", line 287, in update
    self._update_added_items(diff['added'])
  File "openwisp_network_topology/base/topology.py", line 193, in _update_added_items
    self._update_node_properties(node, node_dict, section='added')
  File "openwisp_network_topology/base/topology.py", line 230, in _update_node_properties
    if node.label != node_dict['label']:

AttributeError: 'NoneType' object has no attribute 'label'

AttributeError: 'NoneType' object has no attribute 'label'
  File "django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 505, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 465, in handle_exception
    self.raise_uncaught_exception(exc)
  File "rest_framework/views.py", line 476, in raise_uncaught_exception
    raise exc
  File "rest_framework/views.py", line 502, in dispatch
    response = handler(request, *args, **kwargs)
  File "openwisp_network_topology/api/views.py", line 70, in post
    topology.receive(request.data)
  File "openwisp_network_topology/base/topology.py", line 352, in receive
    self.update(data)
  File "openwisp_network_topology/base/topology.py", line 289, in update
    self._update_changed_items(diff['changed'])
  File "openwisp_network_topology/base/topology.py", line 270, in _update_changed_items
    self._update_node_properties(node, node_dict, section=section)
  File "openwisp_network_topology/base/topology.py", line 230, in _update_node_properties
    if node.label != node_dict['label']:

Let's make sure this module can run with Django 3.2 and Django 4.0 alpha.

A snapshot cannot have organization different from the topology. We can safely remove the organization field.

Note for new contributors
The organization field is added using openwisp_users.mixins.OrgMixin

The receive API resource doesn't work because the organization for new nodes and links is not set.

Right now, properties of nodes and links are editable by users, but get mixed with properties which are automatically determined from the topology data, which is confusing for the user.

Here's what I think it would be best:

• add a user_properties field on Node and Link, ensure this field uses the django-flat-json-widget in the admin, place the new field after properties, label it something like user defined properties, the help_text could be "If you need to add additional data to this node | link (word depends on model) use this field"
• make the old properties field readonly, and show it in the UI as a list of keys and values
• in the model, make sure the new user properties are added to the other properties when generating the json data, but make sure these user generated properties are not shown when the original argument is passed as True
• ensure there's a test which checks the user properties are displayed in the json data, one test for node and one test for link

• rename /receive/{id}/ to /topology/{id}/receive/
• keep backward compatibility with /receive/{id}/, but add some kind of logged warning when that URL is used

The test suite logs error to the console which is not ideal. We should capture these logs like we did in other modules.

 $ ./runtests.py 
Found 143 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
................Failed to perform update topology admin action on <Topology: TestNetwork - NetJSON NetworkGraph>: TopologyRetrievalError
See error log for more information

..................................................................................Failed to perform add on <Node: 192.168.0.1>: IntegrityError
See error log for more information

................................/home/pandafy/openwisp/venv-network-topology/lib/python3.8/site-packages/django/db/models/fields/__init__.py:1409: RuntimeWarning: DateTimeField Ca.validity_start received a naive datetime (2022-02-02 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
/home/pandafy/openwisp/venv-network-topology/lib/python3.8/site-packages/django/db/models/fields/__init__.py:1409: RuntimeWarning: DateTimeField Cert.validity_start received a naive datetime (2022-02-02 00:00:00) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
........

Checkout the following functions:

• https://github.com/openwisp/openwisp-utils#openwisp_utilstestscapture_stdout
• https://github.com/openwisp/openwisp-utils#openwisp_utilstestscapture_stderr
• https://github.com/openwisp/openwisp-utils#openwisp_utilstestscapture_any_output

Take reference from openwisp/openwisp-controller#366

Python2 is very close to the end of it's lifecycle, it's time we drop the support for Python 2.

To do that,

1. We need to update all the dependencies to python3.
2. Remove code that exists for python 2 support.
3. Remove python2 from README & any documentation.

It is possible to create link in a topology with nodes belonging to different topology.

This can be seen in the following setUp method of a tes class

On line 327, the node belongs to topology object while the link is getting created with t object.

We need to add a validation test in AbstractLink class.

Release until openwisp-notifications 0.3 and openwisp-controller 0.8 are released, clean up .travis.yml and replace markdown in requirements-tests.txt with openwisp-notifications.

• allow creating relationships between topology nodes and controller devices
• add support for automatically determining the device to node relationship with openvpn (using the common name attribute)
• when a link changes its status, update the management IP and then trigger the device checks if openwisp-monitoring is installed
• find a way to show the device name as node label in the visualizer

See openwisp/openwisp-users#266.

I guess, this portion of code does the same thing as what we did recently in this PR openwisp/openwisp-users#251

So, will it not better if we simply import it from there?

Add python 3.8 and python 3.9, remove python 3.6 from GitHub actions testing.

Fix any bug that may appear.

Better to address openwisp/openwisp-controller#325 first.

Celery 5 should brings some breaking changes.

We should ensure the command to launch celery that is indicated in the README works and if not we need to update it.
Other than ensuring all tests pass, it's advised to do some manual testing running celery in the development environment, trigger the celery tasks and ensure they work.

Setup travis, coveralls, requires.io and add badges to README

Create a sub app for notifications. Notifications should be generated for following events

• When link status changes
• When a new node get added in topology

Depends on openwisp/openwisp-notifications#1

Hi, as shown in the following full dependency graph of openwisp-network-topology, direct dependency django-netjsongraph(required: >=0.4.2,<0.5.0, installed 0.4.3) depends on django-model-utils (no version constraint)， while the installed version of openwisp-utils(0.2.2) requires django-model-utils>=3.1.2,<3.3.0.

According to Pip's “first found wins” installation strategy, django-model-utils 3.2.0 is the actually installed version.

Although the first found package version django-model-utils 3.2.0 just satisfies the later dependency constraint (django-model-utils>=3.1.2,<3.3.0), it will lead to a build failure once developers release a appdirs whose version number is greater than 3.3.0.

Dependency tree--------

openwisp-network-topology -0.2.1
| +-django-netjsongraph(version range:>=0.4.2,<0.5.0)
| | +-django(version range:>=1.11,<2.1)
| | +-djangorestframework(version range:>=3.3,<3.8)
| | +-django-model-utils(version range:*)
| | +-netdiff(version range:>=0.6.0,<0.7)
| | +-openwisp-utils(version range:>=0.2,<0.3)
| | | +-django-model-utils(version range:>=3.1.2,<3.3.0)
| | +-jsonfield(version range:*)
| | +-six(version range:*)
| +-openwisp-users(version range:>=0.1.8,<0.2)
| | +-django-organizations(version range:>=1.0.0,<1.2.0)
| | +-django-extensions(version range:>=1.7.5,<2.2)
| | +-django-allauth(version range:>=0.35.0,<0.37.0)
| +-openwisp-utils(version range:>=0.2,<0.3)
| | +-django-model-utils(version range:>=3.1.2,<3.3.0)

Thanks for your attention.
Best,
Neolith

• Merge django-netjsongraph into openwisp-network-topology
• Migrate the django settings of each module so that the naming is consistent. (eg: NETJSONCONFIG_[SETTING_NAME] -> OPENWISP_CONTROLLER_[SETTING_NAME])
• Keep backward compatibility for setting names but log a warning to encourage the users to upgrade the name of the setting to the new name.
• Implement as easy as possible extensibility of Django modules.
• Add sample_app for testing extensibility with some changes.
• Update documentation

We see that a test by the name,
test_multiple_receive_remove(openwisp_network_topology.tests.test_topology.TestTopology) fails unusually in the travis builds. There is a need to investigate and fix it.

Implement a RESTfull API for openwisp-network-topology module, the API will allow to perform CRUD(Create, Retrieve, Update, Destroy) operations on these Model's:-

• Topology
• Node
• Link

Endpoints:-

• Topology
  • GET api/v1/topology/ (List topologies)
  • PUT/PATCH/DELETE api/v1/topology/{pk}/ (Change topology detail)
• Node
  • GET api/v1/node/ (List nodes)
  • PUT/PATCH/DELETE api/v1/node/{pk}/ (Change node detail)
• Link
  • GET api/v1/links/ (List links)
  • PUT/PATCH/DELETE api/v1/links/{pk}/ (Change link detail)

All of the above API endpoints:-

• Will have to respect multi-tenancy which can be achieved by various serializer classes present FilterSerializerByOrgMembership, FilterSerializerByOrgManaged, and FilterSerializerByOrgOwned in the openwisp-users module.
• Must be paginated.
• Respect django-permission system and also it may be required to inherit the custom permission and authentication classes from the openwisp-users module to restrict API usage for security reasons. Some of the important custom permission classes are IsOrganizationMember, IsOrganizationManager, IsOrganizationOwner,DjangoModelPermissions, IsAuthenticated, and many more.
• Add authentication classes in every endpoint to allow only the authenticated users to get access to all the available endpoints. BearerAuthentication and SessionAuthentication can be added to achieve this.
• All the serializer classes should be validated by the class ValidatedModelSerializer inheriting it from openwisp-utils.
• All the validation of the model should be preserved, also more validation methods may be introduced.
• Use transaction method wherever necessary when the database operation is taking place.
• Shall be taken care of proper query optimization to not slow down the API and reduce the number of queries performed while fetching data from the database.
• Add proper tests for all the endpoints and make sure the coverage doesn't decrease.
• Handle errors gracefully and return HTTP response codes that indicate what kind of error occurred to eliminate confusion for API users.
• Follow Django REST Framework proper coding practices and RESTful principles for building all the API’s.
• Add proper documentation for all the endpoints both in the README and OpenAPI (DRF Yasg).

Same data migration that is being added in this patch openwisp/django-netjsongraph#62 must be added here too.

• release netdiff
• update minimum netdiff version
• ensure change log is up to date
• tag
• publish on pypi

Organization field in Node and Link is redundant.

We should remove the field and make sure the multi-tenant admin can still work.

Similar to openwisp/openwisp-ipam#28.

Update: due to #107, it may be possible that the organization field in node and link may be useful to make sure nodes and links are always assigned to the right org.

Sometimes a single management VPN is used for an openwisp instance having multiple organizations.

In this case having a topology object which can be flagged as shared would be useful, also for the purpose of the integration with ow-controller and ow-monitoring.

Therefore we should:

• allow Topology objects to be shareable
• update the controller integration to work when the topology is shared (we need to create a test for this use cases and ensure DeviceNode objects are created correctly)

Reserved for GCI

Now that openwisp/openwisp-users#40 is ready we should enable operators to manage the following objects:

• Link
• Node

While these objects should be read-only (will only work for django >= 2.1):

• Topology

Administrators instead will be able to manage everything.

Similar PRs done in other modules:

• openwisp/openwisp-users#40
• openwisp/openwisp-controller#57

Integrate this feature of django-netjsongraph when is ready:

[feature] Topology history (snapshots - way back machine)

Migrate CI to github actions.

Perform the following tasks:

• add models with relations to openwisp_users.Organization
• add admin implementing multi-tenancy with limitations similar to the ones present in openwisp-controller
• add visualizer in the admin: #6
• release version 0.1.0

Additionally, once the previous tasks have been completed we should integrate this new module with ansible-openwisp2.

Remove the deprecated /receive/{id} URL and its related deprecation warning.

See #79 and #62 for more info.

When using the integration with openwisp-controller, instead of using a data migration to create the initial DeviceNode objects, we should use a management command, because the logic to create the initial DeviceNode objects is in a method which cannot be used during a migration because concrete models cannot be used during data migrations without risking future issues when adding/removing fields.

We'll also need to add a few lines in the readme regarding this command (when to use it and an example).

Right now the views (visualizer and API) are available to anyone.
This was done mainly because this module was used by community owned mesh network.

But now this module is also used to monitor private networks and we need to close it by default.
In the API we can use IsAuthenticated, DjangoPermissions and we should check the organization as well.
In the visualizer views we should use an equivalent mechanism

It should still be possible to disable auth via settings, eg: OPENWISP_NETWORK_TOPOLOGY_API_AUTH_REQUIRED.

tests are failing because of the absence of openwisp_notifications as one of the installed apps in TestMonitoringIntegration

Must be done after implementing register menu group functionality openwisp/openwisp-utils#189

Enable swagger using openwisp-utils, see openwisp/openwisp-utils#105.

Add some basic API documentation as indicated in openwisp/openwisp-users#114.

Add support for django 2.2, but first we need to upgrade django-netjsongraph to support django 2.2: openwisp/django-netjsongraph#93.

Once openwisp/netdiff#88 is merged, add support for it here, ensuring changes in nodes and links are detected (write a test for it).

Once the build passes and the PR is ready to be merged, we can release a new version of netdiff and increase the minimum version required here.

Add the customised admin theme of OpenWISP to the application. As implemented in openwisp-controller.

Ensure openwisp-controller 1.0.x runs in tests

With openwisp-utils being launched, all the shared code of openwisp-modules can be safely removed and used from the package.

Like openwisp/openwisp-controller#565.

Look for openwisp_users in migration files.