Git Product home page Git Product logo

ndmtk's Introduction

Network Discovery and Management Toolkit

GitHub version CircleCI PyPI version Documentation Status

Network Discovery and Management Toolkit (ndmtk) makes Ansible "work" for both Traditional and Software-Defined Network (SDN) network management.

Table of Contents

  1. Overview
  2. Workflow Diagram
  3. Getting Started
  4. Documentation
  5. Questions
  6. Contribution

Overview

The future of network management lies in the area of Artificial Intelligence. Any network-enabled device will be able to build connectivity to a remote peer on-demand, without human intervention. The restraint on that ability are the AI-enabled systems acting as gatekeepers. AI is impossible without ongoing data collection, data analysis, probing, and modeling. As such, networks of the future need tools to perform the above tasks.

This toolkit is designed to accomplish the data collection piece of the AI puzzle. Specifically, the toolkit is designed to:

  • discover data on network devices and capture the entirety of available data
  • configure network devices via SSH, telnet, console, or terminal server
  • collect, analyze, and store the data via command-line interactions; it performs data analysisn and, if necessary, it performs additional data collection and/or device configuration tasks.

The intended audience of this toolkit are system and network engineers and designers, as well as the researchers dealing with AI.

The toolkit is delivered in a form of an Ansible plugin. However, it could work well with Chef, or any other orchestration tool. The reason Ansible became a framework of choice is its modularity. The toolkit itself is modular. It allows extended existing functionality. For example, the plugin does not blindly run pre-defined commands. Rather, it first collects all of the commands forming the understanding of the function of a particular device in a network. Once the plugin receives the data, it runs it through its algorithms and determines whether there are any additional command required to further gather data. That process continues until the algorithms determine that the collection is complete.

Importantly, once the plugin completes its tasks it produces a number of reports in JSON, YAML, and JUnit formats. These reports provide a map of what was done, where the collected data reside, and what that data is.

The plugin has no required arguments and parameters, because there are a number of default commands available for various operating systems, e.g. Cisco Nexus OS, Arista EOS, Linux, etc.

⬆️ Back to Top


Workflow Diagram

Plugin Workflow

⬆️ Back to Top


Getting Started

First, a user installs ndmtk with pip:

pip install ndmtk

Second, the user creates Ansible playbook, e.g. playbooks/collect_all.yml:

---
- name: generic data collection
  hosts:
  - ny-fw01
  - ny-sw01
  - ny-sw02
  gather_facts: no
  tasks:
  - name: data collection from three network devices
    action: ndmtk output="/tmp/ndmtk-%Y%m%d%H%M%S" debug=no no_host_key_check=yes on_error=continue

The above playbook collect the data from three devices: ny-fw01, ny-sw01, and ny-sw02.

Third, the user must create a hosts file:

controller ansible_connection=local

[test:children]
cisco-asa-firewalls
arista-eos-switches

[arista-eos-switches]
ny-sw01 os=arista_eos host_overwrite=localhost host_port=8224
ny-sw02 os=arista_eos host_overwrite=localhost host_port=8225

[cisco-asa-firewalls]
ny-fw01 os=cisco_asa host_overwrite=192.168.1.1

[all:vars]
ansible_connection=local

The switches are Arista vEOS switches running on top of Virtual Box. The firewall is a physical Cisco ASA 5505.

Additionally, the user must create Ansible configuration file in either:

  • .ansible.cfg in the user's home directory, or
  • ansible.cfg in the user's current directory

The configuration file contains the following directives:

[defaults]
inventory             = ./hosts
forks                 = 100
local_tmp             = $HOME/.ansible/tmp
retry_files_enabled   = True
retry_files_save_path = $HOME/.ansible/retries/
log_path              = $HOME/.ansible/log/ansible.log
transport             = local

Fourth, the user must create Ansible Vault ~/.ansible.vault.yml and structure it according to ndmtk's documentation. Then, for convinience, the user may store the password to the vault in ~/.ansible.vault.key plain-text file.

For example .ansible.vault.key has a single line for the password:

NX23nKz!

While the vault itself has the following content:

--
credentials:
- regex: ny-fw0[1-9]
  username: admin
  password: 'NX23nKz!'
  password_enable: '3nKz!NX2'
  priority: 1
  description: NY-FW01 password
- default: yes
  username: greenpau
  password: 'My#DefaultPass'
  password_enable: 'Enabled#By$Default'
  priority: 1
  description: my default password

The toolkit accesses ny-fw01 with the first set of credentials because its name matches the regular expression in that set. For the witches, the later, default password is used.

At any point of time, the user could edit or view the vault using the following commands:

ansible-vault edit ~/.ansible.vault.yml --vault-password ~/.ansible.vault.key
ansible-vault view ~/.ansible.vault.yml --vault-password ~/.ansible.vault.key

Finally, the user runs the playbook:

ansible-playbook playbooks/collect_all.yml

⬆️ Back to Top


Documentation

Please read the toolkit's documentation at Read the Docs and review the demo directory containing sample configuration files.

⬆️ Back to Top


Questions

Please open issues and ask questions in Github Issues.

⬆️ Back to Top


Contribution

Please contribute using the following Guidelines.

⬆️ Back to Top


ndmtk's People

Contributors

greenpau avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

ndmtk's Issues

makedirs with py3

Modify line 3190 of /home/pi/ndmtk/lib/python3.5/site-packages/ansible/plugins/action/ndmtk.py.

os.makedirs(commit_dir, mode=0o700, exist_ok=True)

ImportError: No module named boolean

Received the following error Due to changes in Ansible 2.4 related to boolean:

Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 118, in <module>
    exit_code = cli.run()
  File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 122, in run
    results = pbex.run()
  File "/usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py", line 159, in run
    result = self._tqm.run(play=play)
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 289, in run
    play_return = strategy.run(iterator, play_context)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/free.py", line 111, in run
    action = action_loader.get(task.action, class_only=True)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 378, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 357, in _load_module_source
    module = imp.load_source(full_name, path, module_file)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/ndmtk.py", line 259, in <module>
    from ansible.utils.boolean import boolean;
ImportError: No module named boolean

It further resulted in:

The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 138, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 576, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/ndmtk.py", line 425, in run
    self.conf['disable_defaults'] = boolean(self._task.args.get('disable_defaults'));
  File "/usr/lib/python2.7/site-packages/ansible/module_utils/parsing/convert_bool.py", line 26, in boolean
    raise TypeError("The value '%s' is not a valid boolean.  Valid booleans include: %s" % (to_text(value), ', '.join(repr(i) for i in BOOLEANS)))
TypeError: The value 'None' is not a valid boolean.  Valid booleans include: 0, 'on', 'f', 'false', 1, 'no', 'n', '1', '0', 't', 'y', 'off', 'yes', 'true'

plugin does not account for local_tmp configuration

Noticed that the plugin does account for local_tmp directory.

Thus, although Ansible configuration file has the following config, the plugin still creates temporary files under $HOME/.ansible/tmp.

local_tmp             = ./.ansible/tmp/

This issue stems from the following code: https://github.com/greenpau/ndmtk/blob/master/ndmtk/plugins/action/ndmtk.py#L390-L398

        self.conf['temp_dir'] = os.path.join(os.getenv("HOME"), '.ansible', 'tmp', self.plugin_name, self.conf['play_uuid'], self.conf['task_uuid'], self.info['host']);

Ansible reference: https://docs.ansible.com/ansible/latest/reference_appendices/config.html#default-local-tmp

YAMLLoadWarning

Received the following warning:

/usr/lib/python2.7/site-packages/ansible/plugins/action/ndmtk.py:2535: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  fc = yaml.load(f);

document how to address devices via serial console

In the documentation, when searching for "console" or "expect", the only hits are in the README, which speaks of this being part of the feature set. But how to address these devices (especially since Ansible itself usually only uses domains or IP addresses) is not explained.

The module ndmtk was not found in configured module paths

Hello,

Followed below steps in installing ndmtk.

[root@localhost collect_data]# pip install ndmtk
Collecting ndmtk
  Using cached https://files.pythonhosted.org/packages/5d/12/649225c25b943b495e4b4c890542699fd68cbdc678ae5abbc1d4d883e35f/ndmtk-0.2.0.tar.gz
Requirement already satisfied (use --upgrade to upgrade): ansible>=2.0 in /usr/lib/python2.7/site-packages (from ndmtk)
Requirement already satisfied (use --upgrade to upgrade): jinja2 in /usr/lib/python2.7/site-packages (from ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): PyYAML in /usr/lib64/python2.7/site-packages (from ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): cryptography in /usr/lib64/python2.7/site-packages (from ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): markupsafe in /usr/lib64/python2.7/site-packages (from jinja2->ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in /usr/lib/python2.7/site-packages (from cryptography->ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.8 in /usr/lib/python2.7/site-packages (from cryptography->ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /usr/lib/python2.7/site-packages (from cryptography->ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/site-packages (from cryptography->ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): enum34 in /usr/lib/python2.7/site-packages (from cryptography->ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /usr/lib/python2.7/site-packages (from cryptography->ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /usr/lib64/python2.7/site-packages (from cryptography->ansible>=2.0->ndmtk)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography->ansible>=2.0->ndmtk)
Installing collected packages: ndmtk
  Running setup.py install for ndmtk ... done
Successfully installed ndmtk

When tried to run a sample playbook, getting below error.

[root@localhost collect_data]# ansible-playbook -i hosts site.yml

PLAY [Collect Common data] ***********************************************************************************************************************************************

TASK [collect_common : data collection from three network devices] *******************************************************************************************************
fatal: [192.168.43.89]: FAILED! => {"msg": "The module ndmtk was not found in configured module paths"}
fatal: [192.168.43.87]: FAILED! => {"msg": "The module ndmtk was not found in configured module paths"}

PLAY RECAP ***************************************************************************************************************************************************************
192.168.43.89              : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
192.168.43.87              : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

How to fix this issue?

Thanks,
Mohan

can't read "expect_out(buffer)": no such variable

When a connection to remote devices times out:

can't read "expect_out(buffer)": no such variable
    while executing
"parse_cli "$expect_out(buffer)""
    invoked from within
"expect {
        timeout {
            puts $session_log "timestamp='[exec date]' transaction_id='$transaction_id' rc=$ERROR_CONNECTION_TIMEOUT timeou..."
    ("while" body line 7)
    invoked from within
"while 1 {
    incr loop_counter;
    if { $loop_counter > 250 } {
        send_status "connected:no";
        ctrl_exit $ERROR_CONNECTION_TIMEOUT;
   ..."

[ERROR] type object 'CLI' has no attribute 'read_vault_password_file'

During data collection, received the following error:

PLAY [data collection] *********************************************************************************************************************************************************
[ERROR] type object 'CLI' has no attribute 'read_vault_password_file'

This issue is originating from plugins/callback/ndmtk.py:

/usr/lib/python2.7/site-packages/ndmtk/plugins/callback/ndmtk.py:                _safe_lockpick = CLI.read_vault_password_file(_lockpick, loader=_safe_loader);

Per ansible/ansible#30824, The code in inventory.py only uses read_vault_password_file if run on version <2.4 (by checking for the CLI._play_prereqs attr). For 2.4, CLI.setup_vault_secrets() is used.

pip uninstall does not remove symlinks

When running the setup.py uninstall, the symlinks to ansible are getting removed properly:

# python setup.py uninstall
running uninstall
[INFO] deleted '/usr/lib/python2.7/site-packages/ndmtk'
[INFO] deleted '/usr/lib/python2.7/site-packages/ndmtk-0.1.9-py2.7.egg-info'

However, when using pip, the symlinks stay:

# pip uninstall ndmtk
Uninstalling ndmtk-0.1.9:
  Would remove:
    /usr/lib/python2.7/site-packages/ndmtk
    /usr/lib/python2.7/site-packages/ndmtk-0.1.9-py2.7.egg-info
Proceed (y/n)? y
  Successfully uninstalled ndmtk-0.1.9

# find /usr/lib/python2.7/site-packages | grep ndmtk
/usr/lib/python2.7/site-packages/ansible/plugins/action/ndmtk.py
/usr/lib/python2.7/site-packages/ansible/plugins/callback/ndmtk.py

The setup.py needs refactoring. Specifically, pip should remove the following files:

unlink /usr/lib/python2.7/site-packages/ansible/plugins/action/ndmtk.py
unlink /usr/lib/python2.7/site-packages/ansible/plugins/callback/ndmtk.py
rm -rf /usr/lib/python2.7/site-packages/ansible/plugins/action/ndmtk.pyc
rm -rf /usr/lib/python2.7/site-packages/ansible/plugins/callback/ndmtk.pyc

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.