Git Product home page Git Product logo

netcfgbu's Introduction

Python 3.8 Code Style

version

Network Configuration Backup

As a network engineer I need to backup my network configuration files into a version control system, and I need a tool to automate this process. My primary means of accessing the devices is SSH.

Supported Devices
The netcfgbu tool was built specifically to backup network operating system (NOS) configurations that are monolithic in nature. That is to say the entire configuration can be captured using a command such as "show running-config". Any NOS that provides a monolithic configuration should be supported by netcfgbu.

Primary Considerations

  • I have a multi-vendor environment. I need to account for the different commands that are used to obtain the running configuration and disable paging if required.

  • I want to provide my network inventory in a simple CSV format. I want to create this inventory dynamically from one or more sources, for example Netbox. I want the ability to filter this inventory with limit and exclude constraints.

  • I may need to try multiple SSH credentials. I must not store my passwords in any configuration file, so this tool must acquire passwords via environment variables.

  • I will have a large number of devices (>1000) so I want this tool to take advantage of any and all techniques that reduce the total amount of time.


The general approach to netcfgbu is a configuration based methodology so as to not hardcode the tool to work with specific network device drivers and avoid the complexity and dependency of including a collection of 3rd-party libraries specific to network devices.

Read the Documenttion here.
Read the Quick Start here
Example netcfgbu.toml configuration

Introduction

Once you've setup the configuration file and inventory file you can backup all of your configurations using the command:

$ netcfgbu backup

At the end of the run, you will see a report, for example:

# ------------------------------------------------------------------------------
Summary: TOTAL=1482, OK=1482, FAIL=0
         START=2020-Jun-05 01:48:55 PM, STOP=2020-Jun-05 01:50:08 PM
         DURATION=72.566s
# ------------------------------------------------------------------------------

There are a number of other commands provided as shown via --help:

Usage: netcfgbu [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  backup     Backup network configurations.
  inventory  Inventory subcommands.
  login      Verify SSH login to devices.
  probe      Probe device for SSH reachablility.
  vcs        Version Control System subcommands.

Setup

The netcfgbu tool does not require you to create a configuration file, but for practical purposes you will generally need one. The file is TOML format. The default file is netcfgbu.toml and netcfgbu searches for in the current working directory. You can override this location using the -C <filepath> option or using the environment variable NETCFGBU_CONFIG

At a minimum you need to designate the inventory CSV file and a default set of SSH login credentials. The network device configs will be stored in the current working directory, or as specified in the defaults.configs_dir option. The configuration-file supports the use of environment variables.

Example:

[defaults]
    inventory = "$PROJ_DIR/inventory.csv"
    configs_dir = "$PROJ_DIR/configs"
    credentials.username = "$NETWORK_USERNAME"
    credentials.password = "$NETWORK_PASSWORD"

The netcfgbu defines the use of specific Environment Variables. Their use is the equivalent of the following configuration file. That is to say, if you did not provide netcfgbu a configuration file, this would be used:

[defaults]
    inventory = "$NETCFBU_INVENTORY"
    configs_dir = "$NETCFGBU_CONFIGSDIR"
    credentials.username = "$NETCFGBU_DEFAULT_USERNAME"
    credentials.password = "$NETCFGBU_DEFAULT_PASSWORD"

System Requirements and Installation

This tool requires the use of Python3.8.
Installation available via PyPI:

$ pip install netcfgbu

Questions or Suggestions?

Please open a github issue if you have any questions or suggestions.

Thank you!

netcfgbu's People

Contributors

jeremy-schulman-copilot avatar minitriga avatar morrowc avatar mtbutler07 avatar saparikh avatar thepacketgeek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

netcfgbu's Issues

Add CLI options to enable verbose SSH debug

As a user of the netcfgbu I may need to examine more verbose debug messages to troubleshoot SSH issues. Add the following:

--debug-ssh=<level [1-3]>

Where the debug levels correspond to the AsyncSSH documentation: here.

Add login re-try to handle misbehaving AAA controllers

As a network engineer using a AAA server that cannot respond "fast enough" to login requests due to the fact that netcfgbu runs concurrent tasks, I need the tool to account for this disability and perform one or more re-attempt to login using some form of backoff mechanism. I should be able to configure the number of retries; the default could be 2.

Add os_name alias mechanism

In order to avoid copy/paste of os_name section content, provide a means to alias one on_name to another. For example alias "catos" to be the same as "ios":

[os_name.ios]
    aliases = ['catos']
    pre_get_config = "terminal length 0"

Add "--exclude @<filename>"

Add the ability to exclude hosts using the contents of the file. The file must be a text file whose contents will be line-split and then will use the first word on the line as the value to exclude. The line-word-split will use the pattern of whitespace (includes tabs) and comma.

HP Procurve connector, ANSI escape codes in the output

When I was working on a connector for HP Procurve device, initially in order to handle annoying login banner asking to press a key to actually get to a CLI prompt, but despite the fact that the banner was handled properly with custom login(), read_until_prompt() hadn't matched the prompt pattern. The issue was caused by the presence of ANSI escape codes in the output. I managed to workaround it by incorporating strip_ansi_escape_codes() function from
https://github.com/ktbyers/netmiko/blob/0aa11265ea962f743f0ffa97d35ed4fa9aa0837d/netmiko/base_connection.py#L1777 in custom read_until_prompt() within my connector.

Are you interested in a PR which includes connector for HP Procurve in the code base and if so maybe there is more elegant way to handle the issue with escape codes?

Add limit ipaddr=<prefix>

Provide the ability to limit the inventory using an IP-prefix. For example, I want to include only inventory records that are in the 192.168.100.0/24 subnet:

netcfgbu inventory ls --limit ipaddr=192.168.100.0/24

To limit based on a list of ip subnets, for example any record in the 192.168.100.0/24 or the 10.1.255.0/27 subnet:

netcfgbu inventory ls --limit ipaddr=192.168.100.0/24,10.1.155.0/27

Add support for ssh_config options

In the version of AsyncSSH used, the ssh_config file is not parsed. Therefore we need to provide the User with the ability to assign the ssh_config options as describe here:
https://asyncssh.readthedocs.io/en/stable/api.html#supported-algorithms
and here:
https://asyncssh.readthedocs.io/en/stable/api.html#asyncssh.SSHClientConnectionOptions

At the global level in the configuration file:

[ssh_configs]
   kex_algs = ["ecdh-sha2-nistp256", "diffie-hellman-group14-sha1"]
   encryption_algs = [
      "aes128-cbc,3des-cbc",
      "aes192-cbc,aes256-cbc",
      "aes256-ctr,aes192-ctr",
      "aes128-ctr"]

Or at the OS-spec level of the config file:

[os_name.aireos]
   ssh_configs.kex_algs =  ["ecdh-sha2-nistp256", "diffie-hellman-group14-sha1"]
   ssh_configs.encryption_algs = ["aes128-cbc,3des-cbc"]

NOTE: AsyncSSH will support parsing ssh_config file(s); this work is underway in the development branch.

Add exclude=config_context

When fetching devices, include the exclude=config_context parameter to improve the response time in using the Netbox API. In some cases given a large number of devices, if this parameter is omitted and the limit=0 is set, then the API will Timeout.

Add failed list

Automatically create the file "failed.csv" that is the inventory of devices failed the operation. This way this file could them be used in a future "--exclude @" type of usage.

BUG: APP_CFG os_name None

Hitting a bug where os_name is None after it goes through validation.

(venv) axians@ubuntu-automation:~/netcfgbu$ netcfgbu login
Traceback (most recent call last):
  File "/home/axians/netcfgbu/venv/bin/netcfgbu", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/axians/netcfgbu/bin/netcfgbu", line 6, in <module>
    main.run()
  File "/home/axians/netcfgbu/netcfgbu/cli/main.py", line 11, in run
    cli(obj={})
  File "/home/axians/netcfgbu/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/axians/netcfgbu/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/axians/netcfgbu/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/axians/netcfgbu/netcfgbu/cli/root.py", line 66, in invoke
    super().invoke(ctx)
  File "/home/axians/netcfgbu/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/axians/netcfgbu/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/axians/netcfgbu/venv/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/axians/netcfgbu/netcfgbu/cli/login.py", line 84, in cli_login
    exec_test_login(ctx.obj["app_cfg"], ctx.obj["inventory_recs"], cli_opts)
  File "/home/axians/netcfgbu/netcfgbu/cli/login.py", line 25, in exec_test_login
    login_tasks = {
  File "/home/axians/netcfgbu/netcfgbu/cli/login.py", line 26, in <dictcomp>
    make_host_connector(rec, app_cfg).test_login(timeout=cli_opts["timeout"]): rec
  File "/home/axians/netcfgbu/netcfgbu/os_specs/__init__.py", line 11, in make_host_connector
    os_spec_def = get_os_spec(rec, app_cfg)
  File "/home/axians/netcfgbu/netcfgbu/os_specs/__init__.py", line 7, in get_os_spec
    return app_cfg.os_name.get(os_name) or OSNameSpec()
AttributeError: 'NoneType' object has no attribute 'get'

Jeremy found the issue within config_model.py whereby v needed to be returned on line 162.

Cheers for looking into this issue @jeremyschulman.

Git repo URL cannot be set as an environment variable

When I use an environment variable for the Git repo URL, I get the following error:

(cfgbu) samir-mbp15:cicd_book saparikh$ echo $GIT_REPO
[email protected]:saparikh/cfg-backup.git

(cfgbu) samir-mbp15:cicd_book saparikh$ netcfgbu login
Usage: netcfgbu login [OPTIONS]
Try 'netcfgbu login --help' for help.

Error: Configuration errors
    File:[/Users/saparikh/git/experimental/samir/cicd_book/netcfgbu.toml]
    Section: [git.0.repo]: Bad repo URL [$GIT_REPO]: expected to start with ('https:', 'git@').

This is the TOML file section:

[[git]]
    # the first entry does not require a name and it will be treated
    # as a default; i.e. when the --name option is omitted.
    repo = "$GIT_REPO"
    deploy_key = "$GIT_DEPLOY_KEY"

If I replace the env var with the URL, everything works:

[[git]]
    # the first entry does not require a name and it will be treated
    # as a default; i.e. when the --name option is omitted.
    repo = "[email protected]:saparikh/cfg-backup.git"
    deploy_key = "$GIT_DEPLOY_KEY"

Running version 0.60

(cfgbu) samir-mbp15:cicd_book saparikh$ netcfgbu --version
netcfgbu, version 0.6.0

Docs: -C command line option not allowed

Fresh install of netcfgu via pip and get the following error when trying to specify the path to the toml file:

netcfgbu -C netcfgbu.toml 
Usage: netcfgbu [OPTIONS] COMMAND [ARGS]...
Try 'netcfgbu --help' for help.

Error: no such option: -C

Setup section of the docs: https://github.com/jeremyschulman/netcfgbu#setup indicates this should work.

Help output from cli doesn't say anything about -C option

netcfgbu --help
Usage: netcfgbu [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  backup     Backup network configurations.
  inventory  Inventory subcommands.
  login      Verify SSH login to devices.
  probe      Probe device for SSH reachablility.
  vcs        Version Control System subcommands.

Add Missing Dependency (first)

Description

netcfgbu fails to run because the first package is missing from the requirements.txt file

Environment

  • Python version: 3.8.1
  • OS: Linux - RHEL7

Steps to Reproduce

  1. $ git clone https://github.com/jeremyschulman/netcfgbu.git
  2. $ cd netcfgbu
  3. $ python3.8 setup.py install --user
  4. $ netcfgbu

Expected Behavior

netcfg usage instructions

[3667673@lab-auto-dev netcfgbu]$ netcfgbu
Usage: netcfgbu [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  backup     Backup network configurations.
  inventory  [ls, build, ...]
  lint       Remove unwanted content from network config files.
  login      Verify SSH login to devices.
  probe      Probe device for SSH reachablility.

Observed Behavior

ModuleNotFoundError occurred

[marcus@lab-auto-dev netcfgbu]$ netcfgbu
Traceback (most recent call last):
  File "/home/3667673/.local/bin/netcfgbu", line 4, in <module>
    __import__('pkg_resources').run_script('netcfgbu==0.2.0', 'netcfgbu')
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1469, in run_script
    exec(script_code, namespace, namespace)
  File "/home/3667673/.local/lib/python3.8/site-packages/netcfgbu-0.2.0-py3.8.egg/EGG-INFO/scripts/netcfgbu", line 27, in <module>
ModuleNotFoundError: No module named 'first'

Add "linting" of configuration files

Need a mechanism to "clean" the configuration files so that extraneous content is removed and any content that would cause a false-positive diff-change is removed.

Example:

!Time: Fri Jun  5 17:49:00 2020

Change config section [[github]] to [[git]]

The netcfgbu tool can be used with any git based system, which includes GitLab for example. This issue tracks the BREAKING CHANGE to the configuration file to rename the section name from "github" to simply "git".

See discussion in #47 for further details.

cc @minitriga

Add support for configurable prompt pattern

The built-in prompt pattern doesn't work for Cumulus devices. Because of this pre_get_config is not usable.

You end up with this error when trying to use pre_get_config:

FAILURES: 20
host     os_name    reason
-------  ---------  -----------------------------------
leaf02   cumulus    TIMEOUT('Timeout awaiting prompt',)
leaf03   cumulus    TIMEOUT('Timeout awaiting prompt',)
leaf13   cumulus    TIMEOUT('Timeout awaiting prompt',)

The need for pre_get_config is coming from the fact that using this setting get_config = "( cat /etc/hostname; cat /etc/network/interfaces; cat /etc/cumulus/ports.conf; sudo cat /etc/frr/frr.conf)" results in the first few lines still being lost.

This is what should be retrieved:

cumulus@leaf01:mgmt-vrf:~$ ( cat /etc/hostname; cat /etc/network/interfaces; cat /etc/cumulus/ports.conf; sudo cat /etc/frr/frr.conf )
leaf01
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*.intf

auto lo
iface lo inet loopback
    address 10.16.1.1/32
    address 192.168.123.1/32

This is what is currently retrieved:

rmation, see interfaces(5).

source /etc/network/interfaces.d/*.intf

auto lo
iface lo inet loopback
    address 10.16.1.1/32
    address 192.168.123.1/32

Add config-file validation

Need to add code to check that the entire structure of the User provided configuration is valid before proceeding to use it. Some of the config file is programmatically checked in various functions as the content is used. What is needed is effectively a first-global-check before proceeding. Probably use a schema package like pydantic.

Add "--exclude" option

The opposite of "--limit" that includes, add an "--exclude"

Examples:

--exclude host=myhost
--exclude 'host=.*foo\.com'
--exclude os_name=nxos

configure ProxyCommand in netcfgbu.toml

I have a use case where I want one box doing all of the config backups across several data centers. I also have a jumpbox in each data center that I can use to access the mgmt network. However, in order to do this, I use the ssh config option ProxyCommand to proxy my connection to the network gear through the respective data center's jumpbox. Below is a snippet of my ssh config file:

Host !jumpbox.networkandrew.com *.networkandrew.com
    ProxyCommand ssh -W %h:%p [email protected]

If I'm able to use the above ProxyCommand in my netcfgbu.toml file, I'll be able to accomplish this.

Add per os_name credentials

I need to be able to provide one or more credentials to specific os-specs so that I can account for these types of use-caes.

Add doc: Why Netcfgbu?

Folks have been asking why I developed this tool and how it compares to existing options such as Rancid or Oxidized.

This issue tracks to creation of this document.

Add --inventory, -i options

Provide CLI options that specify the location of the inventory file so that the value need not be stored in the configuration-file.

As a workaround, you can set the inventory value to an environment variable as shown in the sample config

Add Environment variables to support default git settings

Add environment variables that would allow a User to define a set of equivalent [[git]] configuration so that it is not required in an actual configuration file.

For example:

NETCFGBU_GIT_REPO
NETCFGBU_GIT_TOKEN
NETCFGBU_GIT_DEPLOY_KEY
NETCFGBU_GIT_DEPLOY_KEY_PASSPHRASE
NETCFGBU_GIT_USERNAME
NETCFGBU_GIT_EMAIL

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.