ciscotestautomation / ncdiff Goto Github PK

View Code? Open in Web Editor NEW

22.0 22.0 5.0 2.35 MB

NETCONF Diff Engine

License: Apache License 2.0

Makefile 0.52% Python 99.48%

ncdiff's People

Contributors

Stargazers

Watchers

Forkers

wouterdb jorschul netroller negmawon bobh66

ncdiff's Issues

TypeError: init() got an unexpected keyword argument 'host'

Hi,

Using version 2.1.3

When i try to connect via manager.connect:
test = manager.connect(host='192.168.0.99',
port='830',
username='ADMIN',
password='ADMIN',
hostkey_verify=False,
look_for_keys = False,
timeout=600)

I get the following error:

Traceback (most recent call last):
  File "test.py", line 209, in <module>
    nc.connect()
  File "test.py", line 39, in connect
    timeout=600)
  File "/usr/local/lib/python3.6/site-packages/ncdiff/manager.py", line 54, in connect
    return ModelDevice(session, device_handler, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/ncdiff/manager.py", line 98, in __init__
    **kwargs)
TypeError: __init__() got an unexpected keyword argument 'host'

When i remove **kwargs are a parameter in ModelDevice thats passed into NCCLIENT manager it works.

Am i missing anything to get it working off the bat? calling NCCLIENT standalone works.

Thanks

Crash on valid config due to leafref treated as identityref

Problem

a config containing the following leafref node <interface-name>vprn-xyz/abcdef:accessP2P</interface-name>

causes

  File "[..]/python3.6/site-packages/ncdiff/netconf.py", line 862, in node_sub
    self.node_sub(child_self, child_other)
  File "[..]/python3.6/site-packages/ncdiff/netconf.py", line 867, in node_sub
    if self._node_le(child_self, child_other) and \
  File "[..]/python3.6/site-packages/ncdiff/calculator.py", line 558, in _node_le
    if not self._node_le(child, child_other):
  File "[..]/python3.6/site-packages/ncdiff/calculator.py", line 558, in _node_le
    if not self._node_le(child, child_other):
  File "[..]/python3.6/site-packages/ncdiff/calculator.py", line 558, in _node_le
    if not self._node_le(child, child_other):
  File "[..]/python3.6/site-packages/ncdiff/calculator.py", line 519, in _node_le
    if not self._same_text(node_self, node_other):
  File "[..]/python3.6/site-packages/ncdiff/calculator.py", line 424, in _same_text
    return self._parse_text(node1) == self._parse_text(node2)
  File "[..]/python3.6/site-packages/ncdiff/calculator.py", line 390, in _parse_text
    return idref.default
  File "[..]/python3.6/site-packages/ncdiff/ref.py", line 53, in default
    return self.convert(self.node.text, to_node=None)
  File "[..]/python3.6/site-packages/ncdiff/ref.py", line 111, in convert
    url, id = self.parse_prefixed_id(tag, self.node)
  File "[..]/python3.6/site-packages/ncdiff/ref.py", line 72, in parse_prefixed_id
    .format(match.group(1), node.tag))
ncdiff.errors.ConfigError: unknown prefix 'vprn-xyz/abcdef' in the node with tag {urn:nokia.com:sros:ns:yang:sr:conf}interface-name

Analysis

When building the type structure, leafref is represented as -> (https://github.com/CiscoTestAutomation/ncdiff/blob/master/src/ncdiff/plugins/cxml.py#L356)

When calling _parse_text -> is handled as an identityref (https://github.com/CiscoTestAutomation/ncdiff/blob/master/src/ncdiff/calculator.py#L388)

unsupported format 'pyimport'

Hi
I have a built a Jenkins pipeline which calls a python script which in turn tries to get the netconf schema.
Important notice is that i'm using a jump host.
But i don't think this is causing any issues because i'm able to retrieve the configuration with
pcr1.get_config(source="running").data_xml
I have this simple script

    jumpbox_public_addr = "x.x.x.x"
    jumpbox_private_addr = jumpbox_public_addr
    target_addr_1 = input_args[1]
    target_addr_2 = input_args[4]

    jumpbox = paramiko.SSHClient()
    k = paramiko.RSAKey.from_private_key_file(os.environ["SSH_Keys"])
    jumpbox.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    jumpbox.connect(
        jumpbox_public_addr,
        username=os.environ["jump_user"],
        allow_agent=False,
        pkey=k,
    )
    # First connection
    jumpbox_transport = jumpbox.get_transport()
    src_addr = (jumpbox_private_addr, 22)
    dest_addr = (target_addr_1, 830)
    jumpbox_channel = jumpbox_transport.open_channel(
        "direct-tcpip", dest_addr, src_addr
    )
    # ncdiff.manager.ModelDevice
    pcr1 = manager.connect(
        host=target_addr_1,
        username=input_args[2],
        password=input_args[3],
        hostkey_verify=False,
        sock=jumpbox_channel,
    )
pcr1.scan_models(download="force")

But scan_models returns an error:

error /home/jenkins/workspace/PCR_Pipes/PCR_Diff/PCR/yang/.yang: [Errno 2] No such file or directory: '/home/jenkins/workspace/PCR_Pipes/PCR_Diff/PCR/yang/.yang'
Traceback (most recent call last):
File "pcr_compare.py", line 131, in
main(sys.argv)
File "pcr_compare.py", line 82, in main
pcr1.scan_models(download="force")
File "/usr/local/lib/python3.8/site-packages/ncdiff/manager.py", line 266, in scan_models
self.compiler = ModelCompiler(folder)
File "/usr/local/lib/python3.8/site-packages/ncdiff/model.py", line 628, in init
self.build_dependencies()
File "/usr/local/lib/python3.8/site-packages/ncdiff/model.py", line 676, in build_dependencies
self.dependencies = etree.XML(stdout.decode(), parser)
File "src/lxml/etree.pyx", line 3216, in lxml.etree.XML
File "src/lxml/parser.pxi", line 1896, in lxml.etree._parseMemoryDocument
File "src/lxml/parser.pxi", line 1777, in lxml.etree._parseDoc
File "src/lxml/parser.pxi", line 1082, in lxml.etree._BaseParser._parseUnicodeDoc
File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError
File "", line 1
lxml.etree.XMLSyntaxError: Document is empty, line 1, column 1
pcr_compare.py
Building dependencies: pyang --plugindir /plugins -p ./yang/ -f pyimport ./yang/*.yang
pyang return code is 1
unsupported format 'pyimport'

It seems that the directory (self.pyang_plugins = os.path.dirname(__file__) + '/plugins') is not correct. I don't understand why.
I'm using a docker container with
I'm using python 3.8.7 and ncdiff 20.10

Thank you for your help

How do you load roots for cases where the feature module uses augments ?

I was trying to understand the flow for a case where there is a top level yang module and then the other yang modules (features) use augments for updating the top level module. In such cases, the only way to retreive any configurational data is to use the top module which returns the entire configuration while using the other feature modules results in a tree without a root which does not work directly.

Timeout during schema files download

Timeout exception can be raised during schema download, and scan_models can still succeed (capabilites.txt file is created, no exception is raised).
Some yang files may be missing, and because scan_models verifies only capabilities.txt against the remote ones, there is no attempt to download missing yang files on subsequent calls.

Traceback of worker thread:

Exception in thread download_worker_0:
Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/home/user/ws/.venv39-yang/lib64/python3.9/site-packages/ncdiff/model.py", line 497, in run
    self.downloader.download(module)
  File "/home/user/ws/.venv39-yang/lib64/python3.9/site-packages/ncdiff/model.py", line 794, in download                                    
    reply = super(ModelDevice, self.device).execute(
  File "/home/user/ws/.venv39-yang/lib64/python3.9/site-packages/ncclient/manager.py", line 246, in execute                                                                                                          return cls(self._session,
  File "/home/user/ws/.venv39-yang/lib64/python3.9/site-packages/ncclient/operations/retrieve.py", line 199, in request                                                                                              return self._request(node)
  File "/home/user/ws/.venv39-yang/lib64/python3.9/site-packages/ncclient/operations/rpc.py", line 381, in _request                                                                                                  raise TimeoutExpiredError('ncclient timed out while waiting for an rpc reply.')
ncclient.operations.errors.TimeoutExpiredError: ncclient timed out while waiting for an rpc reply.

Main thread interrupted with ^C:

Traceback (most recent call last):
  File "/home/user/ws/solutions/c/yang/reprod.py", line 67, in <module>
    session.scan_models()
  File "/home/user/ws/.venv39-yang/lib64/python3.9/site-packages/ncdiff/manager.py", line 274, in scan_models
    d.download_all(check_before_download=(download == 'check'))
  File "/home/user/ws/.venv39-yang/lib64/python3.9/site-packages/ncdiff/model.py", line 763, in download_all
    self.download_queue.join()
  File "/usr/lib64/python3.9/queue.py", line 90, in join
    self.all_tasks_done.wait()
  File "/usr/lib64/python3.9/threading.py", line 312, in wait
    waiter.acquire()
KeyboardInterrupt

Model download can hang if both workers get exceptions

It is possible for the Model Downloader to hang if both of the workers encounter exceptions and leave their tasks in the queue.

In the DownloadWorker.run() method if self.downloader.download() raises an Exception, the worker will stop without marking the queued task as done. If that happens to both worker threads the process will hang.

A possible solution would be to catch the exceptions:

def run(self):
    while not self.downloader.download_queue.empty():
        try:
            module = self.downloader.download_queue.get(timeout=0.01)
        except Empty:
            pass
        else:
            try:
                self.downloader.download(module)
            except Exception as e:
                logger.exception(e)
            self.downloader.download_queue.task_done()
    logger.debug('Thread {} exits'.format(current_thread().name))

which will at least let the process finish without hanging.

validate_config() is not called

When a ConfigDelta object is initiated by delta argument, non-presence containers are not cleaned in config_dst:

config1 = """
<nc:config xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
    <aaa>
      <new-model xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-aaa"/>
      <common-criteria xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-aaa">
        <policy>genericstring</policy>
        <lifetime>
          <day>15</day>
        </lifetime>
      </common-criteria>
      <session-id xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-aaa">common</session-id>
    </aaa>
  </native>
</nc:config>
"""

config2 = """
<nc:config xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
    <aaa>
      <new-model xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-aaa"/>
      <common-criteria xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-aaa">
        <policy>genericstring</policy>
      </common-criteria>
      <session-id xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-aaa">common</session-id>
    </aaa>
  </native>
</nc:config>
"""
edit_config = """
<nc:config xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
    <aaa>
      <common-criteria xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-aaa">
        <policy>genericstring</policy>
        <lifetime>
          <day nc:operation="delete">15</day>
        </lifetime>
      </common-criteria>
    </aaa>
  </native>
</nc:config>
"""

c1 = Config(ncdevice=md, config=config1)
c2 = Config(ncdevice=md, config=config2)
delta = ConfigDelta(
  config_src=c1,
  delta=xml_.to_ele(edit_config),
)
print(delta.config_dst == c2)
delta.config_dst.validate_config()
print(delta.config_dst == c2)

The first print is False and the second is True. @wouterdb, why do we choose to skip validate_config()?
https://github.com/CiscoTestAutomation/ncdiff/blob/master/src/ncdiff/config.py#L115

Thanks,
Jonathan

Valid Unicode Characters Are Not Handled Correctly

In the following test the valid unicode character causes an exception in model.py:
11659: File "/ws/ranautiy-sjc/pytest1/lib/python3.6/site-packages/yang/ncdiff/model.py", line 555, in download
11660: f.write(reply.data)
11661: UnicodeEncodeError: 'ascii' codec can't encode character '\xa0' in position 813: ordinal not in range(128)

This code (the write is on line 555as complained about below):

    if reply.ok:
        fname = self.dir_yang + '/' + module + '.yang'
        with open(fname, 'w') as f:
            f.write(reply.data)
        self.downloaded.add(module)

…needs to be updated to be:

    if reply.ok:
        fname = self.dir_yang + '/' + module + '.yang'
        with open(fname, 'wb') as f:
            f.write(reply.data.encode('UTF-8'))
        self.downloaded.add(module)

There may be other places in the code that need to be modified to handle valid Unicode characters

NameError - undefined variable

Current master (bd1566f) throws NameError because of undefined 'repository' variable

Traceback (most recent call last):
  File "script1.py", line 14, in <module>
    m.scan_models()
  File "/home/user/ws/ncdiff/src/ncdiff/manager.py", line 273, in scan_models
    d = ModelDownloader(self, folder)
  File "/home/user/ws/ncdiff/src/ncdiff/model.py", line 738, in __init__
    repo = repository.FileRepository(path=self.dir_yang)
NameError: name 'repository' is not defined

$ flake8 src --select=F821
src/ncdiff/config.py:116:32: F821 undefined name 'Request'
src/ncdiff/config.py:118:27: F821 undefined name 'RestconfCalculator'
src/ncdiff/config.py:119:32: F821 undefined name 'SetRequest'
src/ncdiff/config.py:121:27: F821 undefined name 'gNMICalculator'
src/ncdiff/model.py:738:16: F821 undefined name 'repository'

Manually download YANG modules

Hi,

For certain reasons, for example you cannot download the YANG modules from the device due to a vendor bug, you might like to manually populate the ./yang folder with the modules yourself and do not try to download them from the device.

It is possible to pass an arbitrary value to scan_models() for the download parameter, like scan_models(download='ignore'), which will ignore downloading or checking YANG modules, but this is not documented and I guess not an 'official' way of doing it. Either check or force is expected.

Would it make sense to support this option and update the documentation?

Thanks,

Dan

Setting timeout causes an exception

When passing a timeout, the following happens

  for arg in supported_args:
            if arg in kwargs:
>               setattr(self, kwarg, kwargs[arg])
E               NameError: name 'kwarg' is not defined

/lib64/python3.8/site-packages/ncdiff/manager.py:104: NameError

This is looks like a typo, I will produce a patch.

Wouter

Wrong diff with whitespace in empty containers

Problem

When comparing the following configs

<spoke-sdp>
    <sdp-bind-id>10:1500</sdp-bind-id>
    <admin-state>enable</admin-state>
    <hash-label/>
</spoke-sdp>

<spoke-sdp>
    <sdp-bind-id>10:1500</sdp-bind-id>
    <admin-state>enable</admin-state>
    <hash-label>
    </hash-label>
</spoke-sdp>

They are found to be different.

Root cause

When xml input is parsed, this is done with XMLParser(remove_blank_text=True). However, from empty elements, whitespace is not removed

The following testcase demonstrates this. Adding a child element cause text to be None

import lxml.etree as etree

def test_parser():
    sample = """
     <hash-label>
                        </hash-label>"""
    parser = etree.XMLParser(remove_blank_text=True)
    tree = etree.XML(sample, parser)
    them = tree.xpath("/hash-label")
    assert len(them) == 1
    assert them[0].text is None  # fails

ciscotestautomation / ncdiff Goto Github PK

ncdiff's People

Contributors

Stargazers

Watchers

Forkers

ncdiff's Issues

Problem

Analysis

Problem

Root cause

Recommend Projects

Recommend Topics

Recommend Org