corenova / yang-js Goto Github PK

I was trying to run yang-cord project and saw this error when starting the server -

events.js:221
target._eventsCount = 0;
^
TypeError: Can't add property _eventsCount, object is not extensible
at _addListener (events.js:221:25)
at Model.addListener (events.js:270:10)
at EventEmitter. (*....../yang-cord/lib/api/server.js:14:17)

I patched it like below to make it work locally -
https://github.com/corenova/yang-js/blob/master/src/yang.coffee#L29

      '_events':   writable: true
      '_eventsCount':   writable: true

I am unable to compile the following yang:
https://github.com/YangModels/yang/blob/master/vendor/cisco/xr/612/Cisco-IOS-XR-types.yang
throws following error:
ElementError: invalid YANG syntax detected around: '"(([a-f0-9]{16}-)(([1-9]?[0-9]|1[0-9][0-9]|2[0-4]"+ "[0-9]|25[0-5]).){3}([1-9]?[0-9]|1[0-9]'

At this statement:
typedef Bgp-ipv4-tunnel-addr {
type string {
pattern "((0:|[1-9][0-9]{0,4}:)"+
"(([1-9]?[0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}"+
"([1-9]?[0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]))";
}
It passes the syntax in the following scenarios.
If I change the pattern to (Remove \ before .)
pattern "(([a-f0-9]{16}-)(([1-9]?[0-9]|1[0-9][0-9]|2[0-4]"+
"[0-9]|25[0-5]).){3}([1-9]?[0-9]|1[0-9][0-9]|2[0-4][0-9]"+
"|25[0-5]))";

or to (replace " with ')
pattern '(([a-f0-9]{16}-)(([1-9]?[0-9]|1[0-9][0-9]|2[0-4]'+
'[0-9]|25[0-5]).){3}([1-9]?[0-9]|1[0-9][0-9]|2[0-4][0-9]'+
'|25[0-5]))';

Not sure what is the correct way of representing this syntax.
Thanks for your help.

I know that you are using https://www.npmjs.com/package/yang-parser which doesn't provide syntactic or semantic checks but it's too hard to use a tool like yang-js without that feature.

Getting warnings about event listeners when using yang-js and also when running tests. Is this normal beaviour?

Running node 4.4.7

(node) warning: possible EventEmitter memory leak detected. 11 created listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Yang.addListener (events.js:239:17)
    at Yang.once (events.js:265:8)
    at new Yang (/home/demonicblue/Code/yangtest/node_modules/yang-js/lib/yang.js:110:21)
    at Yang.<anonymous> (/home/demonicblue/Code/yangtest/node_modules/yang-js/lib/yang.js:130:20)
    at Yang._extend (/home/demonicblue/Code/yangtest/node_modules/yang-js/lib/yang.js:132:10)
    at /home/demonicblue/Code/yangtest/node_modules/yang-js/lib/expression.js:166:24
    at Array.forEach (native)
    at Yang.Expression.extends (/home/demonicblue/Code/yangtest/node_modules/yang-js/lib/expression.js:164:13)
    at new Yang (/home/demonicblue/Code/yangtest/node_modules/yang-js/lib/yang.js:95:23)
    at Object.<anonymous> (/home/demonicblue/Code/yangtest/node_modules/yang-js/lib/index.js:36:12)

It appears that when using Model.in '/some/xpath' to access an augmented node, it may not be resolving the XPATH properly. Need to diagnose the root cause issue and address it.

Need to revise the schema change adaptive property bindings to reduce the number of listener associations.

Creating a pet using yang-express:petstore example returns this error -

{
  "error": {
    "message": "callback.call(...).toJSON is not a function"
  }
}

The reason is that a 'true/false' value is returned by Property.create as part of its last statement execution (which is @emit 'create', value). I think the intent was to return the model/property (this) as part of create.

Adding a return statement in property.litcoffee like below fixes the issue:

      create: (value) ->
        @merge value, replace: false
        @emit 'create', this
        return this

As per discussion in #25, we should support mechanism for lazy population of the in-memory config state in order to minimize the potential performance/scale hit incurred when recreating the complete configuration state if the dataset in question is large.

I tried few updates on existing data and see this warning on the server console -

(node:874) Warning: Possible EventEmitter memory leak detected. 11 change listeners added. Use emitter.setMaxListeners() to increase limit

Currently, when Yang.eval is used on a given JS object instance, it internally modifies/changes the source data object tree, which enforces Yang expression schema directly inside the provided object.

When that same source data object is used again with Yang.eval, it will cause validation error for any auto-computed read-only state data since it will return the computed value when it should not be configurable from the schema perspective.

We will need to detect for such case at the entry point to the Yang.eval routine and unlink values for such data object nodes.

Currently, parse returns the same invalid schema instead of throwing an error.
So for following example:

var test = require('yang-js').parse(`
  module test {
    prefix "test";

    asd
  }
`)

test will have 'module test { prefix "test";\nasd }' but load will return following:

YinError: must pass in proper 'schema' to preprocess
  at Yin.Origin.error (/notebooks/node_modules/yang-js/lib/origin.js:171:13)
  at Yin.error (/notebooks/node_modules/yang-js/lib/yin.js:257:33)
  at Yin.preprocess (/notebooks/node_modules/yang-js/lib/yin.js:393:20)
  at Yin.compile (/notebooks/node_modules/yang-js/lib/yin.js:296:20)
  at /notebooks/node_modules/yang-js/lib/yin.js:246:34
  at Array.forEach (native)
  at Yin.use (/notebooks/node_modules/yang-js/lib/yin.js:244:49)
  at Yin.load (/notebooks/node_modules/yang-js/lib/yin.js:153:39)

Capture the timestamp of when events are triggered.

timestamp format? UTC? unixtimestamp?

I am trying to compile openconfig-bgp.yang.
https://github.com/openconfig/public/blob/master/release/models/bgp/openconfig-bgp.yang
This has two submodules
https://github.com/openconfig/public/blob/master/release/models/bgp/openconfig-bgp-common.yang
and
https://github.com/openconfig/public/blob/master/release/models/bgp/openconfig-bgp-common-multiprotocol.yang
both of them "belongs-to" the same module "oc-bgp"

Now, openconfig-bgp-common-multiprotocol.yang includes openconfig-bgp-common.yang
As per the spec, this seems to be valid.
Section 7.2.2 specifies:
A submodule MUST only be included by the module to which it belongs,
or by another submodule that belongs to that module.

But the parser throws an error here:
ExpressionError: [module(openconfig-bgp)/include(openconfig-bgp-common)/submodule(openconfig-bgp-common)] requested submodule 'openconfig-bgp-common' not belongs-to 'openconfig-bgp-common-multiprotocol'

Please let me know if I am doing something wrong here.

Model

module foo {
  list foo {
    key "name";
    leaf name {
      type string;
    }
    list bar {
      leaf name {
        type string;
      }
    }
  }
}

Event Subscription

model.on 'update', '/foo:foo/bar', (prop, prev) ->
    console.log 'updated:', prop.path.toString()

Seed data for application

{
  "foo:foo": [
    { 
      "name": "foo1"
    }
  ]
}

YangExpress Invocation

1. curl -i -X POST localhost:5050/foo:foo/foo1/bar -H 'content-type: application/json' -d '{ "name": "bar1" }'
2. curl -i -X POST localhost:5050/foo:foo/foo1/bar -H 'content-type: application/json' -d '[{ "name": "bar1" }]'

Result: None of these trigger my event subscription. I also tried modifying subscriptions to use '/foo/bar' or '/foo:foo/foo:bar' or '/foo:foo/bar/', none of those worked.

Should I expect the event triggered in this example, what am I doing wrong here ?

Does the framework allow binding a function only to "action", "rpc", "non-config" attributes in the model ? I tried binding a function to an attribute but the function never gets invoked.

e.g. "/subscriber/foo" where "leaf foo { type string; }" and the function is -

'/cord:subscriber/foo': -> "foo-#{@get '../id'}"

Should this be allowed at all ? The idea is to run a bound function when an attribute in the model changes. It may be my wrong understanding/expectation from this framework. However, the documentation in yang-js says I could bind a function to any part of the YANG data model.

According to RFC7951#6.9 [null] should be the valid value of an empty type data.
However when I evaluated a JSON value with [null], the following error was thrown:

ExpressionError: [module(example-sports)/leaf(leaf_empty)] predicate validation error during apply

Is there a concept of detached models, if not deep clone of model properties ?

Use Case: Do not trigger recursive events

1. I have a event subscription for update on something.
2. I receive prop as argument to that event.
3. I need to transform few properties in this prop object to something else e.g. prop.foo = newfoo
4. Step 3 triggers the change event again. Is there a better way to do this transform on the object ?

I am trying to parse open-config yangs using this library. But it is throwing some errors. These yangs compile fine with other compilers.
Some examples of the errors:

1. Following yang:
   https://github.com/openconfig/public/blob/master/release/models/interfaces/openconfig-interfaces.yang

throws following error:
ExpressionError: [module(openconfig-interfaces)/grouping(interface-ref-common)/leaf(subinterface)/type(leafref)/path] invalid path expression 'oc-if:interface[oc-if:name=current()' found in /oc-if:interfaces/oc-if:interface[oc-if:name=current()/../interface]/oc-if:subinterfaces/oc-if:subinterface/oc-if:index

2. This yang:
   https://github.com/openconfig/public/blob/master/release/models/policy/openconfig-routing-policy.yang

throws following error:

ExpressionError: [module(openconfig-routing-policy)/grouping(generic-actions)/choice(route-disposition)] cannot contain more than one non-case data node statement

Please let me know I am doing something wrong.

yang-module:

module uses-aug {

  grouping foo-grp {
    container cont-to-be-aug {

    }
  }
  uses foo-grp {
    augment "./cont-to-be-aug" {
      leaf leaf-a {
        type string;
      }
    }
  }
}

After the module is loaded, I loop over every element in the yang instance, no additional node is augmented.

After checking the RFC, ./cont-to-be-aug does not seem to be a formal node identifier.
Maybe an error message is needed?

Even I set the value as a "./123"(a non-existing node), the situation is the same.

Model (alongside example/petstore.yang)

bash-4.3$ cat example/foo.yang
module foo {
  prefix foo;
  import petstore {
    prefix ps;
  }
  list foo {
    key "name";
    leaf name {
      type string;
    }
    leaf petid {
      type leafref {
        path "/ps:pet/ps:id";
      }
      mandatory true;
    }
  }
}
bash-4.3$

Create a foo:foo with wrong leafref value:

bash-4.3$ curl -i -X  GET http://localhost:5000/pet
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 172
ETag: W/"ac-sBDpdXrQXR+j1xDWHpGmuQ"
Date: Fri, 26 Aug 2016 20:30:04 GMT
Connection: keep-alive

{
  "petstore:pet": [
    {
      "id": 1,
      "name": "happy",
      "tag": "friendly"
    },
    {
      "id": 2,
      "name": "boba",
      "tag": "hyper"
    }
  ]
curl -i -X POST http://localhost:5000/foo:foo -H 'content-type: application/json' -d '{ "name": "foo1", "petid": 3 }'
HTTP/1.1 201 Created
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 433
ETag: W/"1b1-dL2bxXGURlemyRmBIUU0bg"
Date: Fri, 26 Aug 2016 20:30:11 GMT
Connection: keep-alive

{
  "foo:foo": [
    {
      "name": "foo1",
      "petid": {
        "error-tag": "data-missing",
        "error-app-tag": "instance-required",
        "err-path": {
          "kind": "xpath",
          "tag": "/",
          "xpath": {
            "kind": "xpath",
            "tag": "ps:pet",
            "xpath": {
              "kind": "xpath",
              "tag": "ps:id"
            }
          }
        }
      }
    }
  ]
}

Petstore Server Log

[yang-express] initializing...
[openapi] enabling...
[openapi] enabled ok
[yangapi] enabling...
[yangapi] enabled ok
[restjson] enabling...
[restjson] enabled ok
[websocket] enabling...
[petstore] importing 'petstore:pet' from model to the store
{ data: { 'petstore:pet': [ [Object], [Object] ] } }
[petstore] importing 'foo:foo' from model to the store
[yang-express] start of a new journey
[websocket] binding to server
[websocket] enabled ok
[restjson:petstore] calling GET on /pet
[restjson:petstore] calling GET on /pet
[restjson:petstore] calling POST on /foo:foo
[@name] update for /foo:foo

I have a yang file with feature statement for example:

module example-sports {
  namespace "http://example.com/example";
  prefix example;

  feature my_feature {
    description "my feature";
  }
  container feature-test {
    if-feature my_feature;
    leaf a-leaf {
      type string;
    }
  }
}

When I eval the schema with the data:

{
 "example-sports:feature-test":{
    "a-leaf":"hello"
  }
}

a model instance (ex:x) is generated.

I called x.enable("my_feature") and got the following error:
PropertyError: [/example-sports] unable to enable unknown feature 'my_feature'

Is there a any bug or I misunderstand the function?

During parse of a given schema, it can contain type references to definitions that have yet to be parsed. Will need to support deferred resolution after the dependent definitions have been loaded instead of failing when such self-referential resolution is encountered.

I am not sure what do you plan to allow for model persistence in a yang-js based application, hence my question.

Do you have plans to support data provider APIs e.g. yang-js to allow custom data provider plugins to be able to load and store data directly from the provider using a standard provider API.

I see that the eventing mechanism can be used to persist data in a datastore. This applies to create/update/delete may be. However, I am not sure about your plans to support fetch on behalf of a GET. May be I am yet to understand how this currently happens.

Concurrency: Any thoughts ? Depending on single threaded model that will never allow simultaneous update to the same model instance ?

Transactions: Need to incorporate it in all model updates, may be enforce the application developer to first obtain a transaction handle from the store and rollback changes in case of a failure ?

Probably a thing to think about while designing default store implementation (currently in-memory).

The new 0.14.0 release has been published into npm.

This release is a complete forklift over prior yang-js. It's leaner, faster and simply awesome.

Please be sure to refer to the README for more info on the new APIs and usage examples.

When the description field is empty string e.g. description ""
Parser is throwing exception. This seems like a valid syntax, but I am not sure.

The following code references a key in a grouping and fails parsing.

module foo {
    grouping quisling {
      leaf first {
        type string;
      }
      leaf second {
        type int32;
      }
    }
    container top {
      list kek {
        key "first";
        uses quisling;
      }
    }
  }

ExpressionError: foo/top/kek/key[first] referenced key items do not have leaf elements

I can write a test for this case but I'm unsure where it would fit. list.coffee or maybe grouping.coffee

The toObject()-function is returning an error:
[TypeError: Object.keys called on non-object]
when trying to convert a yang model into json.

I've tried a minimal model and found this converts correctly into a json-object:
module my-module { prefix "mymodule"; }

While this produces the error:
module my-module { prefix "mymodule"; container foo { description "bar"; } }

This bug does not seem to exist in v.0.14.10, but is present in v.0.14.18.

For faster lookups during key-based list item selection, we should introduce a dynamic hash index on every list property. It should use caching strategy and only re-calculate when the keys change within the collection. Some of the triggers for this would be an item getting added or removed but also if a key for an existing item gets updated.

There are additional methods available that should be described in the README.

Below are comprehensive listing of public methods (undocumented ones marked with *).

module-level:

• yang.parse
• yang.compose
• yang.require
• yang.register
• yang.use*
• yang.resolve*

Yang/Expression/Element instance:

• bind
• eval
• extends
• toString
• toObject
• locate*
• match*
• access*
• resolve*
• clone*
• merge*
• update*
• lookup*
• error*
• debug*

When there are multiple include statements in a file, looks like only the first include is compiled.
When we encounter the first include, the 'compile' flag is set to false, causing further includes not to be compiled

When using the provided yang.require facility, it would be convenient to try to dynamically resolve import dependencies for the YANG schema being loaded. This would reduce the need to explicitly pre-require dependencies and present similar functionality as would be expected of require behavior.

Few questions -

• Does an emitted event indicate what exactly what action was performed against the model e.g. CREATE/UPDATE/DELETE ?
• Do I have an option to subscribe to only part of the model changes and not the complete model itself. e.g. I am interested in change events for a subscriber model, but only for its "id" field change. If the event subscription API allows to specify an xpath filter to be specified for this kind of "model change interest".
• Do the event listeners get to know the "delta changes" that happens to a model or just the latest model i.e. prop.content ? I see that the delta changes notion is available in some form for PUT requests.

When an element is removed from an array via the .remove() which internally calls Array.slice(), it triggers a cascade of shift operations for each of the higher index entries to be newly .set() on the prior position.

We need to bypass schema validations for such scenario so that performance won't be affected when the array is large and the value being removed is at the beginning part of the array.

See https://datatracker.ietf.org/doc/draft-ietf-netmod-rfc6020bis/

When we deep clone a grouping statement while resolving uses, we lose the parent relationship mapping for elements within grouping if there are additional grouping/uses within the grouping being cloned. This causes issues in properly identifying which elements were dynamically extended via compile vs. those that were inherently defined as part of the schema definition.

The require method currently only handles imports. Handling include for submodules should also work.

I'll try to fix this and create a pull request.

I'm having issues parsing the yang files that comes with ConfD, specifically tailf-common. The error reads

{ [ExpressionError: [constructor] invalid YANG syntax detected]
  name: 'ExpressionError',
  context: 'tring { pattern "S(\\d+G)?(\\d+M)?(\\d+K)?(\\d+B)?' }

and the corresponding code is

  typedef size {
    type string {
      pattern
        "S(\d+G)?(\d+M)?(\d+K)?(\d+B)?";
    }
    description
      "A value that represents a number of bytes.  An example could be
       S1G8M7K956B; meaning 1GB + 8MB + 7KB + 956B = 1082138556 bytes.
       The value must start with an S.  Any byte magnifier can be left
       out, e.g. S1K1B equals 1025 bytes.  The order is significant
       though, i.e. S1B56G is not a valid byte size.

       In ConfD, a 'size' value is represented as an uint64.";
  }

However, parsing the typedef out of the context of the tailf-common module renders no errors.

Any ideas on how to debug?

Current facility expects extensions/typedefs to be internally resolvable while parsing a YANG schema that uses those extensions.

When a YANG schema defines new custom extensions, it requires preparing such extensions at the Registry or Source level which is currently considered private.

Need to come up with a better way of passing in these implementation Extension instance(s) during resolve, so that it becomes easier for developers to create custom extensions.

a unique clause that reference to not the immediate children of the container/group, but to a descendant further down does not seem to work.
Following yang: https://github.com/openconfig/public/blob/master/release/models/network-instance/openconfig-network-instance.yang
It throws an error:
ExpressionError: [module(openconfig-network-instance)/grouping(network-instance-top)/container(network-instances)/list(network-instance)/container(interfaces)/list(interface)/unique(config/interface,config/subinterface)] referenced unique items do not have leaf elements

I have following models:

foo.yang

module foo {
  prefix foo;
  list foo {
   leaf name {
      type string;
   }
}

bar.yang

module bar {
  prefix bar;
  import foo { prefix f; }

  leaf bar {
    type leafref { path "/f:foo/f:name"; }
  }
}

When I create an instance of foo (under yang-express using published URLs), and then try to create a bar instance supplying correct value of bar leafref, I always see error regarding unable to resolve the leafref.

From the context of an Event handler routine, it should be easier to find relative data element(s) from the perspective of the prop which generated the event. Since the prop which triggers the event of interest can be any sub-tree element inside the Model, we need a way to extract sufficient level of XPath details that we can then extend from in order to get to the data element(s) of interest.

We will need to enhance the Property instance to provide access to the XPath instance which you can tweak in order to allow you to grab the related data of interest.

This enhancement was derived from suggestions from @ramukima.

For following example:

var test = require('yang-js').load(`
  module test {
    prefix "test";

    leaf ttt {
        type string {
            length 1;
        };
    }
  }
`)

I have had following error:

YinError: must pass in proper 'schema' to preprocess
  at Yin.Origin.error (/notebooks/node_modules/yang-js/lib/origin.js:171:13)
  at Yin.error (/notebooks/node_modules/yang-js/lib/yin.js:257:33)
  at Yin.preprocess (/notebooks/node_modules/yang-js/lib/yin.js:393:20)
  at Yin.compile (/notebooks/node_modules/yang-js/lib/yin.js:296:20)
  at /notebooks/node_modules/yang-js/lib/yin.js:246:34
  at Array.forEach (native)
  at Yin.use (/notebooks/node_modules/yang-js/lib/yin.js:244:49)
  at Yin.load (/notebooks/node_modules/yang-js/lib/yin.js:153:39)

The parser seems to ignore concatenation of quoted strings. For example, "hel" + "lo" should be equivalent to "hello". This behavior is noticed when parsing patterns containing regular patterns.
An example is
typedef domain-name
in ietf-inet-types.yang.

The parses seems to only parse the first substring as a regular expression which will fail as it is not valid on its own.

Note: ietf-inet-types.yang will not parse anyway beacause of #6

Refer to #52 and #53.

Implement the boolean conditional logic for if-feature statement using existing comparse library.

If an import gets declared more than once in a given module (due to duplicative import during submodule include) we observe this error. Need to review and come up with a graceful way to handle this.

A Store should house a collection of Models and serve as the root for XPATH accessibility tree (see #31).

Currently, the Yang schema instances are captured by the Yang class. This enables schema-level resolutions during Yang.parse.

The Store will need to attach the Model properties and allow population of data state. The Store will be the primary entity for data provider (persistence layer) adaptors and support load/dump type operations (see #24).

We still want to preserve direct per-Model interactions. Since a Model does not need to be a YANG module schema, we will need to differentiate "module" models vs non-module models and handle them appropriately from the Store.