netopeer2-server crashing while establish-push notification generated about sysrepo HOT 7 OPEN

As long as node is not NULL (which it cannot), value_tree should point to the duplicated tree, which must then be printed into xml, so I do not know which of these assumptions is incorrect. But you should be able to find out using gdb. You can also provide the steps to reproduce and I can then debug this myself.

from sysrepo.

Steps I followed to arrive at the issue:

1. Start netopeer2-server: $ sudo netopeer2-server -d -v3
2. start two netopeer2-cli: $ neopeer2-cli -d -v3 and $ connect
3. On one netopeer2-cli: $ establish-push --datastore running --on-change --no-sync-on-start
4. On the other netopeer2-cli: $ edit-config --target running --config=my_xml_file.xml
5. On server side, the issue occurs.

One observation is that it doesn't fail everytime, sometimes if the xml change is small, it succeeds repeatedly, but for the same change next time if we perform the edit-config with different values for generating the notification, it fails. In case of a big change, it definitely fails. I've ensured my xml content is valid.

I will check your first point by debugging further and will update.

from sysrepo.

node and value_tree both are not NULL. out is also a valid pointer everytime. Not sure how to check for duplicated tree but is pointing somewhere.

I have two more observations in the failure case:

1. I'm performing edit-config for one yang module, but when I'm printing this xml before assert(xml), it is printing old xml data of another yang module which I edited yesterday. Guess it is still present somewhere in the shm.

2. After crashing of netopeer2-server, next time when it is started, seems like it is recovering some locks. The logs are:

`[INF]: SR: Connection 45 created.

[INF]: SR: Session 915 (user "root", CID 45) created.
[WRN]: SR: Connection with CID 44 is dead.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:get-config" subscription of CID 44.
[WRN]: SR: Recovered a read-lock of CID 44 (_sr_rpc_subscribe).
[WRN]: SR: Recovering RPC/action "/ietf-netconf:edit-config" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:copy-config" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:delete-config" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:lock" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:unlock" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:get" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:kill-session" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:commit" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:cancel-commit" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:discard-changes" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf:validate" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf-monitoring:get-schema" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/notifications:create-subscription" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf-nmda:get-data" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-netconf-nmda:edit-data" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-subscribed-notifications:establish-subscription" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-subscribed-notifications:modify-subscription" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-subscribed-notifications:delete-subscription" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-subscribed-notifications:kill-subscription" subscription of CID 44.
[WRN]: SR: Recovering RPC/action "/ietf-yang-push:resync-subscription" subscription of CID 44.
[WRN]: SR: Recovering module "ietf-netconf-monitoring" operational get subscription of CID 44.
[WRN]: SR: Recovering module "nc-notifications" operational get subscription of CID 44.
[WRN]: SR: Recovering module "iana-ssh-public-key-algs" operational get subscription of CID 44.
[WRN]: SR: Recovering module "iana-ssh-key-exchange-algs" operational get subscription of CID 44.
[WRN]: SR: Recovering module "iana-ssh-encryption-algs" operational get subscription of CID 44.
[WRN]: SR: Recovering module "iana-ssh-mac-algs" operational get subscription of CID 44.
[WRN]: SR: Recovering module "ietf-subscribed-notifications" operational get subscription of CID 44.
[WRN]: SR: Recovering module "ietf-subscribed-notifications" operational get subscription of CID 44.
[INF]: SR: Triggering "ietf-netconf-server" "done" event on enabled data.
[INF]: LN: Listening on 0.0.0.0:830 for SSH connections.
[INF]: LN: Listening on 0.0.0.0:6513 for TLS connections.
[INF]: SR: Triggering "ietf-keystore" "done" event on enabled data.
[INF]: SR: Triggering "ietf-truststore" "done" event on enabled data.
[INF]: SR: Triggering "ietf-netconf-acm" "done" event on enabled data.
[INF]: SR: Triggering "ietf-netconf-acm" "done" event on enabled data.
[INF]: SR: Triggering "ietf-netconf-acm" "done" event on enabled data.
[INF]: SR: Triggering "ietf-netconf-acm" "done" event on enabled data.
[WRN]: SR: Recovering module "sysrepo-monitoring" operational get subscription of CID 44.`

from sysrepo.

Hi Michal,
For testing purpose I commented out the assert(xml) thereby preventing the server from crash.

My observation is that for whichever node, if xml comes out to be NULL, the value for that node isn't edited by edit-confg rpc. So the notification is generated for all the nodes present in the module, but not all the nodes' values are changed.

Here is the output of the establish-push on-change rpc:

notification (2024-06-13T19:11:32.511027866+05:30)
<push-change-update xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push">
  <id>1</id>
  <datastore-changes>
    <yang-patch>
      <patch-id>patch-1</patch-id>
      <edit>
        <edit-id>edit-1</edit-id>
        <operation>create</operation>
        <target>xpath of node1</target>		------>>> value node is created.
        <value>
          <node1_name>
            <node2_name>7</node2_name>
          </node1_name>
        </value>
      </edit>
      <edit>
        <edit-id>edit-2</edit-id>
        <operation>create</operation>
        <target>node2_name</target>			------->>> value node is not created
      </edit>
    </yang-patch>
  </datastore-changes>
</push-change-update>

In this case, for node1_name, xml is not NULL, but for node2_name, xml is NULL. Because I haven't modified the value for node2_name.

I think the if we establish-push for the entire module or for all the modules in general, only changed nodes shall be included in the notification and not unchanged. Then assert(xml) will not fail.

from sysrepo.

Could you help me reproduce it, with all the YANG and data files? I am quite lost and another test case would be useful, too.

from sysrepo.

Hi Michal, because of restriction I'm not able to share the yang models but one thing about them is multiple yang models are interlinked. i.e. nodes from one yang model are augmented into another, Several leafrefs are used from another model.
So when the notification is coming for a specific module, it also consists of nodes which are not defined in that module but due to leafrefs and aumentation, those nodes are also included in the notification structure.
So when we're traversing through the notification structure, we don't find the value for them and that is causing assert(xml) to fail. Now that I commented out assert(xml), it is working fine.

from sysrepo.

It should still be possible to print data, even when their targets are not present in the same data tree so while commenting out the assert may avoid the crash, it is not causing it to work properly (there is no value but there should be). But if I cannot reproduce it, I will not able able to fix it. You can always send the steps, YANG modules, and any other files over email.

from sysrepo.