Function sr_apply_changes is too slow about sysrepo HOT 10 CLOSED

I think it's not a python problem because the python function sess.apply_changes() is just a direct wrapper of sr_apply_changes. I'll test with a C program and check the results.

from sysrepo.

What sysrepo version are you using? Those numbers look rather extreme.

from sysrepo.

It's built from the master version yesterday.

from sysrepo.

Then please include more details. You are first calling sr_set_item_str() for all the interfaces and then sr_apply_changes()? What did you measure, only the sr_apply_changes() call? What nodes are created for each interface?

from sysrepo.

I tested with a python script:

#!/usr/bin/env python
# benchmark.py
import sysrepo
import sys
import time

def main():
    count = 1000
    if len(sys.argv) > 1:
        count = int(sys.argv[1])
    with sysrepo.SysrepoConnection() as conn:
        with conn.start_session() as sess:
            sess.switch_datastore("running")
            sess.delete_item("/ietf-interfaces:interfaces")
            sess.apply_changes()
            # generate fake config
            data = {
                "interfaces": {
                    "interface": []
                }
            }
            for i in range(count):
                data["interfaces"]["interface"].append({
                    "name": f"eth{i}",
                    "type": "ethernet-std:ethernetCsmacd"
                })
            sess.edit_batch(data, "ietf-interfaces")
            
            start = time.time()
            sess.apply_changes()
            end = time.time()
            print(f"Time used in apply_changes: {end-start}")

if __name__ == '__main__':
    main()

Run with python3 benchmark.py <count>.

from sysrepo.

Meaning it is at least to some extent running in Python. Then is not relevant, Python is known to be slow. Please try measuring C code directly, otherwise I can never be sure the benchmark has accurate numbers.

from sysrepo.

I created the same program in C and tested again. Here are my findings:

1. I removed all yang modules, and only install [email protected] and [email protected]. Both C and python programs run very fast. The time looks very linear. The function sr_apply_changes can finish in 6 seconds with 100k interfaces.
2. When I load our yang, the performance drops significantly. We use a lot of augment, leafref, and when conditions. I'm not sure which place caused the performance issue. I'll continue investigating.

from sysrepo.

I found time was spent in data validation. Our yang has augments like this:

  augment "/if:interfaces/if:interface" {
    when "if:type = 'eth-std:ethernetCsmacd'";
    container ethernet {
      leaf mtu {
        type eth-std:IF_MTU;
        default "2000";
        description
          "Ethernet port MTU";
      }
  // more stuff....
  }

Though my test program only sets name and type, sysrepo generates the missing leaf nodes with default values, then does the validation. My questions are:

1. Can we skip generating nodes with default values?
2. Can we skip validation?

from sysrepo.

1. No, they are required to be part of the datastore data for everything to work correctly.
2. No, the stored data are expected to and must be valid.

I am guessing you have added much more nodes and conditions because the ones posted should not cause any significant validation issues. I would suggest you optimize those conditions (and I can help with that if you can post it) or even validation can perhaps be optimized in case there are inefficiencies.

from sysrepo.

Thanks. My yang files are a bit chaotic. I think you can close this issue now. I'll try to clean up my yang modules.

from sysrepo.