Implement kvitems in the C extension for increased performance. about ijson HOT 5 CLOSED

Thanks - I'm now at 13s (0.01ms/pair), memory issues seem gone.

Looking great!

from ijson.

@ltalirz could you check this one? As expected I'm getting about ~4x improvement on kvitems times for the C backend compared to the previous method, and (also as expected) kvitems seems to be slightly faster than items.

I also sped up a bit the python implementation of kvitems, which now seems to be ~10, 20% faster depending on the use case, as far as I could measure.

from ijson.

Hi @rtobar - thanks a lot for your efforts!

A few observations:

• Parsing the first 1M items of my JSON now takes ~16s (0.02ms/pair). That is within a factor of 2 of my revised timings for json.load, which is great.
• It seems like the C implementation leaks memory (it grows up to 2GB in my test)
• I checked that the pure python implementation uses ~13MB of memory throughout, i.e. the problem seems to be in the C extension

from ijson.

@ltalirz oops, bad reference counting there... I just pushed some further fixes to the kvitems branch (which also boosts performance by yet another tiny bit, at least in my local benchmarks), please give it another nudge when possible.

from ijson.

Code merged to the master branch, closing issue now.

from ijson.