Memory leak in yajl2_c backend about ijson HOT 10 CLOSED

Thanks a lot @tomplex for the report. The problem wasn't specific to items, but was actually in the underlying parse_basecoro method (so it affected not only items, but also kvitems and parse, and all their *_async and *_coro variants). As stated in the commit message, there are no tests yet to automatically check that memory consumption remains ~constant throughout a given ijson execution, and I usually center my own manual checks around basic_parse, which is why this problem wasn't caught earlier.

I'm actually a bit surprised that this problem wasn't reported earlier, given that keeping memory usage low is the whole point of ijson. But on the other hand the leak was only triggered by array structures, so if your JSON document had few arrays you wouldn't see your memory grow much, while in array-rich documents (like yours) it became more obvious.

BTW the error wasn't present in the 2.X ijson versions, and the leak was introduced by a line of code that got lost during the code refactoring for version 3.

Please give the new code a try (it's already on the master branch), and after confirmation I'll release a new ijson patch version to PyPI.

from ijson.

Oh, and BTW, the garbage collector bit is unnecessary -- memory was leaked not because of circular references, but simply because its count references were not reduced. In other words users shouldn't ever need to call the garbage collector. I take it you added it to your tests to ensure you were doing all you could to clean memory usage, but with the fix in place you shouldn't even see your memory grow in the first place.

from ijson.

Thanks for the quick response @rtobar! I'm running my test again right now, I'll let you know how it looks.

from ijson.

Awesome, it looks good! Here's the output:

using backend yajl2_c
starting memory usage: 0.008796 GB
memory usage after reading file: 0.010052 GB
memory usage after garbage collection: 0.010052 GB

As for the the garbage collector, you're right, it was just to make sure I was doing everything possible to clean up memory. I appreciate the clarification, though.

Again, thanks for the very quick response. I'm glad I'll be able to switch back to the yajl2_c backend for work tomorrow, because it is way faster. =) thanks for your great work on this project!!

from ijson.

No worries, I'll close this issue for the time being then and will leave a note once a new version is up in PyPI.

from ijson.

ijson 3.0.2 is now available in PyPI.

from ijson.

Hello,

Thank you for maintaining ijson. I wanted to say that this also seems to fix a segfault happening when doing something like:

import ijson

with open("test.json", "rb") as fp:
    l = list(ijson.items(fp, "item"))
print(len(l))

with open("test.json", "rb") as fp:
    l = list(ijson.items(fp, "item"))
print(len(l))

Where the second call to ijson.items would yield a segfault.

from ijson.

@lopter interesting. Out of curiosity, are you sure it was fixed in 3.0.2 and not in 3.0.3? In any case it's good to hear the problem is not present anymore.

from ijson.

@lopter interesting. Out of curiosity, are you sure it was fixed in 3.0.2 and not in 3.0.3? In any case it's good to hear the problem is not present anymore.

Oh yes, that was fixed in 3.0.3, I am now realizing that I should have posted in #29 but got confused.

from ijson.

Ok, now it makes more sense :-). Yes, that's exactly the type of situation that would have triggered the segfault reported at #29 provided that the top-level JSON value was an array.

from ijson.