How to read json records in chunks using ijson? about ijson HOT 4 CLOSED

Thanks @Rstar1998, that's much clearer now :-)

The problem is that you are creating a single list with all the results, then feeding it to MongoDB. That is what's causing the problem, not the ijson iteration itself. What you need is to indeed chunk the results from the ijson iteration and feed those chunks to MongoDB.

To answer you direct question: no, ijson doesn't offer chunking itself. The good news is that we don't really need to, as this is a simple and common task. You could for example use itertools.islice for that, which doesn't require much work. Something like (taken from https://docs.python.org/3/library/itertools.html#itertools-recipes, see the one for "batched"):

    items = ijson.items(f, "item")
    while (batch := tuple(islice(items, n))):
        # insert batch into MongoDB

from ijson.

@Rstar1998 please follow the advice given in the template: share what you've tried, ask more precise questions, hopefully also some example data, etc. With such a broad description there's little help you can get.

from ijson.

@rtobar I have updated my description. Let me know if any more info is needed.

from ijson.

@rtobar . Thank you very much .

from ijson.