It is not possible to parse a file with newline (\n etc.) about ijson HOT 3 CLOSED

@k-zaytsev this was a puzzling one, but I think the culprit is aiofile (version 3.1.1 locally), and this had nothing to do with newlines:

import asyncio

from aiofile import AIOFile

data = b'hi'

async def main():
    with open("test.data", "wb") as fp:
        fp.write(data)
    async with AIOFile("test.data", "rb") as fp:
        while True:
            buf = await fp.read(2)
            if not buf:
                break
            print(buf)

asyncio.run(main())

When running locally, that script never finishes. It doesn't matter how many bytes you try to read from fp (I'm reading 2), it will always read them from the start of the file without updating its current position. This is certainly not what one usually expects from a file-like object (and certainly against ijson's expectations, which stops parsing when no more data is read). In fact in the original code you could see two as being printed before the failure: one from the synchronous case, and another for the parsing of the first read from AIOFile; it's the second read from AIOFile (that should return an empty string) that generates the error.

Please confirm that you can reproduce what I see with that sample script. If you agree with the diagnose, please close this issue; I'd also suggest you either create an issue against aiofile, or switch to using aiofiles, which seems to be the more popular choice.

from ijson.

Thanks! This helped me a lot to understand the problem!

from ijson.

It's missuse of aiofile cause the AIOFile object expects the offset argument (so it's zero by default 😀). For linear-like file reading you have to use Reader helper.

The reason is the aiofile is a library to made real asyncronous file operations. Asyncronous API with builtin file position pointer doesn't known to me. So for user this abstracitons will be provided as addidional utilities.

Please double check the aiofile README, it contains a solution.

from ijson.