Comments (3)
@Bubu thanks for the very good question. I have pondered about this myself in the past, I have also thought it would be nice to have -- so having someone else express interest in the idea is definitely good.
I think this should be possible, but needs some care to great care. The ijson functions actually already support generators as inputs, but those are assumed to be the lower-level generator functions of ijson itself (e.g., you can use ijson.parse() as the input to ijson.items, see the "Intercepting events" section of the README). It should still be possible to detect those separately from any other arbitrary generators and act differently though. After that it should all work, because funnily enough we internally turn file objects into generators!
Line 215 in ed2182e
The C backend might need some more extra care as well.
I can't promise anything in terms of deadlines. But like I said, I'm onboard with the idea, and if someone decides to step in and give it a crack in the meanwhile I'll be happy to review code and PRs.
from ijson.
For those coming in the future: see #58 (comment) for an (untested, personally) example of a simple file-like wrapper around a generator as a workaround.
from ijson.
For those coming in the future: see #58 (comment) for an (untested, personally) example of a simple file-like wrapper around a generator as a workaround.
Based on the above, here is what I'm using with httpx:
import httpx
import ijson
from contextlib import asynccontextmanager
from typing import AsyncIterator
class HttpxStreamAsFile:
def __init__(self, url: str):
self.url = url
self.data = None
self.client = httpx.AsyncClient()
@asynccontextmanager
async def create_stream(self) -> AsyncIterator:
try:
await self._create_stream()
yield
finally:
await self.client.aclose()
async def _create_stream(self) -> None:
req = self.client.build_request("GET", self.url)
res = await self.client.send(req, stream=True)
self.data = res.aiter_bytes()
async def read(self, n: int) -> None:
if self.data is None or n == 0:
return b""
return await anext(self.data, b"")
async def main():
url = "your-url"
httpx_as_file = HttpxStreamAsFile(
url
)
async with httpx_as_file.create_stream():
async for prefix, event, name in ijson.items(httpx_as_file):
print(prefix, event, name)
from ijson.
Related Issues (20)
- yajl2_c backend crashes on PyPy3 HOT 19
- Is there a way to recursively iterate the key? HOT 4
- ijson.items(file, prefix) waits for EOF HOT 8
- Wheels for Python 3.12 with yajl2_c backend HOT 4
- Include array index HOT 2
- Iterate over more than one prefix? HOT 2
- How to parse a large gzip json file. HOT 2
- Make new release HOT 2
- yajl2_c backend for lambda function HOT 2
- How to use ijson to covert string to dict? HOT 3
- How to read json records in chunks using ijson? HOT 4
- Question: is it possible that returing bytes instead of str could speedup parsing? HOT 3
- Thread safety HOT 9
- Full support for byte stream generator HOT 9
- Allow to use ijson package by a relative import HOT 4
- How can I most-efficiently check for a key in the top-level of a json object? HOT 3
- Python3.12 compilation error: ‘PyGenObject’ has no member named ‘gi_code’ HOT 5
- Is it possible to use isjon with Jsonl, ndjson ? HOT 5
- Memory leak on exception handling with yajl2_c backend HOT 6
- _yajl2 backend broken with Python 3.12 HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ijson.