Comments (7)
@faker09 you'll need to provide more details, otherwise it's nearly impossible to give any advice. Also look if any ijson questions in StackOverflow cover what you need.
from ijson.
because my json file is too large(8GB), i wanna use ijson.items to read this file.
below example
{"datum":{"com.bbn.tc.schema.avro.cdm20.Event":{"uuid":"A.-\u0003\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","sequence":null,"type":"EVENT_CREATE_OBJECT","threadId":{"int":3056},"subject":{"com.bbn.tc.schema.avro.cdm20.UUID":"u*N\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"predicateObject":{"com.bbn.tc.schema.avro.cdm20.UUID":">\"-\u0003\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"predicateObjectPath":{"string":"C:\\Users\\admin\\Documents\\Documents"},"predicateObject2":null,"predicateObject2Path":null,"timestampNanos":1557272968456297200,"names":null,"parameters":null,"location":null,"size":null,"programPoint":null,"properties":{"map":{"HasMacro":"4"}}}},"CDMVersion":"20","type":"RECORD_HOST","hostId":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","sessionNumber":0,"source":"SOURCE_WINDOWS_MARPLE"}
{"datum":{"com.bbn.tc.schema.avro.cdm20.Event":{"uuid":"B.-\u0003\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","sequence":null,"type":"EVENT_OTHER","threadId":{"int":3056},"subject":{"com.bbn.tc.schema.avro.cdm20.UUID":"u*N\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"predicateObject":{"com.bbn.tc.schema.avro.cdm20.UUID":">\"-\u0003\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"predicateObjectPath":{"string":"C:\\Users\\admin\\Documents\\Documents"},"predicateObject2":null,"predicateObject2Path":null,"timestampNanos":1557272968456415900,"names":{"array":["FileIoClose"]},"parameters":null,"location":null,"size":null,"programPoint":null,"properties":null}},"CDMVersion":"20","type":"RECORD_HOST","hostId":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","sessionNumber":0,"source":"SOURCE_WINDOWS_MARPLE"}
{"datum":{"com.bbn.tc.schema.avro.cdm20.Event":{"uuid":"C.-\u0003\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","sequence":null,"type":"EVENT_CREATE_OBJECT","threadId":{"int":3056},"subject":{"com.bbn.tc.schema.avro.cdm20.UUID":"u*N\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"predicateObject":{"com.bbn.tc.schema.avro.cdm20.UUID":">\"-\u0003\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"predicateObjectPath":{"string":"C:\\Users\\admin\\Documents\\Documents"},"predicateObject2":null,"predicateObject2Path":null,"timestampNanos":1557272968456469400,"names":null,"parameters":null,"location":null,"size":null,"programPoint":null,"properties":{"map":{"HasMacro":"4"}}}},"CDMVersion":"20","type":"RECORD_HOST","hostId":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000","sessionNumber":0,"source":"SOURCE_WINDOWS_MARPLE"}
from ijson.
i means that if i use
data_json=ijson.items(filename, 'item')
for i in data_json:
print(i)
it don't output anything
from ijson.
@faker09 the problem is you are passing ijson the filename instead of a file object. You need to open your file yourself, then give the resulting file object to ijson.
from ijson.
@ faker09问题是您要通过ijson传递文件名而不是文件对象。您需要自己打开文件,然后将生成的文件对象提供给ijson。
sorry, i use a confusing parameter name. actually, i opened a file and pass a file object to ijson.items()
from ijson.
@faker09 can you please then provide a cleaner extract of your JSON file? In particular, how does the file start? I get the impression you have a file with multiple top-level JSON objects instead of a single one, in which case you'll need to use an empty prefix and multiple_values=True
when invoking ijson.items
. But again, if you provide a cleaner extract of the JSON file (make sure it formats correctly when putting it into the comments here), the code you are using, and any error you might be receiving, it would be better.
from ijson.
Closing for lack of clearer explanations.
from ijson.
Related Issues (20)
- yajl2_c backend crashes on PyPy3 HOT 19
- Is there a way to recursively iterate the key? HOT 4
- ijson.items(file, prefix) waits for EOF HOT 8
- Wheels for Python 3.12 with yajl2_c backend HOT 4
- Include array index HOT 2
- Iterate over more than one prefix? HOT 2
- How to parse a large gzip json file. HOT 2
- Make new release HOT 2
- yajl2_c backend for lambda function HOT 2
- How to use ijson to covert string to dict? HOT 3
- How to read json records in chunks using ijson? HOT 4
- Question: is it possible that returing bytes instead of str could speedup parsing? HOT 3
- Thread safety HOT 9
- Full support for byte stream generator HOT 9
- Allow to use ijson package by a relative import HOT 4
- How can I most-efficiently check for a key in the top-level of a json object? HOT 3
- Python3.12 compilation error: ‘PyGenObject’ has no member named ‘gi_code’ HOT 5
- Is it possible to use isjon with Jsonl, ndjson ? HOT 5
- Memory leak on exception handling with yajl2_c backend HOT 6
- _yajl2 backend broken with Python 3.12 HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ijson.