MemoryError when importing 484mb JSON File about firebase-streaming-import HOT 10 CLOSED

Scratch that, still leaking.

Reading the result appears to resolve the memory issue, but it slows the individuals requests by a ton. I'll leave it running overnight and see where it's at in the AM.

from firebase-streaming-import.

Same issue with me and the file size is 135MB

from firebase-streaming-import.

I'm not sure, but I think this error is due to a particular object/property
being too large. I don't how to get the Streaming Import tool to work, but
writing a custom script in Node.js ended up being far more efficient.
Here's the code I used, you can modify it to fit your needs:
https://raw.githubusercontent.com/ROODAY/NECIR-Hackathon/conversion-script/uploader.js

It converts CSV to JSON before uploading, but you could just make use of
the uploading bit if you already have JSON. Just note that in its current
state, it notifies when the last object has been uploaded to Firebase, but
the script will still need to run for a while if you are uploading a lot of
data, as it seemed that Firebase needs some more time to save the data it
receives.

from firebase-streaming-import.

Might be unrelated, but I think there's a memory leak. I'm running the import script against ~400MB json file and it's taking on the order of 10GB of RAM to complete.

from firebase-streaming-import.

Alright think I figured it out. Apparently the pp module that this is using to do the requests in parallel won't release the memory until you read the result, and reading the result significantly slows down the process.

I replaced this line with parallelJobs = pp.Server('autodetect', (), None, True), and it seemed to do the trick (the fourth param is whether to restart worker process after each task completion and defaults to false).

from firebase-streaming-import.

@lowe0292 is it still running? ;)

What's the status of this repository?
I see no updates for 4 years, and my uploads never finish.

from firebase-streaming-import.

@philshem never solved it and gave up, Sorry!

from firebase-streaming-import.

@philshem Did you try the uploader script I made? I haven't touched it since I made it but it worked at the time, might help ya here.

from firebase-streaming-import.

@ROODAY I saw it but didn't use it because simpler than editing the script was for me to splitting my BIG json in 10 smaller ones, and looping around firebase-import.

from firebase-streaming-import.

Closing as there are posted workarounds above, feel free to re-open if the bug is being worked on!

from firebase-streaming-import.