firebaseextended / firebase-streaming-import Goto Github PK

The only outputs from this script are the start, end and error states. If it's streaming, could you print a time estimation or some kind of progress?

I'd like to send a PR, but my python is pretty rusty.

I have made a python3 script based on your work but is very specific to my needs. You may want to make it more general.

https://github.com/anibalrivero/firestore_streaming_import

Thank you so much for providing such an awesome tool.

My issue is that the large json file I have consists of a list of jsons
[ {...} , {...} , {...} , {...} ]
And when I try to import this json file to Firebase using your tool, each item in the list gets the same key
["item" :{..}, "item" : {..}, "item" : {..} ]
instead of using indices
["0" : {..} , "1" : {..} , "2" : {..} ]

So, what happens when I use your tool is the first item in the array list gets created and then the tool just keeps changing the first item's values with values from the next item as it goes through the list

Getting File "import.py", line 60 except Exception, e: ^ everytime I try to run the import.py

When running

python import.py https://necir-hackathon.firebaseio.com/ test.json

I get this stacktrace:

started at 1469480152.98
Traceback (most recent call last):
  File "import.py", line 90, in <module>
    main(argParser.parse_args())
  File "import.py", line 20, in main
    for prefix, event, value in parser:
  File "R:\Python27\lib\site-packages\ijson\common.py", line 65, in parse
    for event, value in basic_events:
  File "R:\Python27\lib\site-packages\ijson\backends\python.py", line 185, in basic_parse
    for value in parse_value(lexer):
  File "R:\Python27\lib\site-packages\ijson\backends\python.py", line 116, in parse_value
    for event in parse_array(lexer):
  File "R:\Python27\lib\site-packages\ijson\backends\python.py", line 138, in parse_array
    for event in parse_value(lexer, symbol, pos):
  File "R:\Python27\lib\site-packages\ijson\backends\python.py", line 119, in parse_value
    for event in parse_object(lexer):
  File "R:\Python27\lib\site-packages\ijson\backends\python.py", line 160, in parse_object
    pos, symbol = next(lexer)
  File "R:\Python27\lib\site-packages\ijson\backends\python.py", line 59, in Lexer
    buf += data
MemoryError

Test.json is 484 mb, but shouldn't that not be a problem?

On running the import I get the following error

Traceback (most recent call last):
File "import.py", line 1, in
import ijson
ImportError: No module named ijson

Note: During the install I did get the following error as well
$ pip install -r requirements.txt
Collecting requests (from -r requirements.txt (line 1))
Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
100% |████████████████████████████████| 92kB 875kB/s
Collecting argparse (from -r requirements.txt (line 2))
Downloading argparse-1.4.0-py2.py3-none-any.whl
Collecting ijson (from -r requirements.txt (line 3))
Downloading ijson-2.3-py2.py3-none-any.whl
Collecting traceback (from -r requirements.txt (line 4))
Could not find a version that satisfies the requirement traceback (from -r requirements.txt (line 4)) (from versions: )
No matching distribution found for traceback (from -r requirements.txt (line 4))

I removed the traceback line and tried a re-install and then get the 2nd error
Installing collected packages: certifi, chardet, idna, urllib3, requests, argparse, ijson
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 784, in install
**kwargs
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 345, in move_wheel_files
clobber(source, lib_dir, True)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 316, in clobber
ensure_dir(destdir)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/init.py", line 83, in ensure_dir
os.makedirs(path)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/certifi-2017.11.5.dist-info'

I'm testing this out with a json file that's only 1.7 mb right now and It's been on 'start at' after running the script for about 10 minutes now. Is this normal?

I have a 861 MB database backup. I tried importing it multiple times on a clean database. I've left it for 4+ hours and overnight one time. I'm running the tool with 8 threads. 7 of the threads don't show any sings of activity and only one of them is active. Memory goes up to around 8GB and CPU to over 100% sometimes for only that one thread. The other threads stay idle. At some point network traffic stops and everything just hangs there. There is activity in memory and cpu, but no packets sent.