phumpal / multi-cloud-mirror Goto Github PK

Not sure if anyone reads this, but this is compatible with the current latest 
versions of boto and python-cloudfiles

easy_install python-cloudfiles==1.7.10
easy_install boto==2.5.2

For use on rightscale, you must clone the server template, download a new 
Multi-CloudImage (I'm using Ubuntu 10.04 EBS.) and add that as the default 
image in your cloned server template.

What steps will reproduce the problem?
1. Start the sync on a relatively large bucket (in my case that was about 10 
million files)
2. After some time, the script simply dies:

ubuntu@ip-10-86-150-235:~/bin$ PYTHONUNBUFFERED=1 
./2011-10-14-multi_cloud_mirror.py --debug 1 's3://mybucket->cf://mybucket' > 
/mnt/tmp/s3-to-cf.log
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 302, in _handle_workers
    pool._maintain_pool()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 206, in _maintain_pool
    self._repopulate_pool()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 199, in _repopulate_pool
    w.start()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 120, in __init__
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory 

It works fine on smaller buckets.

Apparently the script tries to built the complete sync scenario before actually 
doing anything - not really how a sync tool should work.