Minion, which achives an approoach to broadbrand traffic like BT, implemented by python and based on HTTP protocol
Minion can be easily used to dispatch data, similar to curl/wget. It also provides python API libs.
minion cli need python 2.7, requests
minion tracker need python 2.7, django mysql
git clone git://github.com/alibaba/minion
cd minion
python setup.py install
deploy your tracker server
tracker need django mysql
python tracker/manage.py syncdb
python tracker/manage.py runserver # or use wsgi/nginx
try your minion cli
minion get http://foo.bar/testfile \
--tracker some.tracker.server \
--dest-path=/tmp/tops1 \
--upload-rate 10M \
--download-rate 100M \
--callback delete \
--upload-time 5 \
--fallback
- --tracker specify tracker
- --dest-path specify the path data will be wrote to, default to current dir
- --upload-time specify time for upload after download complete, default to 60s
- --download-rate --upload-rate specify rate for upload/download can used with unit, like 10M
- --fallback option, if no resource exists in tracker download directly from origin url
- --callback
- --verbose for more log
Minion work as picture below
PEER: The host ready to download SOURCE
TRACKER: The host manage PEERs and SOURCEs
SOURCE: URL of some data
STEP
- PEER0 access TRACKER to get peers that was uploading SOURCE, TRACKER return null
- PEER0 get SOURCE from origin url directly
- PEER0 upload SOURCE, telling that to TRACKER
- PEER1 do like step 1,return a list contain PEER0
- PEER1 get SOURCE from PEER0
- PEER1 do like step 3
Minion as a whole is released under the GNU General Public License version 2.