Working with Redis Queues: Queueing jobs and processing them in the background with concurrent workers
Using any language and data-store of your choice, write an application that reads a set of photos from a network store (S3), parses the EXIF data from the photos and indexes the EXIF key/value pairs into a query-able store by unique photo.
- redis
update settings.py file for all confugurable parameters
- virtualenv waldoenv
- source waldoenv/bin/activate
- pip install -r requirements.txt
Start multiple workers based on the configurations of your system
- source waldoenv/bin/activate
- rq worker --exception-handler exceptionhandler.retry_handler download_queue read_exif_queue store_data_in_db_queue
- rq worker --exception-handler exceptionhandler.retry_handler download_queue
- source waldoenv/bin/activate
- python imagedownloader.py
- tail -f waldophotos.log
Start rq-dashboard and open http://localhost:9181 in browser
- rq-dashboard
- S3 Bucket (input): http://s3.amazonaws.com/waldo-recruiting
- RQ (Redis Queue) http://python-rq.org/
- As the Exif info is present in header, no need to download complete image
- Improved downloading of image by continuing from the previous fail point