Working with Redis Queues: Queueing jobs and processing them in the background with concurrent workers

Using any language and data-store of your choice, write an application that reads a set of photos from a network store (S3), parses the EXIF data from the photos and indexes the EXIF key/value pairs into a query-able store by unique photo.

• redis

update settings.py file for all confugurable parameters

• virtualenv waldoenv
• source waldoenv/bin/activate
• pip install -r requirements.txt

Start multiple workers based on the configurations of your system

• source waldoenv/bin/activate
• rq worker --exception-handler exceptionhandler.retry_handler download_queue read_exif_queue store_data_in_db_queue
• rq worker --exception-handler exceptionhandler.retry_handler download_queue

• source waldoenv/bin/activate
• python imagedownloader.py

• tail -f waldophotos.log

Start rq-dashboard and open http://localhost:9181 in browser

• rq-dashboard

• S3 Bucket (input): http://s3.amazonaws.com/waldo-recruiting
• RQ (Redis Queue) http://python-rq.org/

• As the Exif info is present in header, no need to download complete image
• Improved downloading of image by continuing from the previous fail point