This is a small utility to help pull the data from the Learning Registry into a datastore of your choice.

LR-Data requires:

• RabbitMQ
• Redis
• Python
• Celery

On OS X, you will also need libevent, which can be installed with homebrew: brew install libevent

sudo apt-get update
sudo apt-get install -y virtualenv python-dev libevent-dev libxml2-dev libxmlsec1-dev

# Create virtual environment
virtualenv env
. env/bin/activate
# Install python library requirements
pip install -U -r requirements.txt

All configuration is done in the src/celeryconfig.py file. For information of configuring Celery please see their document. For lr-data configuration modify

config = {

	"lrUrl": "http://lrdev02.learningregistry.org/harvest/listrecords",

	"couchdb":{

		"dbUrl":"http://localhost:5984/lr-data"

	},
	"tasks": {
		"insert": "tasks.save.insertDocumentMongo",

		"validate": "tasks.validate.emptyValidate",
	}

	"redis":{

		"host":"localhost",

		"port":6379,

		"db":0
	}

	"mongodb":{

		"database":"lr",

		"collection":"envelope",

		"host": "localhost",

		"port": 27017,

	},
}

Customizable tasks are defined in the tasks hash. validate is the task name for validating incoming docs. insert is the task you wish to use to save the data

There are scripts inside of src to get you started.

• start_celery_workers.sh - will start your default worker threads responsible for harvesting, validating, and saving LR data
• stop_celery_workers.sh - stops workers
• start_harvesting.py - sends harvest request into queue for workers to start working

To start harvesting, activate your virtualenv, start the celery workers, then start harvesting request

cd src
. ../env/bin/activate
./start_celery_workers.sh
./start_harvesting.py

Should you want to stop (or pause) the processing you can stop the celery workers:

./stop_celery_workers.sh

When you want to resume processing, you need only start the celery workers again:

./start_celery_workers.sh