This repo contains the data pipeline responsible for deriving GP2GP metrics from data sources such as NMS.

Run the following query for the desired time range and save the results as a csv.

index="spine2vfmmonitor" service="gp2gp" logReference="MPS0053d"
| table _time, conversationID, GUID, interactionID, messageSender, messageRecipient, messageRef, jdiEvent, toSystem, fromSystem

Common development workflows are defined in the tasks script.

These generally work by running a command in a virtual environment configured via tox.ini.

• Python 3.9
• Tox pip install tox
• Docker

./tasks test

./tasks e2e-test

./tasks validate

This will run the validation commands in the same container used by the GoCD pipeline.

./tasks dojo-validate

./tasks format

./tasks check-deps

If you need ODS codes and names of all active GP practices please check this out.

This pipeline will derive GP2GP metrics and metadata for practices produced by the ODS Portal Pipeline. It does this by performing a number of transformations on GP2GP messages provided by NMS.

The following examples show how to run this pipeline. Run platform-metrics-pipeline --help for usage details.

Example: platform-metrics-pipeline --month 1 --year 2021 --organisation-list-file "data/organisation-list.json" --input-files "data/jan.csv.gz,data/feb.csv.gz" --output-directory "data"

Example: platform-metrics-pipeline --month 1 --year 2021 --organisation-list-file "data/organisation-list.json" --input-files "data/jan.csv.gz,data/feb.csv.gz" --output-bucket "example-bucket"

• When outputting to AWS ensure the environment has the appropriate access.
• Note this will use the year and month as part of the s3 key structure, as well 'v2' (data pipeline output version).

ERROR: InvocationError for command /YOUR_PROJECT_DIRECTORY/prm-gp2gp-data-pipeline/.tox/check-format/bin/black --check -t py38 -l100 src/ tests/ setup.py (exited with code 2)

If you see this error, you need to delete the .tox package and try again