Plugins to work in conjunction with collectors for the alphagov/backdrop system.

An example configuration is given at the end of this document.

The basic form of a collector is like so:

documents = get_documents_from_somewhere()

... magic happens ...

send_documents_to_backdrop(documents)

This package is a part of the ... magic happens .... To extend it, add your code in the plugins directory, import the relevant symbols in the __init__.py and send a pull request.

In the backdrop-ga-collector, if plugins are used, the final plugin must be ComputeIdFrom at the moment, and this is likely to be true for a while.

This is because only the person writing plugin sequence can really know what makes records unique.

This module stands alone and is pip installable, but not currently on PyPi.

Indirection is painful and the tests are small, so they live in the same module as the code they are testing. Nosetests find them by virtual of the --all-modules configuration option. If anyone disagrees strongly with this arrangement, a pull request is welcome to fix it.

Tests are run on travis-ci.

Plugins are arbitrary python expressions, which are evaluated in the backdrop.collector.plugins namespace. They return a callable which accepts a list of dictionaries and returns a list of dictionaries.

Plugins can do whatever they like to those dictionaries and the resulting list, including modifying them in place or discarding some. This allows for aggregation and discarding records.

The plugin loading mechanism is very simple and if in doubt you should take a look at how it and other example plugins work.

Computes a department field based on the contents of the given variable name.

Recomputes the _id and humanId fields from the specified

Fills the varname key with the rank of the document, starting from 1.

Useful key for filling "top N" tables.

Delete the given keys from all documents.

Aggregates the given values according to the given aggregation_func where aggregation_func might be aggregate_count('varname') which sums 'varname', or aggregate_rate('rate_varname', 'count_varname') which can be used to combine records where 'rate_varname' represent a rate (e.g, a bounce rate), weighted according to an appropriate 'count_varname'

Ignores its arguments, useful for putting comments into the list of plugins to explain what they are doing.

{
  "dataType": "content_dashboard_visitors_count",
  "query": {
    "id": "ga:123456",
    "metrics": [
      "visitors"
    ],
    "dimensions": [
      "customVarValue9"
    ]
  },
  "target": {
    "url": "http://localhost:3039/data/foo/visitors-count",
    "token": "foo-bearer-token"
  },
  "plugins": [
    "ComputeDepartmentKey('customVarValue9')",
    "Comment('customVarValue9 must be removed from the document before aggregation')",
    "RemoveKey('_id', 'humanId', 'customVarValue9')",
    "AggregateKey(aggregate_count('visitors'))",
    "ComputeIdFrom('_timestamp', 'timeSpan', 'dataType', 'department')"
  ]
}