Git Product home page Git Product logo

provenance-service's Introduction

Provenance Service

Current directory contains:

  • provenance-service implementation in src/prov folder

VERSION: 0.2

Docker container

Using the Graph Manager Service Docker container:

  • docker pull attxproject/provenance-service:dev in the current folder;
  • running the container docker run -d -p 7030:7030 attxproject/provenance-service:dev runs the container in detached mode on the 7030 port (production version should have this port hidden);
  • using the endpoints http://localhost:7030/{versionNb}/{endpoint} or the other listed below.

The version number is specified in src/prov/app.py under version variable.

Overview

The Provenance Service registers provenance Document in the Graph Store either via the Message Broker or the REST API. Processing the provenance related information is done asyncronously.

Full information on how to run and work with the Provenance Service available at: https://attx-project.github.io/Service-Provenance.html

API Endpoints

The Graph Manager REST API has the following endpoints:

  • graph - interface to the Fuseki Graph Store;
  • prov - API for registering and retriving provenance;
  • health - checks if the application is running.

Develop

Requirements

  1. Python 2.7
  2. Gradle 3.0+ https://gradle.org/
  3. Pypi Ivy repository either a local one (see https://github.com/linkedin/pygradle/blob/master/docs/pivy-importer.md for more information) or you can deploy your own version using https://github.com/blankdots/ivy-pypi-repo

Building the Artifact with Gradle

Install gradle. The tasks available are listed below:

  • do clean build: gradle clean build
  • see tasks: gradle tasks --all and depenencies gradle depenencies
  • see test coverage gradle pytest coverage it will generate a html report in htmlcov

Run without Gradle

To run the server, please execute the following (preferably in a virtual environment):

pip install -r pinned.txt
python src/prov/provservice.py server
python src/prov/provservice.py queue
python src/prov/provservice.py consumer

For testing purposes the application requires a running Fuseki, RabbitMQ. Also the health endpoint provides information on running services the service has detected: http://localhost:7030/health

The Swagger definition lives here:swagger-provservice.yml.

provenance-service's People

Contributors

attx-github avatar blankdots avatar jkesanie avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

blankdots

provenance-service's Issues

ProvenanceMessage: Document provenance message structure

Description

We have at least two use cases: message based protocol that creates little bit of provenance and the REST api case that can be used to create more complex documents. This means that we might need to have minimum structure and full structure.

DoD

JSON schema.

Testing

prov-service: Add integration tests

Description

Describe the issue or task at hand with dependencies and possible links to code/external/internal sources.

DoD

Itemized things or artifacts of the task at hand, what one expects to achieve - tangible output.

Testing

How one would test that the things/artifacts have been achieved. Options (Peer review, Unit Test, BDD testing, Integration Test).

Prototype CI/CD pipeline with Kontena

Description

Test and document the setup of a CI/CD pipeline with Kontena.

DoD

The basic use should be running an ATTX container triggered by Jenkins/Git. Preferably, we should be able to run our tests as well.

Testing

Manual testing, peer review.

Unit tests for provenance service

Description

Describe the issue or task at hand with dependencies and possible links to code/external/internal sources.

DoD

Itemized things or artifacts of the task at hand, what one expects to achieve - tangible output.

Testing

How one would test that the things/artifacts have been achieved. Options (Peer review, Unit Test, BDD testing, Integration Test).

Setup environment for Integration testing

Description

Describe the issue or task at hand with dependencies and possible links to code/external/internal sources.

DoD

Itemized things or artifacts of the task at hand, what one expects to achieve - tangible output.

Testing

How one would test that the things/artifacts have been achieved. Options (Peer review, Unit Test, BDD testing, Integration Test).

Setup Camel based routing and queues for provenance services

Description

Provenance service should intercept every message that goes through the activeMQ. This can be implemented using Camel routes and provenance specific queues that are consumed by the ProvenanceService.

This implementation should allows us to get the communication related provenance data.

DoD

Working queues and routing.

Testing

Skip it for now.

Package provenance service

Description

DoD

Set of images that allow the stack to register, process and store provenance data through REST and MQ apis.

Testing

Unit and integration tests

Testing provenance service in a distributed environment

Description

Testing provenance service in Kontena cloud and cPouta (Swarm). Run mock message provider and provenance service in order to test HA capabilities of platforms and the message persistance.

For example: Is the infra able to automatically start up a new instance of ProvService if we delete the old container manually? Are the messages sent during the absence of the service preserved? Load balancing?

DoD

Report.

Testing

Handling/logging message related errors with messagebroker

Description

Current implementation leaves the invalid messages to the queue by not acknowledging them and that causes weird errors and would require manual purging of the messages.

Possible solutions:

  • Error queues
  • acknowledge and raise error
  • can validation be implemented in the broker?
  • how are the messages in the error queue processed?
  • routing based implementation (Camel?)

DoD

Better error handling for services

Testing

Integration testing

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.