gitter-badger / sda-pipeline Goto Github PK

View Code? Open in Web Editor NEW

A federated storage for sensitive data, NeIC SDA

License: GNU Affero General Public License v3.0

Go 63.35% Dockerfile 0.20% Shell 10.01% HTML 9.54% JavaScript 12.43% CSS 4.48%

sda-pipeline's Introduction

sda-pipeline

sda-pipeline is part of NeIC Sensitive Data Archive and implements the components required for data submission. It can be used as part of a Federated EGA or as a isolated Sensitive Data Archive. sda-pipeline was built with support for both S3 and POSIX storage.

Deployment

Recommended provisioning method for production is:

on a , kubernetes cluster using the helm chart;

For local development/testing see instructions in dev_utils folder.

Core Components

Component	Role
intercept	The intercept service relays message between the queue provided from the federated service and local queues. (Required only for Federated EGA use case)
ingest	The ingest service accepts messages for files uploaded to the inbox, registers the files in the database with their headers, and stores them header-stripped in the archive storage.
verify	The verify service reads and decrypts ingested files from the archive storage and sends accession requests.
finalize	The finalize command accepts messages with accessionIDs for ingested files and registers them in the database.
mapper	The mapper service registers the mapping of accessionIDs (IDs for files) to datasetIDs.
sync	The sync service accepts messages with accessionIDs for ingested files and copies them to the second/backup storage.

Internal Components

Component	Role
broker	Package containing communication with Message Broker https://github.com/neicnordic/sda-mq
config	Package for managing configuration.
database	Provides functionalities for using the database, providing high level functions https://github.com/neicnordic/sda-db.
storage	Provides interface for storage areas such as a regular file system or as a S3 object store.