neicnordic / localega Goto Github PK

Expected Behavior

Unit tests failed

Current Behavior

=========================================================================================== ERRORS ===========================================================================================
_________________________________________________________________________ ERROR collecting tests/unit/test_ingest.py _________________________________________________________________________
ImportError while importing test module '/home/wahbi/LocalEGA/tests/unit/test_ingest.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/unit/test_ingest.py:2: in
from lega.ingest import main, work
lega/ingest.py:24: in
from legacryptor.crypt4gh import get_header
.tox/unit_tests/lib/python3.7/site-packages/legacryptor/crypt4gh.py:16: in
import pgpy # Simply used to encrypt/decrypt the records, all in memory.
.tox/unit_tests/lib/python3.7/site-packages/pgpy/init.py:5: in
from .pgp import PGPKey
.tox/unit_tests/lib/python3.7/site-packages/pgpy/pgp.py:45: in
from .packet import Key
.tox/unit_tests/lib/python3.7/site-packages/pgpy/packet/init.py:9: in
from .packets import * # NOQA
.tox/unit_tests/lib/python3.7/site-packages/pgpy/packet/packets.py:18: in
from .fields import DSAPriv, DSAPub, DSASignature
.tox/unit_tests/lib/python3.7/site-packages/pgpy/packet/fields.py:30: in
from cryptography.hazmat.primitives.asymmetric import ed25519
E ImportError: cannot import name 'ed25519' from 'cryptography.hazmat.primitives.asymmetric' (/home/wahbi/LocalEGA/.tox/unit_tests/lib/python3.7/site-packages/cryptography/hazmat/primitives/asymmetric/init.py)
====================================================================================== 1 error in 0.32s ======================================================================================
ERROR: InvocationError for command /home/wahbi/LocalEGA/.tox/unit_tests/bin/coverage run -m pytest -x tests/unit (exited with code 1)
__________________________________________________________________________________________ summary ___________________________________________________________________________________________
flake8: commands succeeded
ERROR: unit_tests: commands failed

Possible Solution

Steps to Reproduce

Context (Environment)

==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.3, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
cachedir: .tox/unit_tests/.pytest_cache
rootdir: /home/wahbi/LocalEGA
plugins: cov-2.8.1
collected 29 items / 1 errors / 28 selected

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce

Context (Environment)

Description

We should probably mention db migration in docs and help new developers.

Description

verify, ingest and maybe finalizer components failed to start when configured via ENVs. Puting the same config in a file worked so that's why I went that way for M7.
It mainly boils down to the fact that the configs for RabbitMQ changed to use a URL format which is a p.i.t.a. to work with.

Definition of Done

Env vars are usable in deployments as well as config files.

How to test

Integration tests pass and e2e test passes

Expected Behavior

Data in should work for any username, including usernames of Elixir type: [email protected].

Current Behavior

Due to "sanitizing", finalization fails.
Ingestion: user_id = sanitize_user_id(data['user'])
Finalization: user = data['user'].
So we get different users: just dummy in ingestion and [email protected] in finalization. And because of that finalization doesn't work, since it tries to update the database where user is [email protected], but it's actually dummy.

Possible Solution

According to @silverdaz, there's no need to sanitize the username anymore. So we need to remove it from the ingestion.

Steps to Reproduce

Try to perform data in with username like [email protected]. Observe that finalization doesn't work properly.

Description

I realized that one of the messages published internally (to the archived queue) contains the file header blob. This is not very secure (in case the broker gets compromised somehow, this message might get leaked), and also there's no practical need for this: headers are stored in the database anyway, so all the microservices have direct access to them.

Proposed solution

Remove the header blob from the corresponding messages.

Definition of Done

Make sure that messages that are circulated in the RabbitMQ (even internally) don't contain the header blobs.

Description

after building all containers via docker-compose up -d , the user does not know how to use the system , the documentation in this link https://localega.readthedocs.io/en/latest/ is not clear enough .

the questions are :
how to add a user from cega ? does cega container have a web interface ?
how to upload a file ?
how to download a file ?

Proposed solution

the answers here will help me in implementing a local ega in tunisia , thank you very much

Expected Behavior

Unit tests and Integration tests broken

Current Behavior

Unit tests and integration tests fail.

Possible Solution

Fix python packages to a version that worked.
Updating source to crypt4gh new version.

Steps to Reproduce

Unit Tests:

1. tox -r in root directory

=========================================================================================== ERRORS ===========================================================================================
_________________________________________________________________________ ERROR collecting tests/unit/test_ingest.py _________________________________________________________________________
ImportError while importing test module '/home/stenegru/csc/dev/neic-LocalEGA/tests/unit/test_ingest.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/unit/test_ingest.py:2: in <module>
    from lega.ingest import main, work
lega/ingest.py:24: in <module>
    from legacryptor.crypt4gh import get_header
.tox/unit_tests/lib/python3.7/site-packages/legacryptor/crypt4gh.py:16: in <module>
    import pgpy # Simply used to encrypt/decrypt the records, all in memory.
.tox/unit_tests/lib/python3.7/site-packages/pgpy/__init__.py:5: in <module>
    from .pgp import PGPKey
.tox/unit_tests/lib/python3.7/site-packages/pgpy/pgp.py:45: in <module>
    from .packet import Key
.tox/unit_tests/lib/python3.7/site-packages/pgpy/packet/__init__.py:9: in <module>
    from .packets import *  # NOQA
.tox/unit_tests/lib/python3.7/site-packages/pgpy/packet/packets.py:18: in <module>
    from .fields import DSAPriv, DSAPub, DSASignature
.tox/unit_tests/lib/python3.7/site-packages/pgpy/packet/fields.py:30: in <module>
    from cryptography.hazmat.primitives.asymmetric import ed25519
E   ImportError: cannot import name 'ed25519' from 'cryptography.hazmat.primitives.asymmetric' (/home/stenegru/csc/dev/neic-LocalEGA/.tox/unit_tests/lib/python3.7/site-packages/cryptography/hazmat/primitives/asymmetric/__init__.py)
====================================================================================== 1 error in 0.52s ======================================================================================
ERROR: InvocationError for command '/home/stenegru/csc/dev/neic-LocalEGA/.tox/unit_tests/bin/coverage run -m pytest -x tests/unit' (exited with code 1)

Integration tests:

1. bats integration after the system has been bootstraped - see https://travis-ci.org/neicnordic/LocalEGA/jobs/591996474

Context (Environment)

• Travis
• local Development

Related:

EGA-archive#78

Description

We are going to use Apache Mina Inbox by default

Definition of Done

Change Integration tests and deployments to work with Apache Mina Inbox

How to test

Integration tests pass

Description

Bucket name variable needs to be part of the S3 class or the users will be forced to use the default bucket name.

Definition of Done

Add bucket name variable to S3 storage class.

How to test

Tests passed.

Description

See #29 (review)

Definition of Done

make the exchanges configurable

How to test

integration tests pass

Description

Logstash logging may be broken in Python microservices.

Definition of Done

Check logging whether LocalEGA Python microservices can work with ELK. Fix if not.

How to test

Run the int test and observe logs in Kibana.

DoD
Investigate bringing back ELK and with specific profiles for ELK.

Consider Graphana as an option.

Description

We need to be able to add accession md5 in order to receive stableIDs from CEGA

Definition of Done

Adding md5 and sha256 when sending message from verify service

How to test

Able to get stableID from CEGA.

Description

We need to use different docker respository so that we don't interfere with EGA-archive

Definition of Done

hooks changed for docker and new images avaiable on nbisweden docker hub

How to test

Peer review.

Expected Behavior

When uploading a file, files in the archive that had the same name in the inbox should not be disabled.

Current Behavior

The already archived file gets disabled.

This disabling happens somewhere in ingestion/verify or finalize.

Possible Solution

Steps to Reproduce

1. Upload a file with name
2. Ingest it into the archive and make it ready
3. Upload a file with the same name
4. Ingest it into the archive and make it ready

The first file is disabled.

Context (Environment)

Description

Default config.ini is out of date we need to update it, or remove it the requirement to have it that specific file.
We should keep the read of the file.

Split config file per microservice

Definition of Done

Each microservice has its own separate configuration files but Env vars can still be used

How to test

Integration tests pass as well as e2e test.

Description

Currently archive paths are vague numbers which provide no useful information from the file. Having a checksum would be quite handy for the outgestion process.

Proposed solution

Replace sequential ids with checksums of the files + some metadata (?).