sjoerdk / idis Goto Github PK

Is your feature request related to a problem? Please describe.

• Check pipeline status. There is a study in /errored, which usually means the anonymization job indicates error status
• What went wrong with this job? It is possible to find this in the logs, and possible in principle to find this on the anonymization server by looking at all jobs created. But it is not easy.

Describe the solution you'd like
For any study in the pipeline it should be easy to get the job ID associated with it. Either through the web interface or via some CLI admin command

CI is still in Travis, but Travis has turned sad some time ago https://blog.travis-ci.com/2019-01-23-travis-ci-joins-idera-inc.

Move to github actions

Describe the bug
See title

To Reproduce

• send a study to a stream. It goes through the pipeline and ends up in the finished folder.
• send the same study again
  This will cause

raise StudyPushException(e)
idissend.core.StudyPushException: Destination path '/root/stages/stage/stream/studyfolder' already exists

Expected behavior
Not sure. Possibly merging the two folders? Definitly not a hard crash.

Is your feature request related to a problem? Please describe.
To be able to implement a solution to #8, the pipeline system needs to be based on django models. Currently it is pure python. Convert.

Currently these are rather deep python structures. See for example a SafePrivateDefinition:

safe_private = SafePrivateDefinition(
    blocks=[
        SafePrivateBlock(
            tags=[
                "0023[SIEMENS MED SP DXMG WH AWS 1]10",
                "0023[SIEMENS MED SP DXMG WH AWS 1]11",
                "00b1[TestCreator]01",
                "00b1[TestCreator]02",
            ],
            criterion=lambda x: x.Modality == "CT",
            comment="Some test tags, only valid for CT datasets",
        ),
        SafePrivateBlock(
            tags=["00b1[othercreator]11", "00b1[othercreator]12"],
            comment="Some more test tags, without a criterion",
        ),
    ]
)

This is hard to read and edit, easy to get wrong. Make this more like this:

if Modality == "CT"  # Some test tags, only valid for CT datasets
    "0023[SIEMENS MED SP DXMG WH AWS 1]10",
    "0023[SIEMENS MED SP DXMG WH AWS 1]11",
    "00b1[TestCreator]01",
    "00b1[TestCreator]02",

"00b1[othercreator]11"
"00b1[othercreator]12"

Is your feature request related to a problem? Please describe.

• There is pending data for which an IDIS job has been made. The data is in the pending folder.
• This job get an error status for some reason
• The pending data is moved to 'errored' in pipeline which moves it to the errored folder
  Now I'd like to retry the IDIS job. However, this will fail, as the source data is no longer in pending
  I can move the data back, but the pipeline will move it back to errored as soon as it finds error status in IDIS

Describe the solution you'd like
Both IDIS and pipeline need to be reset at the same time.
There should be some command for this somewhere in the pipeline. The pipeline can reset the IDIS job and move the data back simultaneously.
I'm unsure of where to put his command though. In the django website? as a CLI command? as a django admin command?

When a study has been anonymized, send a post request to post-processing reveivers

Is your feature request related to a problem? Please describe.
For full problem and solution see sjoerdk/research-bureau-roadmap#5

It should be possible to deactivate a datastream without completely deleting it.
It should be possible to indicate whether a datastream is active or inactive

Describe the bug
Exception in one stream will halt processing of all streams

To Reproduce
Steps to reproduce the behavior:

• Send a study to a stream, make sure something goes wrong so the folder for that study remains
• Send the same study again (will cause exception due to folder already existing)
• Now streams without a problem will not be processed

Expected behavior
Exceptions in one stream should be logged, but processing of other streams should continue