neicnordic / bifrost Goto Github PK

Describe pain points or successes here until completion.

Local EGA has been suggested to be used in the schizophrenia implementation plan (point 10) as well as the imputation implementation plan (point 11) for the final full scale version. This issue will serve as a placeholder until we can look closer at this issue.

The splitByChromosome.sh script was added because the imputation server expects the input files to only contain a single chromosome, and as the name reveals, the script splits VCF files into separate chromosomes.

The input files for the imputation project are sensitive and should thus be encrypted before transfer to TSD is initiated. Currently the idea is to use the crypt4gh tool for this.

Document everything before moving on to the next milestone.

After the files have been successfully transferred to TSD they need to be moved from the /tsd/pXXX/data/durable/... disk to the disk where the compute will happen.
This will be handled by a cron job that will trigger a script that will handle the task.
Below is a preliminary outline of what steps needs to be done.

• Make a cron job that runs a script that does the following:
• Verify that files have been transferred successfully by verifying md5sum
• Initiate file transfer of input files and yaml file to uniquely named directory
• Log these events in a log file

And also write documentation when this is done:

• Write documentation

Describe pain points or success here.

The plan is to use ARC to submit queries in a way that would make it possible to do computations on several of the sensitive compute infrastructures within one query.

ARC can delegate jobs to the correct sensitive compute infrastructure and submit the job to the clusters and collect the results.

ARC can also handle queuing and resource management.

How this would work in practice needs to be discussed.

The known obstacles at the moment are with authentication/authorization and privilege management.

Describe pain points or successes here until completion.

I suggest we use the yaml file format to make the job config file, it is readable and has a simple syntax.

I suggest that the config file should contain the following:

• Job type: imputation or whatever kind of analysis that the schizophrenia project does
• Input file md5sum: "45f6esf435se4f56w", this is a mechanism to verify that the transferred files have been transferred correctly. This solves both the issue of incomplete transfers as well as files corrupted during the transfer.

What more should it contain? Is there a better solution to the incomplete/corrupted files issue?

Document everything before moving on to the next milestone.

What input data do we need per project?

Imputation project:
VCF files.

Schizophrenia project:
13/03 - No input data from the outside, only query file with parameters.

This is all in progress, will close once enough documentation has been written.

• Verify file integrity
• Start job
• Log these events
• Write documentation

When I run the readConfig.py script on an encrypted "schizophrenia" input file it prints the error message No supported encryption method and continues with the rest of the script. This error is not printed when running an imputation job.

Describe pain points or success here.

Describe pain points or success here.

The job submit script has been created now, it needs to be edited with all of the schizophrenia related code. Currently it has some placeholders.

When the input files have been transferred to TSD they need to be moved to a scratch disk of some sort, as of right now it is not known where this will be.

Automatically move your cards to the right place based on the status and activity of your issues and pull requests.

Describe pain points or successes here until completion.

Access to the compute VM (rhel7.tsd.usit.no) has not been granted yet, this needs to be done.

Document all changes and issues related to the config file reading script here.

Document everything before finishing and closing this milestone.