Git Product home page Git Product logo

access_qc_generation's People

Contributors

ionox0 avatar murphycj2 avatar rhshah avatar

Stargazers

 avatar

Watchers

 avatar  avatar

access_qc_generation's Issues

toil mutex filenotfound error

Just putting this here to keep track, it appears to be sporadic and is fixed with a retry

<=========
[2021-07-01T14:58:39-0400] [Thread-261] [D] [toil.batchSystems.singleMachine] Launched job 240 as child 20646
Traceback (most recent call last):
  File "/home/accessbot/miniconda3/envs/toil_test/bin/_toil_worker", line 8, in <module>
    sys.exit(main())
  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/worker.py", line 704, in main
    with in_contexts(options.context):
  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/worker.py", line 678, in in_contexts
    with manager:
  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 494, in __enter__
    self.arena.enter()
  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/lib/threading.py", line 448, in enter
    fcntl.lockf(self.lockfileFD, fcntl.LOCK_EX) # type: ignore
  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/lib/threading.py", line 366, in global_mutex
    os.unlink(lock_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/juno/work/access/testing/users/johnsoni/access_qc/5500_HE/work/toil-mutex-cf0426ab-aa3d-4829-a54f-c264832f82aa-cleanup-arena-lock'

Write QC Generation workflow which should include:

BAM QC:
Inputs:

  • BAM
  • BED

RUN:

  • Picard CollectMultipleMetrics
  • Picard CollectHSmetrics
  • Waltz PileupMetrics
  • Waltz CountReads

Collapsed FASTQ QC:

Input:

  • Collapsed R1 FASTQ
  • Collapsed R2 FASTQ

Run

  • BASH SCRIPT CWL

CWL_UNSUPPORTED_REQUIREMENT_EXIT_CODE

...
	    self.add_file_or_directory_volume(runtime, vol, host_outdir_tgt)
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/cwltool/singularity.py", line 310, in add_file_or_directory_volume
	    shutil.copytree(volume.resolved, host_outdir_tgt)
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/shutil.py", line 318, in copytree
	    names = os.listdir(src)
	FileNotFoundError: [Errno 2] No such file or directory: '_:4694b3bc-a86a-4be4-8d99-b6876552c8a8'

	The above exception was the direct cause of the following exception:

	Traceback (most recent call last):
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/worker.py", line 400, in workerScript
	    job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/job.py", line 2382, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/job.py", line 2303, in _run
	    return self.run(fileStore)
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/cwl/cwltoil.py", line 1921, in run
	    logger=cwllogger,
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/cwltool/executors.py", line 149, in execute
	    self.run_jobs(process, job_order_object, logger, runtime_context)
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/toil/cwl/cwltoil.py", line 848, in run_jobs
	    return super().run_jobs(process, job_order_object, logger, runtime_context)
	  File "/home/accessbot/miniconda3/envs/toil_test/lib/python3.7/site-packages/cwltool/executors.py", line 267, in run_jobs
	    raise WorkflowException(str(err)) from err
	cwltool.errors.WorkflowException: [Errno 2] No such file or directory: '_:4694b3bc-a86a-4be4-8d99-b6876552c8a8'
	[2021-07-01T15:49:09-0400] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host jx04
<=========
[2021-07-01T15:54:22-0400] [MainThread] [E] [root] Your workflow uses a CWL requirement that Toil does not support!

output directory structure

  • avoid duplicate outputs (biometrics collapsed / duplex folders)
  • avoid all_qc_files/all_qc_files duplicate dir name
  • put extraneous _mqc.html files inside all_qc_files folder (preferably with the associated sequence_qc / biometrics outputs)

Fingerprint Module

Create a fingerprinting python module, probably using click from python to have multiple subcommands.

Here are things that we talked about @maysunh

Finger Printing

Things to do:

  • Genotyping
  • Major / Minor Contamination
  • Contamination across samples

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.