microsoft / genomicsnotebook Goto Github PK

Jupyter Notebooks on Azure for Genomics Data Analysis

License: MIT License

Jupyter Notebook 98.93% TSQL 1.07%

genomicsnotebook's Introduction

Genomics Data Analysis with Jupyter Notebooks on Azure

Jupyter notebook is a great tool for data scientists who are working on genomics data analysis. In this repo, we demonstrate the use of Azure Notebooks for genomics data analysis via GATK, Picard, Bioconductor and Python libraries.

3/31/2022: NEW DEMO VIDEO: How to use 'genomicsnotebook' repo in GitHub Codespaces?

For more information about Codespaces please visit the product page

Here is the list of sample notebooks on this repo:

genomics.ipynb: Analysis from 'uBAM' to 'structured data table' analysis.
genomicsML.ipynb: Train Machine Learning models with Genomics + Clinical Data
genomics-platinum-genomes.ipynb: Accessing Illumina Platinum Genomes data from Azure Open Datasets* and to make initial data analysis.
genomics-reference-genomes.ipynb: Accessing reference genomes from Azure Open Datasets*
genomics-clinvar.ipynb: Accessing ClinVar data from Azure Open Datasets*
genomics-giab.ipynb: Accessing Genome in a Bottle data from Azure Open Datasets*
SnpEff.ipynb: Accessing SnpEff databases from Azure Open Datasets*
1000 Genomes.ipynb: Accessing 1000 Genomes dataset from Azure Open Datasets*
GATKResourceBundle.ipynb: Accessing GATK resource bundle from Azure Open Datasets*
ENCODE.ipynb: Accessing ENCODE dataset from Azure Open Datasets*
genomics-OpenCRAVAT.ipynb: Accessing OpenCRAVAT dataset from Azure Open Datasets and deploy built-in Azure Data Science VM for OpenCRAVAT*
Bioconductor.ipynb: Pulling Bioconductor Docker image from Microsoft Container Registry
simtotable.ipynb: Simulate NGS data, use Cromwell on Azure OR Microsoft Genomics service for secondary analysis and convert the gVCF data to a structured data table.
igv_jupyter_extension_sample.ipynb: Download sample VCF file from Azure Open Datasets and use igv-jupyter extension on Jupyter Lab environment.
radiogenomics.ipynb: Combine DICOM, VCF and gene expression data for patient segmentation analysis.
fhir+PacBio.ipynb: Convert Synthetic FHIR and PacBio VCF Data to parquet and Explore with Azure Synapse Analytics
fhir-vcf-clustering.ipynb: Convert Synthetic FHIR and PacBio VCF Data to parquet and Explore with Azure Synapse Analytics

*Technical note: Explore Azure Genomics Data Lake with Azure Storage Explorer

1. Prerequisites

Create and manage Azure Machine Learning workspaces in the Azure portal

For further details on creation of Azure ML workspace please visit this page.

Run the notebook in your workspace

This chapter uses the cloud notebook server in your workspace for an install-free and pre-configured experience. Use your own environment if you prefer to have control over your environment, packages and dependencies.

Follow along with this video or use the detailed steps below to clone and run the tutorial from your workspace.

2. Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

3. References

genomicsnotebook's People

Contributors

Stargazers

Watchers

genomicsnotebook's Issues

Description on simulated clinical and phenotypic datasets referred in `/sample-notebooks /genomicsML.ipynb`

Thank you very much for sharing an informative set of Jupyter Notebooks.

I've been reviewing the Train Machine Learning Models with Genomics + Clinical Data notebook that uses simulated clinical and phenotypic datasets. However, I couldn't find details on how this datasets are generated.

Could you provide insight into how this data is generated, or direct me to any resources or documentation on this matter?

Thank you very much in advance

Feedback on FHIR > 1_export_data.ipynb

the mnt path is kind of confusing. Where it says USERNAME here is not my AD username, but rather than name of the compute env? I think. Either way, this path took some digging to figure out and could be clarified in the notebook.

import subprocess

subprocess.run(["./run_synthea",
"-s", "42",
"-cs", "99",
"-p", "10",
f'--exporter.baseDirectory=/mnt/batch/tasks/shared/LS_root/mounts/clusters/USERNAME/code'
]);

Step 2.1 below, should somehow be after step 2.4, because the fhir server has not yet been created. First the user needs to go to Azure API for FHIR, create the FHIR server, then they can do the rest. Creating the server is not described in the instructions.

2.1) Create an "Azure API for FHIR"[3] instance, named <fhir_server>

Navigate to https://<fhir_server>.azurehealthcareapis.com/metadata and verify a "Capability Statement" is retrieved.
That means the FHIR server[3] is running.
Set fhir_server in Section 3.1
Use RBAC[6]: <fhir_server> left pane "Identity" -> "On" -> "Save"

This line : for filename in glob(f"/home/azureuser/cloudfiles/data/datastore/synthea/fhir/*.json"): did not work for me, I had to use the mnt path from the top of the notebook.
4. Set up the FHIR->Synapse Sync Agent

This notebook section follows the "FHIR to Synapse Sync Agent" tutorial provided Microsoft's "FHIR Analytics Pipelines" Github repository^[13].

4.1) Deploy the custom Azure template provided by the "FHIR to Synapse Sync Agent" tutorial^[13].

Navigate to the Github repo by clicking this link.

The GitHub link is no longer valid. I went to that repo, but its not clear which doc I use to deploy.

5.3) Convert all PacBio VCFs to TSV

This step assumes you already have VCF files in a storage account container. You could download the vcfs directly into the VM and then copy to the container or leave in the VM. Either way, should not assume the user already has the data.

Username misleading in mnt path

In the first FHIR notebook, the mnt path is kind of confusing. Where it says USERNAME here is not my AD username, but rather than name of the compute env? I think. Either way, this path took some digging to figure out and could be clarified in the notebook.

import subprocess

subprocess.run(["./run_synthea",
"-s", "42",
"-cs", "99",
"-p", "10",
f'--exporter.baseDirectory=/mnt/batch/tasks/shared/LS_root/mounts/clusters//code'
]);

3-pharmacogenomics-confidential.ipynb Need to index BAM file

Before running samtools view, need to add this line !samtools index HG002_GRCh38.haplotag.10x.bam

Creating FHIR Server

In this notebook 1-data-export, step 2.1 below, should somehow be after step 2.4, because the fhir server has not yet been created. First the user needs to go to Azure API for FHIR, create the FHIR server, then they can do the rest. Creating the server is not described in the instructions.

2.1) Create an "Azure API for FHIR"^[3] instance, named <fhir_server>

Navigate to https://<fhir_server>.azurehealthcareapis.com/metadata and verify a "Capability Statement" is retrieved.
That means the FHIR server^[3] is running.
Set fhir_server in Section 3.1
Use RBAC^[6]: <fhir_server> left pane "Identity" -> "On" -> "Save"

Json file list path to add to fhir store invalid path

This line : for filename in glob(f"/home/azureuser/cloudfiles/data/datastore/synthea/fhir/*.json"): did not work for me, I had to use the mnt path from the top of the notebook.

No link to Download PacBio VCF Files

In Notebook FHIR 1_export_data,

5.3) Convert all PacBio VCFs to TSV