To have pathway activation score, we need to localize MSigDB datset

• Think about MLOps strategy with Hydra + AWS + Github Action

Reference:
https://towardsdatascience.com/jupyter-data-science-stack-docker-in-under-15-minutes-19d8f822bd45

• Boto3 supports AWS control by python, current bash script needs to be replaced to boto

• Need to upload normalized expression data for each dataset(CD4, CD8, CD14, etc) to S3
• Need to upload Rcode (DEseq2) which was used to get DEG

@kicheolkim It would be good if we have a new person for AI/ML support. We have knowledge about that obviously, but we need more advanced optimization techniques for ML model. What do you think? Do we have to find another person for this project?

*Newly setup reviewNB for the repo, and need to be tested with jupyter code

• Setting Github repo
• Data preparation in the local PC or AWS (S3 bucket)
• Setup feather instead of CSV
• Make test-set for gene expression data with random samples
• Establishing engineering strategy (IDE, Jupyter notebook)

dataset summary (what we have, what we can do), data analysis strategy

Think about archiving

tar cvfz - ./ | aws s3 cp - s3://openkbc-test-lambda-bucket/test1.tar.gz

• Need progress page for monitoring snakemake log in controller

publications, GTeX, GEO for validation or other analysis

Pipelines are working in a container through snakamake workflow. It needs a controller for user configuration and data input dynamically without up and down running a container.

• AWS Aurora, DynamoDB or etc

Currently, we are using docker instance to deploy notebook and it could be replaced with AWS sageMaker, possibly, pipeline container can be deployed in ECS.

DEG pipeline has memory issue in reading all files

• Docker container has default memory limit(=2g).
• Attempted to change the limit to 4g, but failed (reference: docker/compose#4513)
• Needs code optimization

#Error occurred in Mac conda-
dyld: Library not loaded: @rpath/libreadline.6.2.dylib

mamba update -c rdonnellyr -c main --all

mamba create --name new-env r-essentials=3.6.1 #it works, not working with DEseq

# First, create conda env with r-essentials
mamba create --name new-env r-essentials=3.6.1 

# Install packages with Rscript
Rscript notebook/installers/installer_Rpackage.R