Git Product home page Git Product logo

bridges-cheatsheet's Introduction

Bridges cheat sheet

Environment variables

Variable Meaning
$PROJECT /ocean/projects/{med_project_number}/{username}
$LOCAL Local storage in current node - temporary but much faster

Simple commands

Interactive session

interact -p RM-shared -N 1 --ntasks-per-node=10 -t 03:00:00

List current jobs

squeue --me

Live print log file as it gets updated

tail -f my_out_file.log

Kill all your jobs

scancel --me

Singularity/Apptainer

.bashrc additions

# Singularity cache
export SINGULARITY_CACHEDIR=$PROJECT/.singularity/cache
export SINGULARITY_LOCALCACHEDIR=$PROJECT/.singularity/tmp
export SINGULARITY_TMPDIR=$LOCAL

export APPTAINER_CACHEDIR=$SINGULARITY_CACHEDIR
export APPTAINER_LOCALCACHEDIR=$SINGULARITY_LOCALCACHEDIR
export APPTAINER_TMPDIR=$SINGULARITY_TMPDIR
  • Avoids overflowing home directory
  • Uses local storage for temporary files - much faster image builds

Build C-PAC nightly container

singularity build $PROJECT/images/nightly.sif docker://ghcr.io/fcp-indi/c-pac:nightly

Anaconda

module load anaconda3/2022.10
conda create -n myenvironment python=3.11
conda activate myenvironment

(List available modules with module av)

Install / update github hosted Python package

pip uninstall -y clmunch && sleep 2 && pip install git+https://github.com/childmindresearch/cpac-log-muncher
  • For some reason pip install -U ... does not always work.

C-PAC job file template

#!/usr/bin/bash
#SBATCH --job-name my_job_name
#SBATCH --output my_job_out.log
#SBATCH --nodes 1
#SBATCH --partition RM-shared
#SBATCH --time 48:00:00
#SBATCH --ntasks-per-node 16

set -x

cd /my/job/folder

singularity run --cleanenv \
-B "$PROJECT"/input:"$PROJECT"/input:ro \
-B "$PROJECT"/output:"$PROJECT"/output \
"$PROJECT"/images/nightly.sif \
"$PROJECT"/input \
"$PROJECT"/output \
participant \
--skip_bids_validator \
--n_cpus 15 \
--mem_gb 31.0 \
--participant_label sub-123 \
--preconfig abcd-options \
--save_working_dir "$PROJECT"/my/job/folder 
  • Memory is always 2GB / task

Overwrite image C-PAC version with local development version

-B {cpac_dir}/CPAC:/code/CPAC \
-B {cpac_dir}/dev/docker_data/run.py:/code/run.py \
-B {cpac_dir}/dev/docker_data:/cpac_resources \

Utilities

Links

bridges-cheatsheet's People

Contributors

nx10 avatar

Watchers

Steve Giavasis avatar Greg Kiar avatar Nathalia Esper avatar Connor Lane avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.