ccbaumler / 2022-benchmark Goto Github PK

https://stackoverflow.com/questions/65603381/slurm-nodes-tasks-cores-and-cpus

@ccbaumler and I have developed a benchmarking workflow to analyze various different metrics of sourmash commands. This will help us identify parts of the program which can be further optimized.

The input sig files vary in sample location, instrumentation, base quantity, and file size. They are listed below:

(Metadata found here and Project information here)

Metrics are measured two different ways:

1. Flamegraphs constructed using python profiler (py-spy). An example of a flamegraph for the gather command:
   

2. Furthermore, the workflow measures Computational metrics by snakemake's benchmark directive. The benchmark outputs a tsv file with the following metrics ref :

Possible additions to the benchmarking work flow:

1. Running the command on each sample multiple times of an arbitrary standard (say 3 times) and aggregating these results. This will prevent one time events and outliers from influencing our analysis.
2. Use the perf command to gather more robust metrics. See the flamegraph off-shoots here

https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html

Examples of benchmarking from lab

An example of using snakemake for benchmarking

The benchmark output files from the above repo

The use of mprof which profile memory usage on sourmash gather

Jessica's benchmarking process

sourmash compare memory needs profiling!!

Tools for benchmarking software

Flamegraph

• Repo
• Flamegraph
• Memory profile flamegraph
• differential flamegraph
• Rust edition (Read the bottom)

py-spy

• documentation
• repo

timeit

• documentation

snakemake

• Benchmarking a workflow
• Explanation of benchmarking variables
• Git repo organization for snakemake workflow
• Catalog of workflow
• example of properly organized workflow

The benchmark files created by snakemake do not come with an internal id column only the benchmarks. This could become an issue when running the same input file multiple times.