Test Scope about alphapept HOT 12 CLOSED

For the Workflow test, we should set up something that runs on Linux and Mac OS X.
A nice solution for Windows would be to run these tests on self-hosted cloud runners. One way to achieve this would be to use an Auto Scaling Group on AWS: https://dev.to/jimmydqv/github-self-hosted-runners-on-aws-part-2-ec2-3jhj

This however would not work for Mac Os X - but maybe we could use something like this: https://www.macstadium.com

from alphapept.

Is it possible top share the test files? This allows a local test to avoid spamming the runners too much by checking if a commit actually passes the tests...

from alphapept.

I just uploaded the current test files here:
https://datashare.biochem.mpg.de/s/sYaphoaccYDTnez

from alphapept.

Thanks! Can you upload the settings and fasta files as well so I can fully reproduce?

from alphapept.

Thanks! Can you upload the settings and fasta files as well so I can fully reproduce?
@swillems I uploaded the FASTA, contaminants to the folder and the settings of the test runner.

from alphapept.

Great! They probably could/should be deleted from the source folder then? Also a minor side note/question: Why is the (old) version number of alphapept in the settings file? This seems more logical in a log file if you ask me...

from alphapept.

Great! They probably could/should be deleted from the source folder then?

On the long run I would probably make a dedicated folder and upload them so that people can also use this to test things.

Also a minor side note/question: Why is the (old) version number of alphapept in the settings file? This seems more logical in a log file if you ask me...

It should certainly be in a log file but I also think it is useful within a settings file. The idea would be if at a later stage we rename some settings we could enable backwards compatibility to older settings-files..

from alphapept.

Both make sense to me, thanks for explaining

from alphapept.

Github Actions now push the run results to a DB so that we can visualize the run results.
https://charts.mongodb.com/charts-alphapept-itfxv/public/dashboards/5f671dcf-bcd6-4d90-8494-8c7f724b727b

Let me know if you can think of any metrics we should display.

from alphapept.

• By now we also have 5 thermo IRT runs, of which one could/should be included. Probably good to share them on the drive as well and include the irt.fasta and .yaml files. Come to think of it, we might actually opt to use these sample/test files directly with github for the sake of consistency and transparancy. This might require a download which is rather redundant, so perhaps a check to see if the file-for-download is updated might be beneficial.
• I would also track the number of peptide IDs and not only the proteins. Number of PSMs is relevant as well.
• We should split the timing for individual steps (db creation, FF, search, score, LFQ...). Perhaps we should also look into implementing the CLI commands properly, since we (or at least I) only use the workflow command and the others are mostly stubs. This way we don't need to write to much "custom" code purely for testing purposes, bu can recycle code that is relevant for end-users. This obviously requires good control of intermediary results, I will look into that some more.
• We have a bumpversion, but I think that we have been at 0.2.8-dev for a while now, even with multiple commits in between. We should update our versions more frequently (automatically as a git action when pushing into the dev branch perhaps?)
• I see you also track file sizes in the test_ci.py, but I am not directly sure how to access this in the charts?

from alphapept.

Not sure if it is worth considering. But in relation to test files, consistency, transparency and reproducibility we could consider a git LFS bundle. Irt test files and others can even be included with a free account.

from alphapept.

I just found out why your Bruker IRT testing takes 4min and a local test is <2min: you use a human db, take a FASTA of iRT peptides from Biognosys instead

from alphapept.