Would be nice to add a link and/or the DOI for the PeerJ publication on the main /README.md

https://peerj.com/articles/13821/

https://doi.org/10.7717/peerj.13821

I'm curious if there are plans to add this tool to bioconda?

It would greatly benefit the community if one could easily install the dependencies via conda or use the docker image that is auto-generated via the biocontainers project.

I'd be happy to write up a Dockerfile, but it would probably reach more users if it was available on bioconda

Not sure what's causing this, I can manually fix the files but I really shouldn't need to. It causes any paired-end analysis to instantly fail.

Upon further testing so is VOI/VOC, I am fairly certain the method you are using to split the reads is flawed.

I've done a little digging and I think the use of fastq-dump --gzip is to blame, it is known to be buggy and the users inbuilt gzip should probably be used instead.

It was actually fastq-dump --split-files, I have submitted a PR.

Hello,

I was able to install and run the script outlined here to download all 6 datasets.

An odd thing is happening for datasets 'sars-cov-2-coronahit-rapid.tsv' and 'sars-cov-2-coronahit-routine.tsv' however.

All of the forward read fastq.gz files are downloading (i.e. NORW-F0A6F_CoronaHiT-ONT_1.fastq.gz) but only some of the corresponding reverse read files are downloading, while most of them are just empty files. Also all of the .fna files are empty.

An example of output:

I am wondering if this is normal behavior (if those empty “_2.fastq.gz” files really don’t exist and therefore can’t be downloaded for some samples), or if there is some error.

The forward and reverse files for the other 4 datasets all downloaded fine.

Note: I have Mac OS and am running this while logged into one of our BCM-HGSC login nodes.

Any help would be appreciated!

Thanks,
Evette

Having information on the primers utilized to generate the reads in the datasets could be useful for laboratories trying to re-create these assemblies while performing primer trimming properly.

Hi,

I downloaded the GISAID/NCBI genome based on accession numbers provided in the dataset#6 table and ran the pangolin v3.1.3 to get lineage assignment. All but one not matches with the expected lineage result (B.1.1.391) in the dataset#6.

Could you confirm the lineage of the hCoV-19_USA_CA-CZB-15265_2020 genome?

Here is the result I ran with the pangolin v3.1.3

taxon,lineage,conflict,ambiguity_score,scorpio_call,scorpio_support,scorpio_conflict,version,pangolin_version,pangoLEARN_version,pango_version,status,note

MW564975.1,B.1.1.450,0.0,0.9413064438373775,,,,PLEARN-v1.2.13,3.1.3,2021-06-15,v1.2.13,passed_qc,

hCoV-19/USA/CA-CZB-15265/2020|EPI_ISL_738705|2020-11-24,B.1.1.450,0.0,0.9413064438373775,,,,PLEARN-v1.2.13,3.1.3,2021-06-15,v1.2.13,passed_qc,

Here are the screenshots from the NCBI and GISAID metadata.

Thank you.

As I mentioned on SPHERES Slack, the table for VOI/VOC https://github.com/CDCgov/datasets-sars-cov-2/blob/master/datasets/sars-cov-2-voivoc.tsv is missing accessions for the consensus sequence.

Would be great to add these Genbank/GISAID accessions and also possibly the PANGO lineage assigned at the time these were generated, along with version. But accessions would be most crucial to add.

Apologies if this is documented somewhere and I missed it. I just wanted to know what the process was for producing and qc-ibg or curating the truth consensuses

Thanks so much for collecting these, is a huge contribution

Good morning,

I'm a student in Computer Science at Università degli Studi di Milano and for my thesis I am assessing some pipeline for the analysis of SARS-CoV-2 samples.
In order to select the best pipeline for our requirements, I'm using the benchmark datasets available here.
I found the Supplementary_table2 in your paper (Xiaoli L, Hagey JV, Park DJ, Gulvik CA, Young EL, Alikhan N-F, Lawsin A, Hassell N, Knipe K, Oakeson KF, Retchless AC, Shakya M, Lo C-C, Chain P, Page AJ, Metcalf BJ, Su M, Rowell J, Vidyaprakash E, Paden CR, Huang AD, Roellig D, Patel K, Winglee K, Weigand MR, Katz LS. 2022. Benchmark datasets for SARS-CoV-2 surveillance bioinformatics. PeerJ 10:e13821 http://doi.org/10.7717/peerj.13821) and I would like to use also the data contained there for evaluations (and not only the file in.tsv available for every dataset).
I'm writing here because I can't understand how the column 'Total reads' is calculated. In particular, I used FastQC (the value of the field 'Total Sequences') to compute this value and I also counted the reads in the original .FASTQ file but the numbers don't correspond to the ones published in the Supplementary_table2.

Do you know why the numbers are different? Is it possible that Supplementary_table2 is outdated with respect to the current version of the dataset?
If this is the case, which version of the dataset is matched to Supplementary_table2 and used in your paper?

Thank you very much for your time :)

Best regards,
Sara Manfredi

Hi TOAST team,

I am interested in curating the representative genomes for VOCs/VBMs. And according to your recent publication (Xiaoli, Lingzi, et al. "Benchmark datasets for SARS-CoV-2 surveillance bioinformatics." PeerJ 10 (2022): e13821.), your dataset 4&5 were prepared based on alignments to the 'internally curated consensus sequences'. May I ask for the details about how you curated those internally? Thank you.

Best,
Gyuhyon

I heard one comment that especially on the VOIVOC dataset, the column header "reference" is misleading because each of these samples is a lineage representative. I do not know the exact solution yet but I did want to earmark this issue.

Need to make instructions on how to contribute new datasets.

• Creating the spreadsheet
• Need to do a quality check by themselves. Submit quality check metrics.
  • Need to have quality checks listed
  • Need to have thresholds listed
  • A checkbox for whether it passed
• Making a pull request
• A note about CI testing - full dataset needs to pass CI

It would be good to have the complete VCF files available for all the sample sets which would be beneficial. This allows that truth set does not rely on couple of interest sites, but allows for complete evaluation across the genome.