Missing data issue about clinvar HOT 4 CLOSED

There are a few reasons why certain variants are skipped.

• The reference file has a mismatch with what's in the ClinVar data
• The reference file used has the reference sequence soft-masked in lower-case, causing a pseudo-mismatch. Because of this line https://github.com/ericminikel/minimal_representation/blob/master/normalize.py#L80 this case results in an "Incorrect REF value" warning being generated and the variant skipped. I have a fork of normalize.py which makes the reference bases uppercase: https://github.com/kristjaneerik/minimal_representation/blob/uppercase-ref/normalize.py#L80
• If the REF and ALT alleles are the same or an invalid nucleotide is seen
• There's no SequenceLocation information in the ClinVar XML, so we don't know where the variant is

However, I see this particular variant in clinvar_with_exac.tsv.gz (line 46777), clinvar.vcf.gz (line 46822) and clinvar.tsv.gz (line 46777) when zgrep-ing for 92130876 in these files that are currently in the repo.

from clinvar.

Hello kristjaneerik !

Thank you for your response, I was checking several variants and here is the one that I have not found there (I have searched on ready built TSV files):

7 99382096 rs2740574 C T

Another problem is that when I was trying to run the pipeline (inregarldess of using your normalize.py fork or original some) I have got the errors (listed below).

Can you please tell me what is wrong with my environment? Or probably FASTA selection (I've tried different reference fastas from UCSC and 1000g, but no success).

[Dec 8 19:23:53]: --> Exec 1.4: Rscript join_data.R variant_summary.txt.gz
[Dec 8 19:23:53]: Output (last mod N/A): clinvar_combined.tsv [doesn't
exist yet]
[Dec 8 19:23:59]: [1] 184803 14
[Dec 8 19:24:12]: Warning message:
[Dec 8 19:24:12]: In scan(file = file, what = what, sep = sep, quote = quote, dec = dec, :
[Dec 8 19:24:12]: embedded nul(s) found in input
[Dec 8 19:24:12]: [1] 354889 30
[Dec 8 19:24:12]: Error in [.data.frame(x, r, vars, drop = drop) :
[Dec 8 19:24:12]: undefined columns selected
[Dec 8 19:24:12]: Calls: subset -> subset.data.frame -> [ -> [.data.frame
[Dec 8 19:24:34]: $subset(txt_download, assembly == "GRCh37", select = desired_columns)
[Dec 8 19:24:34]: <environment: 0x222f190>
[Dec 8 19:24:34]:
[Dec 8 19:24:34]: $subset.data.frame(txt_download, assembly == "GRCh37", select = desired_colu
[Dec 8 19:24:34]: <environment: 0x222de20>
[Dec 8 19:24:34]:
[Dec 8 19:24:34]: $x[r, vars, drop = drop]
[Dec 8 19:24:34]: <environment: 0x2396f20>
[Dec 8 19:24:34]:
[Dec 8 19:24:34]: $``[.data.frame(x, r, vars, drop = drop)
[Dec 8 19:24:34]: <environment: 0x2396f58>
[Dec 8 19:24:34]:
[Dec 8 19:24:34]: $`stop("undefined columns selected")`
[Dec 8 19:24:34]: <environment: 0x2352348>
[Dec 8 19:24:34]:
[Dec 8 19:24:34]: attr(,"error.message")
[Dec 8 19:24:34]: [1] "Error in `[.data.frame`(x, r, vars, drop = drop) : \n undefined columns selected\nCalls: subset -> subset.data.frame -> [ -> [.data.frame\n"
[Dec 8 19:24:34]: attr(,"class")
[Dec 8 19:24:34]: [1] "dump.frames"
[Dec 8 19:24:34]: Finished 1.4. Running time: 0:00:41.048258 sec.
[Dec 8 19:24:34]: WARNING: unable to rename tmp.2016-12-08_19.23.51.clinvar_combined.tsv to clinvar_combined.tsv. tmp.2016-12-08_19.23.51.clinvar_combined.tsv is not readable
[Dec 8 19:24:34]: --> Skipping 1.5: (cat clinvar_combined.tsv | head -1 > tmp.2016-12-08_19.23.51.clinvar_combined_sorted.tsv ) && (cat clinvar_combined.tsv | tail -n +2 | egrep -v "^[XYM]" | sort -k1,1n -k2,2n -k3,3 -k4,4 >> tmp.2016-12-08_19.23.51.clinvar_combined_sorted.tsv ) && (cat clinvar_combined.tsv | tail -n +2 | egrep "^[XYM]" | sort -k1,1 -k2,2n -k3,3 -k4,4 >> tmp.2016-12-08_19.23.51.clinvar_combined_sorted.tsv )
[Dec 8 19:24:34]: Input (last mod N/A): clinvar_combined.tsv [Error - input file not found]
[Dec 8 19:24:34]: Traceback (most recent call last):
[Dec 8 19:24:34]: File "/root/.local/lib/python2.7/site-packages/pypez.py", line 769, in does_command_need_to_run
[Dec 8 19:24:34]: try: raise Exception("File not found: " + str(input_filename))
[Dec 8 19:24:34]: Exception: File not found: clinvar_combined.tsv
[Dec 8 19:24:34]:

Thanks in advance,
Peter

from clinvar.

With regards to 7 99382096 rs2740574 C T: yes, it is not in any of the posted files. When I fetch from the genome I used, that position shows up at c -- so this variant was (most likely) skipped due to softmasking.

The issue you are seeing (join_data.R failing, which triggers other failures downstream) is because ClinVar changed their data format and this codebase hasn't been updated to deal with it as the change is not a simple one. See #18 , #19. The solution for now, unfortunately, is to use older files as outlined in those issues. There's nothing wrong with your environment as such, you just need to specify an older XML and summary table as in #19 .

I don't know if/when the original authors will get to address this join_data.R issue; I have some thoughts on what could be done, but might not have the time to do it.

from clinvar.

#21 from @XiaoleiZ has fixed join_data.R so we'll be making up-to-date releases again.

from clinvar.