dbm.error: db file doesn't exist; use 'c' or 'n' flag to create a new db about nanocompore HOT 21 CLOSED

sunsetyerin,

Could you share all the contents of the nanocompore_sampcomp directory?

Thanks,
Logan

from nanocompore.

Hi @lmulroney, I ran the Metacompore pipeline to run nanocompore

[yekim@gphost08 MetaCompore]$ ls -l /path/to/results/nanocompore/nanocompore_sampcomp
total 286717468
1108278509 Dec 30 04:26 outnanocompore_results.tsv
1167914580 Dec 30 04:26 outnanocompore_shift_stats.tsv
290162606080 Dec 29 03:46 outSampComp.db
8489297 Dec 29 23:52 out_sampcomp.log

from nanocompore.

Hi sunsetyerin,

It appears that you are missing the
outSampComp.db.bak
outSampComp.db.dat
outSampComp.db.dir
files, which are required for the plotting api to properly open the database file.

Do you have values that make sense in the results.tsv file?

Does the end of your log file look something like this:
2022-01-26T17:02:10.709087+0000 INFO - Process-3 | All Done. Transcripts processed: 1
2022-01-26T17:02:10.754651+0000 INFO - MainProcess | Loading SampCompDB
2022-01-26T17:02:10.764074+0000 DEBUG - MainProcess | Reading Metadata
2022-01-26T17:02:10.769103+0000 DEBUG - MainProcess | Loading list of reference ids
2022-01-26T17:02:10.769542+0000 DEBUG - MainProcess | Checking files and arg values
2022-01-26T17:02:10.790533+0000 INFO - MainProcess | Calculate results
2022-01-26T17:02:11.580120+0000 DEBUG - MainProcess | Save reports to ./oligo1
2022-01-26T17:02:11.581240+0000 DEBUG - MainProcess | Saving extended tabular report
2022-01-26T17:02:12.513242+0000 DEBUG - MainProcess | Saving shift results

If yes to either question, could you try executing nanocompore sampcomp outside of the metacompore pipeline to check if those database files are created?

from nanocompore.

from nanocompore.

Sorry I don't see an attachment here on github.

from nanocompore.

out_sampcomp.log
Here is the log file @lmulroney

from nanocompore.

From the log file, it appears that Nanocompore crashed while processing the reference transcript, ENST00000336079.8, with a "KeyboardInterrupt" error code. Usually this means that the job was killed by the user for some reason, or that the run went above the memory limit provided for the job.

There may be an issue with that particular reference transcript, so it might be worth looking at if there is something different for ENST00000336079.8 compared to other transcripts in the reference.

Let me know if any of these explanations make sense.

from nanocompore.

@lmulroney , thanks for the explanation.

I found that when I run nanocompore with many threads, it just crashes with memory issues, and when I run it with fewer threads, it stops running without error messages and doesn't make progress.

Also, when I tried to run nanocompore with container("library://aleg/default/nanocompore:1.0.3"),
it doesn't give other files other than outSampComp.db files.
But the thing is I really need the following files ;
outSampComp.db.bak
outSampComp.db.dat
outSampComp.db.dir

It seems like nanocompore requires a lot of memory, will there be any other ways I can generate the files that I need?

from nanocompore.

I typically run nanocompore sampcomp with 60 Gb of memory with 6 threads on a typical 6 sample human dataset. One thing to note, one thread is dedicated to the error log and one thread is dedicated to writing the results. The 3rd and more threads do data processing. So if you are only using 3 threads it can take a while to process the data.

I honestly do not know why the container is not producing the db support files (.bak, .data, and .dir). Nanocompore writes those three files as the last thing it does, and so if there was a processing error then that could possibly explain the issue.

Depending on what type of plot you are interested in, you can do much of the plotting by reading in the results.tsv file. It will have all of the p-values and statistical test results by position. And then plot using your preferred plotting language (R, python, matlab, etc). You really only need to load the database if you are interested in plotting raw ionic current values for each position.

from nanocompore.

Thanks @lmulroney,
I can see the function SampCompDB.save_to_bed
But I was wondering if there is any ways I can generate .bed file out of .db file without
outSampComp.db.bak
outSampComp.db.dat
outSampComp.db.dir
I need .bed file out of outSampComp.db

from nanocompore.

All of the information required to make a bed file is in the results tsv. You need to parse the position and chromosome information from the results tsv file, and you can use one of the p-value and/or LOR columns to filter which positions are added to your bed file. This also allows you to have a little more control over how the filtering is done and can let you do something like peakcalling on top of the p-value filtering if you want. I personally use my own python script instead of the database methods for these reasons.

from nanocompore.

@lmulroney, thanks.
this is how my result table looks like.

The table doesn't include chromosome information needed to generate .bed files.
Is It because I used a wrong reference file? ("http://ftp.ensembl.org/pub/release-106/fasta/homo_sapiens/cdna/Homo_sapiens.GRCh38.cdna.all.fa.gz")
If so, can you recommend the right reference file?

from nanocompore.

It is strange that the chr information is not in the results tsv. If the chr information is missing from the results tsv file, it is also likely missing from the db, so getting access to the db probably wouldn't help in this case.

Did you save the nanopolish evenalign file or did you pipe it through to nanocompore eventalign collapse? I ask to verify if the chr information is in either or both of those files.

from nanocompore.

@lmulroney, I checked on nanopolish eventalign file and nanocompore eventalign collapse file.
They all don't contain chromosome information. But even in documentation of each software, none of them include chromosome information.
Would it be different if I feed in bed file for nanocompore sampcomp function?
--bed BED BED file with annotation of transcriptome used for mapping (optional)

from nanocompore.

Hi Sunsetyerin,

The contig column (column 1) in the nanopolish eventalign collapse file is your chromosome information, if that is NA, then you have an issue with running nanopolish eventalign (or f5c eventalign).

For the nanocompore eventalign collapse tsv file, the chr information is after the read name. If that is NA or there is no text after the read name in the nanocompore eventalign collapse tsv file, and there was information in the eventalign file, then you have an issue with running nanocompore eventalign collapse.

Using a gzipped reference file shouldn't cause these kinds of issues, but it is possible. If there is missing data in those two files, perhaps try rerunning the pipeline with an uncompressed reference file.

from nanocompore.

@lmulroney,

This is how my eventalign looks like.

And this is how my nanocompore eventalign collapse looks like.

I think the reference fasta file doesn't include chromosome information but only gene names that nanocompore final table doesn't include the information at all.
Can you please recommend the right reference file to use for nanocompore?

from nanocompore.

I see the problem now. Sorry for not grasping what you were trying to do earlier. When you run nanocompore sampcomp you can include the option --bed with a bedfile of the transcriptome in the reference genome coordinates. This should add genome coordinates to the results.tsv file.

from nanocompore.

Hi @lmulroney,
I've tried to feed in bed file for nanocompore sampcomp but continuously receiving errors.
It seems like nanocompore sampcomp cannot read or match bed file info.

I used this trasncriptome fasta file throughout the pipeline;
http://ftp.ensembl.org/pub/release-106/fasta/homo_sapiens/cdna/Homo_sapiens.GRCh38.cdna.all.fa.gz

Do you mind if I ask how did your group generated the bed file for the analyses?

from nanocompore.

Hi @sunsetyerin,

Which bed files have you used for that reference transcriptome file, and what errors were you getting from nanocompore when you used a bed file?

Thanks,
Logan

from nanocompore.

@lmulroney

This is my assumption but since I already fed transcriptome fasta file that didn't include genomic coordinate, even if I feed in the correct bed file, nanocompore sampcomp won't give me results with chromosome information.
Since nanocompore sampcomp requires fast file that was used during the mapping step, in which case I also used transcriptome fasta with no genomic coordinate.

Is that correct?

from nanocompore.

Hi @sunsetyerin,

Yes, you should use the same transcriptome reference in sampcomp that you used for mapping the reads. the bed file that you're using should have the genome coordinates of every transcript that exists in the transcriptome reference that you're using. Sampcomp will do an internal mapping of the transcriptome coordinates to the genome coordinates given the bed file provided. Based on the error message, it appears that there are more references in your fasta file than exist in the bed file you used.

from nanocompore.