justinchu / jupiterplot Goto Github PK

I get this circos error when I run Jupiter. This error is to be found when I install circos via LinuxBrew or compile from source. So I guess it does not originate from circos itself.

circos -noparanoid -conf Ctet_ntJoin_k70_w1000.conf
debuggroup summary 0.10s welcome to circos v0.69-8 15 Jun 2019 on Perl 5.026001
debuggroup summary 0.10s current working directory /home/ashwin/Projects/tetanus/trimmed_raw_data/ntJoin
debuggroup summary 0.10s command /home/ashwin/sources/circos-0.69-9/bin/circos -noparanoid -conf Ctet_ntJoin_k70_w1000.conf
debuggroup summary 0.10s loading configuration from file Ctet_ntJoin_k70_w1000.conf
debuggroup summary 0.10s found conf file Ctet_ntJoin_k70_w1000.conf
debuggroup summary 0.17s debug will appear for these features: output,summary
debuggroup summary 0.17s bitmap output image ./Ctet_ntJoin_k70_w1000.png
debuggroup summary 0.17s SVG output image ./Ctet_ntJoin_k70_w1000.svg
debuggroup summary 0.17s parsing karyotype and organizing ideograms
Use of uninitialized value $start in substitution (s///) at /home/ashwin/sources/circos-0.69-9/bin/../lib/Circos/Karyotype.pm line 128, line 2.
Use of uninitialized value $end in substitution (s///) at /home/ashwin/sources/circos-0.69-9/bin/../lib/Circos/Karyotype.pm line 129, line 2.

*** CIRCOS ERROR ***

  cwd: /home/ashwin/Projects/tetanus/trimmed_raw_data/ntJoin

  command: /home/ashwin/sources/circos-0.69-9/bin/circos -noparanoid -conf
  Ctet_ntJoin_k70_w1000.conf

Start [undef] and/or end [undef] coordinate in karyotype file don't appear
to be numbers. Thousands separators , and _ are allowed, but not any other
characters.

Stack trace:
at /home/ashwin/sources/circos-0.69-9/bin/../lib/Circos/Error.pm line 425, line 2.
Circos::Error::fatal_error("data", "malformed_karyotype_coordinates", undef, undef) called at /home/ashwin/sources/circos-0.69-9/bin/../lib/Circos/Karyotype.pm line 131
Circos::Karyotype::read_karyotype_file("Ctet_ntJoin_k70_w1000.karyotype", HASH(0x564897bd1bf8)) called at /home/ashwin/sources/circos-0.69-9/bin/../lib/Circos/Karyotype.pm line 92
Circos::Karyotype::read_karyotype("file", "Ctet_ntJoin_k70_w1000.karyotype") called at /home/ashwin/sources/circos-0.69-9/bin/../lib/Circos.pm line 218
Circos::run("Circos", "_cwd", "/home/ashwin/Projects/tetanus/trimmed_raw_data/ntJoin", "_argv", "-noparanoid -conf Ctet_ntJoin_k70_w1000.conf", "paranoid", 0, "configfile", ...) called at /home/ashwin/sources/circos-0.69-9/bin/circos line 536
/home/ashwin/sources/JupiterPlot/jupiter:97: recipe for target 'Ctet_ntJoin_k70_w1000.svg' failed
make: *** [Ctet_ntJoin_k70_w1000.svg] Error 255
make: *** Deleting file 'Ctet_ntJoin_k70_w1000.svg'
rm Ctet_ntJoin_k70_w1000.links.final Ctet_ntJoin_k70_w1000.rv.links.bundled Ctet_ntJoin_k70_w1000.conf Ctet_ntJoin_k70_w1000.fw.links.bundled Ctet_ntJoin_k70_w1000_scaffolds.fa.fai Ctet_ntJoin_k70_w1000.fw.links.bundled.flipped

Any idea why this is happening or how to sort this out?
Thanks!

Usually the output for this will be messy, but users should have the option of adding helpful labels in the plot

Hi,

• I would like to change my reference labels and regenerate the plot. Is it possible to run only the plotting step again after the analysis is completed?
• If so, should I just edit the prefix.seqOrder.txt file or something else?

Thanks

Hi,
I have been running JupiterPlot to compare an assembly to a reference genome and ended up with fractions of scaffolds (new assembly) aligning to the reference genome. However, the plot leads to think that whole scaffolds were represented. Is it expected?

Currently if you alter https://github.com/JustinChu/JupiterPlot/blob/master/config/rawConf.conf#L29 to prevent masking of the placeholder names. Then you can convert them using the prefix.scaffold.txt file but this should be made a lot easier for diagnostic purposes.

This visualization requires the use of local alignment of scafftigs (scaffold broken by a stretch of Ns/unfilled scaffold gaps). Thus, if repetitive regions exist they may not align properly even if correctly scaffolded especially if surrounded by these scaffold gaps.

Hi allI,

I am wondering if there's a simple way to change the color and replot the picture?
I try to modify the config file, however I noticed that in the end of the pipeline, the config file would be removed.
Is there any suggestions? Thanks!

Thanks for writing this tool, it works very well. I was wondering if it would be possible to add an option such that all contigs (query and ref) are shown, even if they don't map. I know the ng option can be increased, but this can be kind of fiddly to find the right value. It would be nice if there were a flag to set that automatically just shows everything, even if it would be clunky looking. Thanks!

Hi, thank you for the tool!

I have a problem that Y chr is not seen on the output plot, although I checked the alignment with IGV and Y chr is definetely present. I also did the same plot with CHM13 instead of hg38 - same result. What do you think is the problem?

UPD:

Solved

I changed ng parameter to maximum possible value that doesn't give an error. I guess this number is case specific, so you need to manually adjust the default values (80), for me the number was 87. Also, this issue helped me a lot - #38

When comparing assembles that are both fragmented, some reference seqeunces are likely not well ordered, leading to the appearance of incongurent linkts. It may be possible to iteratively place the reference seqeunces relative to their best locations on the scaffolds.

Current behavior: Reference is ordered by input file order.

Workflow:

1. Order Scaffolds to reference (input: alignment position of scafftigs, order of reference chr; output: array of order of scaffolds)
2. Order Reference to scaffolds (input: alignment position of scafftigs, order of scaffolds; output: array of order of reference chr)
3. go back to 1

If done continously, will this eventually reach an equalibrium?

Hi all,

For the first question, I am curious about how to control the number of how many scaffold to be render, because I wish to plot all the scaffold in my assembly.

My script looks like:

jupiter name=Test minBundleSize=5000 ng=100 m=500 ref=refercnece.fa fa=draft_genome.fa

However, when I mapped to different reference genome, the number of scaffolds are different.
For example, my genome has about 600 scaffolds, and they were all rendered when I mapped to reference genome A.

When I mapped to the reference genome B, only 350 scaffolds were rendered. The script I used were the same except the ref directory. What makes this difference?

For second question, I notice that some of the unmapped reads were insert between those that were mapped. Most of the unmapped scaffolds gather together on one side of the circle, but there were still some of unmapped reads spread among the mapped reads.

How can I control their order?

Thanks for your kindly reply.
Cheers

For debugging on different systems and testing out the code there should I should include a small demo on a reasonable sided genome.

File sizes could be an issue. Should this be hosted on github?

incongruent links would be outliers compared to the true position (chromosome position & distance from in link) of the link. This could be done to highlight where misassemblies occur.

Other than manually editing the config files, removal of chr names is not possible. This feature will be useful if using a draft or fragmented reference seqeunce.

There seems to be an incongruence between JupiterPlot and a "regular" dot plot from a MUMMER (nucmer) alignment of two genomes of a chlorophyte algae species (21 Mbp). JupiterPlot shows no connections, whereas the dot plot shows large regions of similarity pretty much throughout the entire genome.

Please find the genomes and results here: https://uni-bonn.sciebo.de/s/LrubQxY319Lf8nn

I ran both JupiterPlot and MUMMER with default parameters:

cd jupiterplot
jupiter name=Micpus ref=../data/Micpus1545_MpusillaCCMP1545_228_v3.0.fasta fa=../data/MicpusNOUM17_MicpuN3v2_AssemblyScaffolds.fasta

cd mummer_dotplot
nucmer --coords ../data/Micpus1545_MpusillaCCMP1545_228_v3.0.fasta ../data/MicpusNOUM17_MicpuN3v2_AssemblyScaffolds.fasta

To visualise the MUMMER alignment, I used Dot (https://dot.sandbox.bio/). It requires input reformatted by the DotPrep.py script included in the mummer_dotplot directory. It was run as:

python DotPrep.py --delta out.delta --out Micpus

This produces the files Micpus.coords and Micpus.coords.idx that are also in the uploaded results, and can be used in the Dot viewer to reproduce the dot plot above.

I tried setting minBundleSize=1 and maxGap=9999999999 with the effect that it renders three small connections:

Do I need to set other parameters? Or how can I make JupiterPlot's output show the obvious similarity between the two genomes that is visible in the dot plot?

Need to generate:
BED files from contigs
agpFiles from bam file to deal with split alignments

Hello,

I notice that I don't get similar results from ref=reference.fa fa=scaffolds.fa as I do with ref=scaffolds.fa fa=reference.fa. I am looking at one sequence in reference.fa, and 4 sequences in scaffolds.fa. All sequences are greater than the default m=100000, and they do all have matching regions. Sum of scaffold lengths approximately equal to size of reference_1.

jupiter ng=200 name=ref_scaff ref=ref.fa fa=scaff.fa
cat ref_scaff.seqOrder.txt
ref0	reference_1	scaf2	scaffold_2	-
ref0	reference_1	scaf4	scaffold_4	+
ref0	reference_1	scaf1	scaffold_1	+
ref0	reference_1	scaf3	scaffold_3	-

jupiter ng=200 name=scaff_ref ref= scaff.fa fa=ref.fa
cat scaff_ref.seqOrder.txt
ref0	scaffold_1	scaf1	reference_1	+

Is this expected behaviour?
Thanks,
Chris

Hello, I found repetitively this error, with different parameters configuration. Please, any advice or suggestion? Thank you in advance.
(base) user@USER:~/JupiterPlot-1.0$ perl jupiter name=xxx ref=/GCF_00000xxxxx.x_12X_genomic_CHR.fasta fa=xxx_consensus_radtag_corrected.fasta ng=10 maxGap=500000 minBundleSize=50000 m=10000000 i=0 t=20 g=100 gScaff=100000
bwa mem -t 20 -x intractg xxx_reference.fa xxx-agp.fa > xxx-agp.sam
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[main] Version: 0.7.17-r1188
[main] CMD: bwa mem -t 20 -x intractg xxx_reference.fa xxx-agp.fa
[main] Real time: 0.240 sec; CPU: 0.196 sec
grep -v XA xxx-agp.sam | grep -v '^@' | awk '{if($5 > 50 || $5 == "" ) print}' | perl /user/JupiterPlot-1.0/bin/samToBed.pl > xxx-agp.bed
make: *** [jupiter:82: xxx-agp.bed] Error 1
make: *** Deleted file 'xxx-agp.bed'
rm xxx_reference.fa.bwt

Trying out this software. Really excited for a circos plot. However, it seems to fail in the drawing stage. This is the last bit of code (there is more of these "Use of..." above):

Use of uninitialized value $chromosomes{"ref61"} in concatenation (.) or string at /home/mmiller/JupiterPlot-master/bin/generateConf.pl line 388, line 4338.
has no alignments
Use of uninitialized value $chromosomes{"ref63"} in concatenation (.) or string at /home/mmiller/JupiterPlot-master/bin/generateConf.pl line 388, line 4338.
has no alignments
anna_chromW has no alignments
Use of uninitialized value $chromosomes{"ref65"} in concatenation (.) or string at /home/mmiller/JupiterPlot-master/bin/generateConf.pl line 388, line 4338.
has no alignments
anna_chromZ_RaGOO has no alignments
cat test.rv.links | /home/mmiller/JupiterPlot-master/circos-tools-0.22/tools/bundlelinks/bin/bundlelinks -max_gap 100000 -min_bundle_size 50000 > test.rv.links.bundled
num_links 112
num_initial_bundles 38
num_links_in_initial_bundles 112
num_passed_bundles 0 (0.00%)
num_links_in_passed_bundles 0 (0.00%)
#cat test.rv.links | perl /home/mmiller/JupiterPlot-master/bin/linkCollapse.pl -m 100000 -b 50000 > test.rv.links.bundled #simple bundle
cat test.fw.links | /home/mmiller/JupiterPlot-master/circos-tools-0.22/tools/bundlelinks/bin/bundlelinks -max_gap 100000 -min_bundle_size 50000 > test.fw.links.bundled
num_links 1020
num_initial_bundles 147
num_links_in_initial_bundles 1020
num_passed_bundles 5 (3.40%)
num_links_in_passed_bundles 615 (60.29%)
#cat test.fw.links | perl /home/mmiller/JupiterPlot-master/bin/linkCollapse.pl -m 100000 -b 50000 > test.fw.links.bundled #simple bundle
awk ' { t = $5; $5 = $6; $6 = t; print; } ' test.fw.links.bundled > test.fw.links.bundled.flipped
cat test.rv.links.bundled test.fw.links.bundled.flipped > test.links.final
circos -noparanoid -conf test.conf
debuggroup summary 0.19s welcome to circos v0.69-8 15 Jun 2019 on Perl 5.026002
debuggroup summary 0.19s current working directory /home/mmiller/JupiterPlot-master
debuggroup summary 0.19s command /home/mmiller/miniconda3/bin/circos -noparanoid -conf test.conf
debuggroup summary 0.19s loading configuration from file test.conf
debuggroup summary 0.19s found conf file test.conf
debuggroup summary 0.35s debug will appear for these features: output,summary
debuggroup summary 0.35s bitmap output image ./test.png
debuggroup summary 0.35s SVG output image ./test.svg
debuggroup summary 0.35s parsing karyotype and organizing ideograms
Use of uninitialized value $start in substitution (s///) at /home/mmiller/miniconda3/bin/../lib/Circos/Karyotype.pm line 128, line 2.
Use of uninitialized value $end in substitution (s///) at /home/mmiller/miniconda3/bin/../lib/Circos/Karyotype.pm line 129, line 2.

*** CIRCOS ERROR ***

  cwd: /home/mmiller/JupiterPlot-master

  command: /home/mmiller/miniconda3/bin/circos -noparanoid -conf test.conf

Start [undef] and/or end [undef] coordinate in karyotype file don't appear
to be numbers. Thousands separators , and _ are allowed, but not any other
characters.

There is a minor compatibility issue regarding the cp command used. Changing the -f parameter location might be all that is needed.

Hi Developer,
I am wondering how can I keep configure files that are removed at the last step of your script, which shows the information like that:
rm test.rv.links.bundled test.fw.links.bundled test.conf test.fw.links.bundled.flipped
I checked jupiter script, however, I found that you didn't use rm command in bash to delete these files. So I guess you remove these files in your perl scripts, but I don't know which one is the right one that I should modify. Can you tell me how to do that? Thanks in advance.

Alpha of links can be changed by editing the links file (changing the _a5 to something else), but this should be an option.

Hi,

just tried this pipeline to see if I could visualize synteny between two chromosome level assemblies of quite closely related fish species. I was wondering about two things:

Is there a way to get the chromosome names on the plots for the "query" assembly, as well as the reference?

Additionally, I get this issue saying some of the chromosomes - as well as large chunks of others - dont have alignments, is there a threshold of some sort that might be heightened/lowered? Im pretty certain there should be matches for all the chromosomes of species A and B.
here a snippet from the output:

Selecting 10 scaffolds to render Generating Links file chromosomes_order = scaf5,scaf1,scaf6,scaf9,scaf8,scaf7,scaf3,scaf4,scaf2,ref0,ref1,ref2,ref3,ref4,ref5,ref6,ref7,ref8,ref9,ref10,ref11,ref12,ref13,ref14,ref15,ref16,ref17,ref18,ref19,ref20,ref21,ref22,ref23,ref24 Chr4 has no alignments Chr7 has no alignments Chr8 has no alignments Chr10 has no alignments Chr12 has no alignments Chr14 has no alignments Chr15 has no alignments Chr16 has no alignments Chr17 has no alignments Chr18 has no alignments
this missing alignment occurs when I just run default, as well as when I try to run with m=100000.

Thanks for this nice tool!

all the best
S

Minimap2 can produce better gapped alignments than bwa mem, necessitating the use of scafftig splitting (as well as an agp file). It may be possible to generate an alignment with Minimap2 over the whole scaffold instead.

Dear Justin Chu and authors,

When I changed the parameters of minimap2 (from asm10 -> asm20) it helped to increase the connection, and it looks similar to result from nucmer dot plot.

However, there is only a weird connection, but it make the figure become nonsense. Please take a look these figures
bellow are results from minimap2 with asm10 and asm20

Could you please let me know how can I fix it?
P/S: I used minBundleSize=1 and maxGap=9999999999 from the previous issues #38

The pipeline runs fine until I hit this recipe

%.bed: %.sam
	grep -v XA $< | grep -v '^@' | awk '{if($$5 > 50 || $$5 == "" ) print}' | awk '{OFS="\t"; if (and($$2, 16)) print $$3,$$4,$$4+length($$10),$$1,$$5,"-"; else print $$3,$$4,$$4+length($$10),$$1,$$5,"+" }' > $@

I'm getting the error awk: line 2: function and never defined. Is and() a user defined awk function?

Hi there,
I used the JupiterPlot to present the synteny between two genomes using default parameters:
./jupiter name=genome_two ref="polished_assembly.fa" fa="C_1.0_genomic.fna" t=32 ng=75

Attached is the circos plot generated. I wonder how can I change the color and thickness of links (I want wider links) between chromosomes. In addition, how it is possible to add the names of scaffolds (or chromosomes) to second genome (right hand side of image).
Regards

Only occurs sometimes in generateConf.pl step. When printing the order of scaffolds.

Position of scaffold is determined by the median alignment position of scaffold along the sequence. This information may be useful for other applications.

I currently filter MAPQ < 50 and any alignments with an XA tag to minimize noise in the plot. These filtering options should be an option since things like repeat collapses could still be interesting for the user.

I got this circos error when I run Jupiter using own test data using Ecoli.
All perl modules are downloaded and I am stuck in this step..
How can I solve this ? I editted pathway information, because it looks messy to read.

--
bash 1_jupiter_script.sh
grep -v XA test-agp.sam | grep -v '^@' | awk '{if($5 > 50 || $5 == "" ) print}' | perl JupiterPlot-1.0/bin/samToBed.pl > test-agp.bed
samtools faidx test_scaffolds.fa
perl JupiterPlot-1.0/bin/generateConf.pl -n 75 -r JupiterPlot-1.0/config/rawConf.conf -p test -s test_scaffolds.fa.fai -b test-agp.bed -a test.agp -k test_reference.karyotype
sed: -e expression #1, char 18: unknown option to s' sed: -e expression #1, char 14: unknown option to s'
Generating Karyotype file
Selecting 2 scaffolds to render
Generating Links file
chromosomes_order = scaf1,ref0
cat test.rv.links | JupiterPlot-1.0/circos-tools-0.22/tools/bundlelinks/bin/bundlelinks -max_gap 100000 -min_bundle_size 10000 > test.rv.links.bundled
num_links 27
num_initial_bundles 15
num_links_in_initial_bundles 27
num_passed_bundles 0 (0.00%)
num_links_in_passed_bundles 0 (0.00%)
cat test.fw.links | JupiterPlot-1.0/circos-tools-0.22/tools/bundlelinks/bin/bundlelinks -max_gap 100000 -min_bundle_size 10000 > test.fw.links.bundled
num_links 265
num_initial_bundles 9
num_links_in_initial_bundles 265
num_passed_bundles 1 (11.11%)
num_links_in_passed_bundles 247 (93.21%)
awk ' { t = $5; $5 = $6; $6 = t; print; } 'test.fw.links.bundled > test.fw.links.bundled.flipped
cat test.rv.links.bundled test.fw.links.bundled.flipped > test.links.final
perl JupiterPlot-1.0/circos-0.69-3/bin/circos -noparanoid -conf test.conf
debuggroup summary 0.16s welcome to circos v0.69-3 24 Jun 2016 on Perl 5.026001
debuggroup summary 0.16s current working directory circos-test
debuggroup summary 0.16s command JupiterPlot-1.0/circos-0.69-3/bin/circos -noparanoid -conf test.conf
debuggroup summary 0.16s loading configuration from file test.conf
debuggroup summary 0.16s found conf file test.conf
debuggroup summary 0.28s debug will appear for these features: output,summary

*** CIRCOS ERROR ***

  cwd: circos-test

  command: JupiterPlot-1.0/circos-0.69-3/bin/circos
  -noparanoid -conf test.conf

Cannot find the directory [./circos-test] for
writing image files.

If you are having trouble debugging this error, first read the best practices
tutorial for helpful tips that address many common problems

  http://www.circos.ca/documentation/tutorials/reference/best_practices

The debugging facility is helpful to figure out what's happening under the
hood

  http://www.circos.ca/documentation/tutorials/configuration/debugging

If you're still stumped, get support in the Circos Google Group. Please
include this error and all your configuration and data files.

  http://groups.google.com/group/circos-data-visualization

Stack trace:
at JupiterPlot-1.0/circos-0.69-3/bin/../lib/Circos/Error.pm line 423.
Circos::Error::fatal_error("io", "no_directory", ".circos-test", "image files") called at circos-0.69-3/bin/../lib/Circos.pm line 167
Circos::run("Circos", "_argv", "-noparanoid -conf circos-test"..., "paranoid", 0, "_cwd", "circos-test", "configfile", ...) called at JupiterPlot-1.0/circos-0.69-3/bin/circos line 529
JupiterPlot-1.0/jupiter:100: recipe for target 'circos-test/test.svg' failed
make: *** [circos-test/test.svg] Error 255
rm test.links.final circos-test/test.rv.links.bundled test-agp.bed est.conf test.fw.links.bundled test_scaffolds.fa.fai test.fw.links.bundled.flipped

--

And as a result of running Jupiter, I got those outputs.
Apr 4 02:30 1_jupiter_script.sh
4.0K Apr 4 02:22 Ecoli-K12
drw4.0K Apr 4 02:23 Ecoli-O157
188 Apr 4 02:30 test.agp
5.4M Apr 4 02:30 test-agp.fa
16M Apr 4 02:30 test-agp.sam
16K Apr 4 11:40 test.fw.links
69 Apr 4 11:40 test.karyotype
124 Apr 4 02:30 test_reference.fa -> GCF_000005845.2_ASM584v2_genomic.fna
12 Apr 4 02:30 test_reference.fa.amb
39 Apr 4 02:30 test_reference.fa.ann
1.2M Apr 4 02:30 test_reference.fa.pac
2.3M Apr 4 02:30 test_reference.fa.sa
33 Apr 4 02:30 test_reference.karyotype
1.6K Apr 4 11:40 test.rv.links
125 Apr 4 02:30 test_scaffolds.fa -> GCF_000008865.2_ASM886v2_genomic.fna
30 Apr 4 11:40 test.seqOrder.txt

As you requested, I found I needed the following additional modules:
Config::General.pm
Math::VecStat
Set::IntSpan
List::MoreUtils

And those needed for CIRCOS as described here: http://www.circos.ca/documentation/tutorials/configuration/perl_and_modules/

Some repeatitive elements are removed by default and should have option to allow rendering

While I'm at it add gaps to scaffold too

Is there a way to specify where Samtools is located? I'm using a server cluster and Samtools is installed in a common (unmovable) directory.

Some contigs will be duplicated or can be placed in a graph. It would be nice somehow to place contigs as a secondary match in that location and display it as an offset in the plot.

Or

Place it within a gap, if it exists.

Dear @JustinChu,

Thanks a lot for developing this tool. I am wondering if there is an easy way to change the chromosome names in the link files to the original names of the chromosomes.

Thanks a lot,

Marc

Dear Justin,

Thanks for this amazing, super useful and straightforward tool.

I ran an analysis on two mammal genomes (both are 24 almost identical chromosomes) using the following parameters:

./jupiter name=PTru_pMa ref=../prunned_genomes/pMa.fasta fa=../prunned_genomes/pTru.fasta t=20 m=1000000 ng=99 labels=both i=24

Here I attach the png output, which I consider is a little bit weird in terms of the colors of the polygons representing the chromosomes of my reference. Do you know the reason I'm getting those colors? I know that one can set the colors manually in the karyotype file but when I see analyses from other users (as well as the example in your github) colors are different even when running default parameters. I would appreciate if you can tell me if I'm making some mistakes.

Have a good week,

Javier C.