Hi Joao,

thanks for creating reCOGnizer and KeggCharter.

My command:
keggcharter.py -f test.tsv -tcol "value" -koc "KO" -tc "Taxonomic lineage (GENUS)" -gcol "gcol" -o test_out -mm "Methane metabolism"

My output:

2022-07-27 11:30:36: Creating KEGG Pathway representations for 1 metabolic pathways.
Some resources were not found for map [koMethane metabolism]! Going to download them
Could not download resources for [koMethane metabolism]!
Analysis of map Methane metabolism failed!
KEGGCharter analysis finished in 00h00m59s

Could you tell what is happening? Why is it attempting downloading and failing? I installed keggcharter through conda.

Kind Regards
Dany

Hello again, so soon! My input file is now not working as of the new version 1.1.0 due to the newly implemented regex checks - this is the same file I was using just a few days ago (same EC numbers, just summed qcols instead of individual ones) and so must be the syntax for the regex.

Perhaps in the EC# check you are not allowing any letters, when provisional EC#s do have "n1" (for example), which was working before you put the check in. The input file I'm using is attached. Thanks!

MirallesMetaG-unstrat-matrix-RPKM_finalTypeSortAndSummed-forKEGGCharter.txt

Hello!

I was wondering if I could please have some help in that I want to create metabolism maps for the genomes of multiple archaea.

While I got the program to work great! The problem I am having is that the maps only show that one species has a specific gene/enzyme (via a particular colour); however, when I access my excel document, my other species also have this gene.

For example, for gene EO 4.2.1.3, both of my practice genomes have these genes, but when I create metabolic maps, only one colour comes up (suggestive that only one genome has it). However, strange enough, sometimes I do get a split of two colours (hence saying that both genomes have this gene) for some genes in some pathways, but this isn't always the case.

Therefore, if I could please have some help, it would be greatly appreciated.

here is my code
keggcharter.py -f Book223.xlsx --input-quantification -koc "KO_column" -tc "Taxonomic lineage (SPECIES)" -o pratice25

And my excel
Book223.xlsx

Thank you very much

Hi Joao,

1. I might have missed this, but why is there more than 1 column for abundance?
   example: -mgc mg_column1,mg_column2

2. also, does KeggCharter expect it to be normzalized by sample size or does it expect raw abundance info?

3. is it possible to skip giving abundance and taxa info, and still get a png on a specific metabolic pathway from keggcharter?

Kind Regards
Dany

Hi, I have been trying to use the reCOGnizer output with the keggcharter and got following error message after successfully reading the data. Can you tell me whether this is a bug or a mistake on my side?

`2023-04-26 08:39:08: Arguments valid.
2023-04-26 08:39:08: Data successfully read.
Converting 27 KOs to EC numbers through the KEGG API: 100%|███████| 1/1 [00:01<00:00, 1.85s/it]
Converting 13 EC numbers to KOs through the KEGG API: 100%|███████| 1/1 [00:02<00:00, 2.16s/it]
Converting 35 EC numbers to KOs through the KEGG API: 100%|███████| 1/1 [00:02<00:00, 2.09s/it]
Traceback (most recent call last):
File "/gxfs_home/geomar/smomw539/miniconda3/envs/keggcharter/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3652, in get_loc
return self._engine.get_loc(casted_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'KO (KEGGCharter)'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/gxfs_home/geomar/smomw539/miniconda3/envs/keggcharter/bin/keggcharter.py", line 490, in
main()
File "/gxfs_home/geomar/smomw539/miniconda3/envs/keggcharter/bin/keggcharter.py", line 422, in main
data, main_column = further_information(
^^^^^^^^^^^^^^^^^^^^
File "/gxfs_home/geomar/smomw539/miniconda3/envs/keggcharter/bin/keggcharter.py", line 121, in further_information
data = get_cross_references(data, kegg_column=kegg_column, ko_column=ko_column, ec_column=ec_column, step=step)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gxfs_home/geomar/smomw539/miniconda3/envs/keggcharter/bin/keggcharter.py", line 190, in get_cross_references
data = ids_interconversion(data, column='KO (KEGGCharter)', ids_type='ko', step=step)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gxfs_home/geomar/smomw539/miniconda3/envs/keggcharter/bin/keggcharter.py", line 157, in ids_interconversion
ids = list(set(data[data[column].notnull()][column]))
~~~~^^^^^^^^
File "/gxfs_home/geomar/smomw539/miniconda3/envs/keggcharter/lib/python3.11/site-packages/pandas/core/frame.py", line 3761, in getitem
indexer = self.columns.get_loc(key)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gxfs_home/geomar/smomw539/miniconda3/envs/keggcharter/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3654, in get_loc
raise KeyError(key) from err
KeyError: 'KO (KEGGCharter)'`

Hi,

is it possible to run keggcharter on reCOGnizer output files?

If yes, can you show an example code?

Best,
Davide

Hi Joao,

If the gcol is not given:
keggcharter.py -f MOSCA_Entry_Report.xlsx -keggc "Cross-reference (KEGG)" -tcol mt_0.01a_normalized
This is the message:

2022-08-10 12:52:48: Arguments valid.
2022-08-10 12:53:03: Data successfully read.
Converting 16059 KEGG IDs to KOs through the KEGG API: 100%|████████████████████████████████████████████████████████████████████████████████| 108/108 [02:26<00:00,  1.36s/it]
Converting 2416 KOs to EC numbers through the KEGG API: 100%|█████████████████████████████████████████████████████████████████████████████████| 17/17 [00:20<00:00,  1.19s/it]
2022-08-10 12:55:52: Results saved to KEGGCharter_results/KEGGCharter_results.tsv
Traceback (most recent call last):
  File "/shared/homes/152324/miniconda3/envs/keggcharter_env/bin/keggcharter.py", line 451, in <module>
    main()
  File "/shared/homes/152324/miniconda3/envs/keggcharter_env/bin/keggcharter.py", line 415, in main
    args.genomic_columns = args.genomic_columns.split(',')
AttributeError: 'NoneType' object has no attribute 'split'

Not really intuitive. It should just say "gcol is missing"

Hello! Thanks for making what seems to be some very useful software...however, after a seamless install, I'm running into a problem getting the info from KEGG through the API on a simple first test run of data:

(base) andre@vulcan:~/MirallesProjects/CanteraMGS-CEMEX2018$ keggcharter -f MirallesMetaG-unstrat-matrix-RPKM_final.txt -o keggcharter/first_test -it 'MirallesMGS-CEMEX2018' --map-all -t 40 -ecc 'function' -qcol 'N1,N2' -mm 00680
Created keggcharter/first_test/maps
Created keggcharter/first_test/json
Created /home/andre/bin/miniconda3/share/kc_kgmls
Created /home/andre/bin/miniconda3/share/kc_csvs
2023-12-30 01:51:20: Reading input data.
2023-12-30 01:51:20: Arguments valid.
Converting 2880 EC numbers to KOs through the KEGG API:   0%|                                                                                                              | 0/72 [00:00<?, ?it/s]IDs conversion broke at index 0; Error: HTTP Error 400: Bad Request; Trying again...
IDs conversion broke at index 0 again; Error: HTTP Error 400: Bad Request
Converting 2880 EC numbers to KOs through the KEGG API:   1%|=                                                                                                     | 1/72 [00:02<03:05,  2.61s/it]IDs conversion broke at index 40; Error: HTTP Error 400: Bad Request; Trying again...
IDs conversion broke at index 40 again; Error: HTTP Error 400: Bad Request
Converting 2880 EC numbers to KOs through the KEGG API:   3%|==>                                                                                                   | 2/72 [00:05<02:54,  2.49s/it]IDs conversion broke at index 80; Error: HTTP Error 400: Bad Request; Trying again...
IDs conversion broke at index 80 again; Error: HTTP Error 400: Bad Request
Converting 2880 EC numbers to KOs through the KEGG API:   4%|====                                                                                                  | 3/72 [00:07<02:48,  2.44s/it]IDs conversion broke at index 120; Error: HTTP Error 400: Bad Request; Trying again...
IDs conversion broke at index 120 again; Error: HTTP Error 400: Bad Request
Converting 2880 EC numbers to KOs through the KEGG API:   6%|=====>                                                                                                | 4/72 [00:09<02:45,  2.43s/it]IDs conversion broke at index 160; Error: HTTP Error 400: Bad Request; Trying again...
IDs conversion broke at index 160 again; Error: HTTP Error 400: Bad Request

This is happening with the same install on our two different servers and at first I thought it might be a problem with our servers' firewalls preventing it, but I can successfully query the KEGG REST server from our servers (using example wget http://rest.genome.jp/link/hsa:56894) and get the results back. So do you think there has been a change in the way the REST API is called since you created the software? Or do you think from my side still? Thanks!

Hello again! Thanks for the quick responsiveness on my previous question. Now that I have the software working, I'm playing around with the interpretation of MGS data coming out of our MicrobiomeHelper pipeline (https://github.com/LangilleLab/microbiome_helper/wiki) that we use for our own data + offer to clients of our core - if I can get KEGGCharter to work well, we might like to include this in our new MH ver2.0 that might be coming along in 2024. We already are developing a tool for visualizing the stratified output (JarrVis: https://github.com/dhwanidesai/JarrVis) using interactive Sankey diagrams and KEGGCharter could be a nice complement to that for the metabolic maps part, since we don't have a good visualizer for that now (plus we could write some nice scripts to convert our pipeline data to "talk" between the two).

That being said, I have a few questions regarding how the quantifications are handled (PS: there also seem to be some legacy references to --genomic-columns when I think you mean -qcol) - I checked the paper and your wiki here, but there are a few things I wanted to ask and thought would be nice to have them here for other people to see. I initially started playing around with my full data file for input (only using the first two samples to start) when I encountered an issue that the color scale in the "MT" mode didn't seem to match the input RPKM values and so I made a little mock-up example file instead to be able to test and ask the below questions:

EC	N1	N2
1.7.1.4	4000	2000
1.9.6.1	400	200
1.7.2.5	40	20

...for this small test example, I've simply restricted to 3 of the EC numbers corresponding to the Nitrogen Metabolism pathway, which were in our original dataset and are of particular interest to us. After I run this through KC (keggcharter -f TestInput-forKEGGCharter.txt -o KC_test_run -it 'MirallesMGS-CEMEX2018' --map-all -t 40 -ecc 'EC' -qcol 'N1,N2' -mm 00910), I get the following output in the KEGGCharter_results.tsv:

..and then the following for the data_for_charting.tsv:

...this test data then results in the following map:

Therefore, a few questions/problems are coming up here:

1. Why are there two lines (for K02567+4561) down below the lines of the actual data in the KEGGCharter_results file above, instead of having those lines in-line with their corresponding 1.7.2.5+1.9.6.1 entries in the two lines above that to which they seemingly belong? My actual file of the full dataset has 2880 lines of actual data, whereas the KC-processed file then has an extra 1130 lines after the main data in the same way as above (ie: only in the "KO (ec-column)" and "EC (ec-column)" columns).
2. In the data_for_charting, there are data cells missing info and I suspect this is related to your having a minimum score cutoff for inclusion, but I can't seem to find that written anywhere on the Github, nor in the paper - what is it and can it be modified? We express our MGS gene counts normalized as RPKM and so the scale of numbers can look comparatively very small and so could be a problem (would prefer not to have to transform them and then redraw all the scales). This is also then reflected in the final map which does not have the last two "low count" ECs included. Similarly, in my full dataset file processed by KC (inflated from the 2880 original lines to 5123), the "KO (ec-column)" and "EC (ec-column)" columns stop at line 3819 (as does the next KO (KC) line, but the EC (KC) column goes to the end of the 5123).
3. Also in the data_for_charting, which obviously then the basis for the coloring of the final maps, you can see the 1.7.1.4 is associated with 4 KOs, therefore the lines are repeated 4 times, however the mapping is then summing those supposed 4,000 counts into 16,000 which then massively overinflates the total counts. If that EC is associated with four KOs, then at a maximum those 4,000 counts could be distributed among those 4 KOs, but not present 4,000 times in each KO - I would think the idea here, given no a priori info about which KOs they might be, would have been to evenly divide those 4,000 counts by the number of KOs to make avg. 1,000 for each, so that the final summed value would match the original, no?
4. Finally, since many of us would be using the "MT" mode for MGS data (gene counts instead of expression read depth) and the values are then absolute/comparative and not differential (ie: there is no 0 being average), the use of white as a color in the middle of the scale is problematic as it looks like "no counts" or "gene not present" for those boxes that happen to fall in that middle range. Is there a way to change the color scale used?

It is a shame about these current issues, as I had expected KC to basically be a fully automated way to get the same map coloring as doing KEGGMapper manually...however, when I use the above mock/test data in KM manually, this is what I get:

That map then faithfully reproduces the right relationships between the 4,000/2,000 counts (max out their values in each instance of 1.7.1.4 instead of multiplying), and then displays the lower 400/200 + 40/20 mock numbers. Of course, there we can also switch the color scale to avoid the white.

At the moment, I'd be forced to go this way of inputting the values manually into the KM pop-up box online instead of resorting to using KC...however, it would be nice to see if we can get it working! I don't have a ton of maps to do with my current paper I'm working on (just a few of the C/N/P ones), but still nice to see for here and eventually being able to offer to all clients/users of our pipeline. Thanks in advance for answering this long post!

Hello,

I am getting an error even when I apply this code : keggcharter.py -f test.xlsx -tcol occ -gcol gene_name -it -keggc ko -o KEGG_Test

Error message I am getting:

Created KEGG_Test
2022-11-13 10:47:35: Arguments valid.
2022-11-13 10:47:39: Data successfully read.
Converting 2376 KEGG IDs to KOs through the KEGG API: 100%|██████████| 16/16 [00:32<00:00, 2.02s/it]
Converting 0 KOs to EC numbers through the KEGG API: 0it [00:00, ?it/s]
2022-11-13 10:48:11: Results saved to KEGG_Test/KEGGCharter_results.tsv
Getting information for 2 taxa: 50%|█████ | 1/2 [00:03<00:03, 4.00s/it]
Traceback (most recent call last):
File "/ibex/scratch/alamourt/conda_keggcharter_env/bin/keggcharter.py", line 480, in
main()
File "/ibex/scratch/alamourt/conda_keggcharter_env/bin/keggcharter.py", line 450, in main
taxon_to_mmap_to_orthologs = download_resources(data, args.resources_directory, args.taxa_column, metabolic_maps)
File "bin/keggcharter.py", line 331, in download_resources
kegg_prefix = taxon2prefix(taxon, taxa_df)
File "bin/keggcharter.py", line 296, in taxon2prefix
if taxon_name.split(' (')[0] in organism_df.index: # Homo sapiens (human) -> Homo sapiens
AttributeError: 'bool' object has no attribute 'split'

I performed the example file given along with the commands given and it worked perfectly. however in this case it doesn't seem to work, can you please advise?

Hi,
I noticed that many of the IDs I wanted to map weren't included in the map, I think the reason is that in the conversion step of KO's many are lost. Why If I have already gave a KO column your program generates an "KO (KEGGCharter)" column in the results.tsv and use those to fill the map instead of the original column?

Here are the first rows of the KEGGCharter_results.tsv to explain myself better.

For instance, I wanted to draw the map m00910, Nitrogen metabolism. One of the many missing enzymes on the map that I have in the table is K00363, which is correctly converted to EC 1.7.1.15, but nonetheless in the map the corresponding box is left unfilled.

If I use the original KEGG Mapper I get the map with all the KOs correctly charted, but I liked the functionality of your software of being able to draw multiple genomes in a map. Thanks for your work.

Hello

I am using Keggcharter (v. 0.5) to compare the metabolic potential (using KEGG Orthology only) between four species of archaea. The file that I am using is attached below (Book13)Book13.xlsx.

The code I am using to run Keggcharter is keggcharter.py -f Book13.xlsx -o kcar -koc "KO_ids" -tc "Taxonomic lineage (SPECIES)" --input-quantification --metabolic-maps 00020

(I am just running the map for TCA (00020) cycle as a proof of concept for my supervisor).

While the program runs without problem when I look at my results I notice I only have 3 out of 4 species, but also none of the blocks are highlighted even though from manually searching the KEGG Orthology I do have some of the enzymes from the TCA cycle.

Therefore if I could please have some help in solving this problem it would be greatly appreciated.

Thank you very much

Hello, I am using keggcharter with the "output" that I got from reCOgnizer, but I constantly get errors, could you help me build my command line, please?

This is the command line I am using:
keggcharter -f reCOGnizer_results.tsv -ecc 'EC number' -koc 'KO' -o keggcharter_output -it "taxonomic_range_name" -iq -t 90 -mm 00680

These are the headers of the output (reCOGnizer_results.tsv) obtained with reCOgnizer:
qseqid DB ID Protein description DB description EC number CDD ID taxonomic_range_name taxonomic_range pident length mismatch gapopen qstart qend sstart send evalue bitscore General functional category Functional category KO

Hello,
i just installed keggcharter in a clean conda enviroment and cant run it.
The enviroment i installed it in is active and when i try to run it with " keggcharter -h" i get the following output:

``
/home/lfuernwein/.conda/envs/keggcharter/bin/keggcharter:9: DeprecationWarning:
Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
but was not found to be installed on your system.
If this would cause problems for you,
please provide us feedback at pandas-dev/pandas#54466

import pandas as pd
Traceback (most recent call last):
File "/home/lfuernwein/.conda/envs/keggcharter/bin/keggcharter", line 23, in
from keggpathway_map import KEGGPathwayMap, expand_by_list_column
File "/home/lfuernwein/.conda/envs/keggcharter/share/keggpathway_map.py", line 9, in
from matplotlib import pyplot as plt, colors, colormaps
ImportError: cannot import name 'colormaps' from 'matplotlib' (/home/lfuernwein/.conda/envs/keggcharter/lib/python3.10/site-packages/matplotlib/init.py)
``

I installed it as follows:
conda create -n keggcharter
conda activate keggcharter
conda install -c conda-forge -c bioconda keggcharter
i would appreciate your help and thank you in advance.

Hello,

I just read your paper showing the UPIMAPI, reCOGnizer and KEGGChart tools and found it very interesting.

I intend to test your UPIMAPI/reCOGnizer annotation pipeline in future projects, however, I have a gene catalog with 10 million proteins that was already annotated with eggnog-mapper and I have been using this annotation to make all my analysis. Is there an easy way to use visualize the eggnog-mapper results in this KEGGCharter tool?

Thank you for your time
Kind regards
Lucas

Your tool seems very useful for me, but I don't really understand what the -gcol param should specify. I tried out the example data and the example call provided in the readme, but it gives me an exception:
KeyError: "Columns not found: 'mg'"
which makes sense to me, as I also couldn't find that specific column in the example data.
Could you please clarify this for me?

Edit:

the command call I used:
python3 keggcharter.py -f MOSCA_Entry_Report.xlsx -gcol Entry -tcol mt_0.01a_normalized,mt_1a_normalized,mt_100a_normalized,mt_0.01b_normalized,mt_1b_normalized,mt_100b_normalized,mt_0.01c_normalized,mt_1c_normalized,mt_100c_normalized -keggc "Cross-reference (KEGG)" -o test_keggcharter -tc "Taxonomic lineage (GENUS)"

Hi Joao,

As I use taxonomic info obtained through kraken (GTDB) instead of UPIMAPI, I did not expect all genera to be mapped, but it seems that practically none are mapped, not even the infamous Lactobacillus. These are the messages I get for practically all genera:

[52/63] Getting information for taxon [Parabacteroides]
[0] maps inputted for org [pdi]
[0] KGMLs already obtained for org [pdi]

or

[40/63] Getting information for taxon [Anaerovibrio]
[Anaerovibrio] was not found in taxon to KEGG prefix conversion!

or

[48/63] Getting information for taxon [Lactobacillus]
[0] maps inputted for org [ljo]
[0] KGMLs already obtained for org [ljo]

What do you think could the problem be?

My command:
keggcharter.py -f test.tsv -tcol "value" -koc "KO" -tc "Taxonomic lineage (GENUS)" -gcol "gcol" -o test_out -mm "Methane metabolism"

Kind Regards
Dany