I ran the pipeline and everything worked up until til the leaflet generation which failed with pandoc document conversion failed with error 137 , any idea what this could be about?
Seems like it was the actual export of the Cloropleth.html file the errored

  Loading Biodiverse results
  ..Observed indices
  ..SES-scores
  ..P-values
  Preparing data for CANAPE analysis
  ..Running the CANAPE analysis
  ...Inferring endemism type
  Estimating sampling redundancy
  ..Loading a file with the total number of GBIF-records per H3-cell
  ..Adding N records to the main table
  ..Estimating the index
  Exporting the data table
  Preparing gridcell polygons
  ..Adding diversity estimates to polygons
  ..Adding H3 cell names
  ..Fixing antimeridian issue
  Creating leaflet map
  ..Generating polygon labels
  ..Building basemap
  ..Preparing color palettes
  ..Adding polygons
  ...  RICHNESS_ALL 
  ...  PD 
  ...  SES_PD 
  ...  PD_P 
  ...  ENDW_WE 
  ...  SES_ENDW_WE 
  ...  PE_WE 
  ...  SES_PE_WE 
  ... CANAPE
  ... Redundancy
  ..Adding variable selector
  ..Hiding variables
  ..Exporting the results
Command error:
  Killed
  Error: pandoc document conversion failed with error 137
  Execution halted

On the map of phylogenetic diversity, the legend has percentages of phylogenetic diversity, but when you hover over a cell it shows the absolute value of PD. It is not clear what the percentages are based on.

The legend shows intervals of percentages (we are looking at PD), but these intervals seem to vary between 4 or 5 levels for different runs, which makes it hard to compare them.

These 2 phylogenies are with and without Wollemia nobilis. We ran phylonext for both of these phylogenies. All parameters were default except for type of phylogenetic tree labels (Latin) family (Araucariaceae), min year (1980) and country (Australia), number of randomisation (20).

The problem is that some cells where Wollemia nobilis is not present have different amounts of PD between the 2 runs. We would expect if Wollemia nobilis is not present, they would have exactly the same amount of PD.

These are the two phylogenies we ran:

(((Agathis_australis,Agathis_palmerstonii,((((Agathis_atropurpurea,Agathis_microstachya),Agathis_robusta),((Agathis_dammara,Agathis_borneensis),(Agathis_silbae,Agathis_macrophylla))),((Agathis_moorei,(Agathis_ovata,Agathis_corbassonii)),Agathis_lanceolata)),Agathis_montana,Agathis_obtusa,Agathis_vitiensis,Agathis_brownii,Agathis_alba,Agathis_philippinensis,Agathis_flavescens,Agathis_orbicula,Agathis_kinabaluensis,Agathis_labillardierei,Agathis_lenticula,Agathis_endertii,Agathis_silbaii)Agathis,(Wollemia_nobilis)Wollemia),(((Araucaria_heterophylla,((((Araucaria_columnaris,(Araucaria_luxurians,Araucaria_nemorosa)),((Araucaria_humboldtensis,Araucaria_laubenfelsii),Araucaria_rulei)),((Araucaria_bernieri,(Araucaria_schmidii,Araucaria_subulata)),Araucaria_scopulorum)),(Araucaria_biramulata,Araucaria_montana))),(Araucaria_cunninghamii_var._cunninghamii)Araucaria_cunninghamii),((Araucaria_araucana,Araucaria_angustifolia),(Araucaria_bidwillii,Araucaria_hunsteinii)),Araucaria_excelsa,Araucaria_muelleri,Araucaria_braziliana,Araucaria_lignitici,Araucaria_bernieri,Araucaria_annulata,Araucaria_balcombensis,Araucaria_fimbriatus,Araucaria_grandifolia,Araucaria_hastiensis,Araucaria_longifolia,Araucaria_marensii,Araucaria_nathorsti,Araucaria_planus,Araucaria_prominens,Araucaria_readiae,Araucaria_taeriensis,Araucaria_uncinatus,Araucaria_araucoensis,Araucaria_brownii,Araucaria_jeffreyi,Araucaria_marylandica,Araucaria_neocookii,Araucaria_pichileufensis,Araucaria_prodromus)Araucaria,(Agathoxylon_pseudoparenchymatosum)Agathoxylon,(Balmeisporites_minutus)Balmeisporites)Araucariaceae;

((Agathis_australis,Agathis_palmerstonii,((((Agathis_atropurpurea,Agathis_microstachya),Agathis_robusta),((Agathis_dammara,Agathis_borneensis),(Agathis_silbae,Agathis_macrophylla))),((Agathis_moorei,(Agathis_ovata,Agathis_corbassonii)),Agathis_lanceolata)),Agathis_montana,Agathis_obtusa,Agathis_vitiensis,Agathis_brownii,Agathis_alba,Agathis_philippinensis,Agathis_flavescens,Agathis_orbicula,Agathis_kinabaluensis,Agathis_labillardierei,Agathis_lenticula,Agathis_endertii,Agathis_silbaii)Agathis,(((Araucaria_heterophylla,((((Araucaria_columnaris,(Araucaria_luxurians,Araucaria_nemorosa)),((Araucaria_humboldtensis,Araucaria_laubenfelsii),Araucaria_rulei)),((Araucaria_bernieri,(Araucaria_schmidii,Araucaria_subulata)),Araucaria_scopulorum)),(Araucaria_biramulata,Araucaria_montana))),(Araucaria_cunninghamii_var._cunninghamii)Araucaria_cunninghamii),((Araucaria_araucana,Araucaria_angustifolia),(Araucaria_bidwillii,Araucaria_hunsteinii)),Araucaria_excelsa,Araucaria_muelleri,Araucaria_braziliana,Araucaria_lignitici,Araucaria_bernieri,Araucaria_annulata,Araucaria_balcombensis,Araucaria_fimbriatus,Araucaria_grandifolia,Araucaria_hastiensis,Araucaria_longifolia,Araucaria_marensii,Araucaria_nathorsti,Araucaria_planus,Araucaria_prominens,Araucaria_readiae,Araucaria_taeriensis,Araucaria_uncinatus,Araucaria_araucoensis,Araucaria_brownii,Araucaria_jeffreyi,Araucaria_marylandica,Araucaria_neocookii,Araucaria_pichileufensis,Araucaria_prodromus)Araucaria,(Agathoxylon_pseudoparenchymatosum)Agathoxylon,(Balmeisporites_minutus)Balmeisporites)Araucariaceae;

I have a couple of questions

1. Is it correct that omitting the --indices param will make the pipeline default to all indices?
2. If so, and the pipeline is started with only a few indices, like --indices calc_richness,calc_phylo_rpd1 is it correct that the user then also needs to adjust the --leaflet_var param, otherwise some input data will be missing for that step?

If this is correctly understood, is it possible to get a mapping of --leaflet_var values to which indices they depend on?

Best,
Thomas

When you look at the execution report, you can not tell which run it is from, so it would be useful to have the title and description of the run in the execution report, and just the title on the map.

I was able to run a PhyloNext analysis on GBIF using the Acacia_Mishler,2014_Latin-labels.nwk tree on Australia without any problems: https://phylonext.gbif.org/run/eeb8f380-5716-4e47-bb72-2ba6de29da91

I have been trying to use PhyloNext to use the synthetic Open Tree to look up patterns of distribution in the genus Erica in Africa. However, this consistently fails for me (see e.g. https://phylonext.gbif.org/run/d8c5b4ea-abb7-455d-9184-1d53403b2e95). Is this because I neither chose a OToL tree in the "Choose a predefined tree from OTL" field on the form nor entered a phylogeny in the "Phylogenetic tree in newick format (phytree param)" field? I assumed that if I left both of these fields empty, then PhyloNext would look up the occurrence data as per the phylogenetic filters and then pick up the minimum tree for those taxa from OpenTree. Is that incorrect? If leaving these two fields empty is what caused the Internal Server Error in the results, then the form should make that clearer (by making one of those fields required, or by adding a note to indicate that one of those fields need to be filled in); otherwise, let me know and I'll send you the pipeline arguments I provided for that run so we can figure out why it's resulting in an Internal Server Error.

Multiple colour gradients are for when you have above and below a treshold, which is not the case for PD and richness. A single colour gradient would be more appropriate here.

• Switch to the p-ranks for Z-score categorization (instead of absolute values).
  For details, see https://biodiverse-analysis-software.blogspot.com/2023/04/changes-to-randomisation-results-p-rank.html

• Automatically select Biodiverse subroutines given a diversity index

• Automatically fetch the latest container version for CI tests.
  E.g., somehow set the value for this line based on a variable or container configs:
  

  PhyloNext/.github/workflows/Biodiverse_test.yml

  Line 10 in 8a6399d

  image: vmikk/biodiverse:1.0.0

• Set up test caches

• Add unit tests (see nf-core/tools#605)

• Find a way how to estimate coverage for "heterogeneous" code (R, Python, Perl, and Groovy)

• Add digital signature to Singularity images

• Update Biodiverse to 4.3 (released 01 May 2023)

• Update Biodiverse to version 4 (released 26 November 2022!)

• Add CI tests

• OpenTree filtering - use tree tips with phylogeny support only

• Add JSON schema describing pipeline parameters

In order to cite the filtered data used for a pipeline run, we would need a simple csv with two columns datasetKey and occurrenceCount.
This will allow a user (or the web GUI) to register a derived dataset with a citable DOI.
I suggest to add it to the pipeline_info output directory.

• Data filtering by occurrence issues not implemented.
  Currently, Apache Arrow does not support filtering data using columns with array-type data (e.g., the issue column in the GBIF occurrence dump). See ARROW-16702 [apache/arrow issue 31991] and ARROW-16641 [apache/arrow issue 32045] . It is possible to filter using DuckDB, but the query consumes >100 GB RAM.

• Estimation of PD for grid cells with a single species

• Estimation of memory required for each process not implemented
  (could be useful if the pipeline will be launched on HPC or in the cloud)
  Given the size of the input, it is possible to guess the amount of RAM required for a task.
  But we need to collect the data for various use-case scenarios to make a raw estimate.

Map shows legend with colour intervals on user interface, but the generated pdf shows gradients. These should be made the same.

It would be nice to get an overall value for PD in the results considering all data (without spatial component). The reason we want to do this, is to understand phylogenetic diversity loss when we submit multiple different phylogenies.

Currently there is not much data, but in the long term we want to encourage people to indicate whether species are domestic or cultivated. A filter for establishment means would be useful.