In docker/scripts/r2gg_pipeline.sh there is no check of step result before going to next step.

It will be better if the script stop in case of error.

Python package safety returns some vulnerability:

+==============================================================================+

                                   /$$$$$$            /$$
                                  /$$__  $$          | $$
               /$$$$$$$  /$$$$$$ | $$  \__//$$$$$$  /$$$$$$   /$$   /$$
              /$$_____/ |____  $$| $$$$   /$$__  $$|_  $$_/  | $$  | $$
             |  $$$$$$   /$$$$$$$| $$_/  | $$$$$$$$  | $$    | $$  | $$
              \____  $$ /$$__  $$| $$    | $$_____/  | $$ /$$| $$  | $$
              /$$$$$$$/|  $$$$$$$| $$    |  $$$$$$$  |  $$$$/|  $$$$$$$
             |_______/  \_______/|__/     \_______/   \___/   \____  $$
                                                              /$$  | $$
                                                             |  $$$$$$/
      by pyup.io                                              \______/

+==============================================================================+

 REPORT 

  Safety is using PyUp's free open-source vulnerability database. This
data is 30 days old and limited. 
  For real-time enhanced vulnerability data, fix recommendations, severity
reporting, cybersecurity support, team and project policy management and more
sign up at https://pyup.io or email [email protected]

  Safety v2.3.5 is scanning for Vulnerabilities...
  Scanning dependencies in your environment:

  -> /home/jmkerloch/dev/route-graph-generator/.venv/lib/python3.10/site-packages
  -> /home/jmkerloch/dev/route-graph-generator

  Using non-commercial database
  Found and scanned 76 packages
  Timestamp 2023-06-29 08:06:39
  8 vulnerabilities found
  0 vulnerabilities ignored

+==============================================================================+
 VULNERABILITIES FOUND
+==============================================================================+

-> Vulnerability found in sqlparse version 0.4.2
   Vulnerability ID: 55054
   Affected spec: >=0.1.15,<0.4.4
   ADVISORY: Sqlparse 0.4.4 includes a fix for CVE-2023-30608: Parser
   contains a regular expression that is vulnerable to ReDOS (Regular...
   CVE-2023-30608
   For more information, please visit https://pyup.io/v/55054/f17


-> Vulnerability found in setuptools version 45.2.0
   Vulnerability ID: 52495
   Affected spec: <65.5.1
   ADVISORY: Python Packaging Authority (PyPA) setuptools before 65.5.1
   allows remote attackers to cause a denial of service via HTML in a crafted...
   CVE-2022-40897
   For more information, please visit https://pyup.io/v/52495/f17


-> Vulnerability found in requests version 2.28.2
   Vulnerability ID: 58755
   Affected spec: >=2.3.0,<2.31.0
   ADVISORY: Requests is a HTTP library. Since Requests 2.3.0, Requests
   has been leaking Proxy-Authorization headers to destination servers when...
   CVE-2023-32681
   For more information, please visit https://pyup.io/v/58755/f17


-> Vulnerability found in markdown-it-py version 2.1.0
   Vulnerability ID: 54650
   Affected spec: >=0,<2.2.0
   ADVISORY: Denial of service could be caused to markdown-it-py, before
   v2.2.0, if an attacker was allowed to force null assertions with specially...
   CVE-2023-26303
   For more information, please visit https://pyup.io/v/54650/f17


-> Vulnerability found in markdown-it-py version 2.1.0
   Vulnerability ID: 54651
   Affected spec: >=0,<2.2.0
   ADVISORY: Denial of service could be caused to the command line
   interface of markdown-it-py, before v2.2.0, if an attacker was allowed to...
   CVE-2023-26302
   For more information, please visit https://pyup.io/v/54651/f17


-> Vulnerability found in lxml version 4.6.4
   Vulnerability ID: 43366
   Affected spec: <4.6.5
   ADVISORY: Lxml 4.6.5 includes a fix for CVE-2021-43818: Prior to
   version 4.6.5, the HTML Cleaner in lxml.html lets certain crafted script...
   CVE-2021-43818
   For more information, please visit https://pyup.io/v/43366/f17


-> Vulnerability found in lxml version 4.6.4
   Vulnerability ID: 50748
   Affected spec: <4.9.1
   ADVISORY: Lxml 4.9.1 includes a fix for CVE-2022-2309: NULL Pointer
   Dereference allows attackers to cause a denial of service (or application...
   CVE-2022-2309
   For more information, please visit https://pyup.io/v/50748/f17


-> Vulnerability found in cryptography version 39.0.0
   Vulnerability ID: 53048
   Affected spec: >=1.8,<39.0.1
   ADVISORY: Cryptography 39.0.1 includes a fix for CVE-2023-23931: In
   affected versions 'Cipher.update_into' would accept Python objects which...
   CVE-2023-23931
   For more information, please visit https://pyup.io/v/53048/f17

 Scan was completed. 8 vulnerabilities were found. 

+==============================================================================+
   REMEDIATIONS

  8 vulnerabilities were found in 6 packages. For detailed remediation & fix 
  recommendations, upgrade to a commercial license. 

+==============================================================================+

  Safety is using PyUp's free open-source vulnerability database. This
data is 30 days old and limited. 
  For real-time enhanced vulnerability data, fix recommendations, severity
reporting, cybersecurity support, team and project policy management and more
sign up at https://pyup.io or email [email protected]

+==============================================================================+

We should try to update some requirements.

We need to define a JSON-schema describing configuration file.

This JSON-schema could be use to check configuration file instead of crash when a properties is not available.

In current version, r2gg external package are imported in r2gg/_init__.py

This must be changed so we can import __about__.py while building documentation.

• refactor r2gg/__init__.py to remove import of base package => move method to a cli.py file
• update 'setup.py' for new entry point

Create a new .github/workflows/package-publish.yml to add package publication to CI.

• define pypi user that will be used for the publication
• add secrets in GitHub settings to store pypi user and password
• define github action used for pypi publication

Documentation should be available on GitHub pages:

• add GitHub action workflow to generate documentation
• step1 : generate documentation with sphinx (waiting #7)
• step2 : deploy generated documentation on GitHub pages

          Ici psycopg 2 est utilisé alors que dans le reste de l'outillage de la GPF on utilise la v3 ? 

AMHA cela représente une source de confusion et de petites erreurs difficiles à identifier dans le futur.

Originally posted by @Guts in #38 (comment)

@azarz @lgrd All python files are prefixed with _.

Is there any reason for that ?

I'm afraid this kind of naming could be misinterpreted by some parser. It may be better to remove all prefix.

IN IGN Geoplateforme, recommanded python version is 3.10.

We need to check if there are any issue with python 3.10 use and update docker image.

Feature

Be able to do partial updates from OSM data

Is your feature request related to a problem? Please describe.

When I want to update my graph, I have to make a delete and replace.

Describe the solution you'd like

I would like to be able to do partial updates from OSM data.

Describe alternatives you've considered

No alternatives considered.

Additional context

None

Currently the only sql conversion script is for the BD Uni, however many users will want to start the conversion from a local instance of the BD Topo.

A sql script for the conversion from BD Topo should be added.

There are no test in the repository.

We need to add some end to end tests for all entry point:

• r2gg:sql2pivot
• r2gg:pivot2pgrouting
• r2gg:pivot2osm
• r2gg:osm2osrm
• r2gg:osm2valhalla
• r2gg:road2config

TO BE DEFINED :

• test framework (suggestion pytest)
• how to have input data ?
• how to check results ?

Add some value for setup() in setup.py for package publication.

• author
• author_email
• description
• package
• ...

It will be better to store all these value in a __about__.py file

See https://packaging.python.org/guides/single-sourcing-package-version/

          Hi ! I've just seen that schemas are not used for the pivot in _pivot_to_osm.py and _pivot_to_pgr.py. They are used only for the final pgr base. So, if we don't change that, your modification will not be used in the next step of the generation. :/

Do you want to make the changes ? For information, I can help, but only in one or two weeks. Now, I have to make continue my work on the admin API.

Originally posted by @lgrd in #33 (comment)

Should we update repository name to indicate road2 ?

For example : road2-route-graph-generator

We need to have some data available to test end to end pivot base creation :

For this we could use bdtopo from IGN : https://geoservices.ign.fr/bdtopo with table:
- troncon_de_route
- non_communication

This should be stored as a .sql dump.

This dump could then be used in a docker-compose so the input database can be available in test.

Add documentation with sphynx:

• create a root directory documentation
• add conf.py to define sphynx option
• TO BE DISCUSSED : move all .md documentation files to the `documentation directory
• create landing page documentation/index.md

Docker image should not contains commands (only bash).

Need to update :

• docker/Dockerfile : remove r2gg_pipeline.sh call
• docker/Dockerfile: use requirements/base.txt to install dependancies and sync their installation across the project

⚠️ road2 must also be updated to adapt docker-compose.yml : r2gg_pipeline.sh call must be added (issue to be created)

(Version française en fin de ticket!)

I am trying to create a route that passes through the lanes contained within a polygon (IRIS), to create a map that will support a field survey (a pedestrian team survey route).

• The route must pass through each route at least once;
• The route should be as short as possible.
• The route can eventually leave the polygon and use the adjacent lanes to return to the polygon by another route if this allows optimisation.

This corresponds to the problem of the Chinese postman: https://fr.wikipedia.org/wiki/Probl%C3%A8me_du_postier_chinois

pgRouting seems to be able to calculate these routes: https://docs.pgrouting.org/latest/en/chinesePostmanProblem-family.html.
The data I want to work with is the troncon_de_route layer of the BD TOPO. If I'm not mistaken, I'll have to use route-graph-generator to generate the graphs.

I followed the r2gg README and already made progress. Here is what I've done so far:

• The troncon_de_route layer has been imported into a PostGIS database, on which the postgres_fdw and pgRouting extensions have also been installed;
• The procedures https://github.com/IGNF/pgrouting-procedures are also installed;
• I have adapted the general config file config.json to fit my project. My version is available here: https://mapper.space/tests/r2gg/config.json
• The other config files have also been customized (for log management, database connection management, and cost calculation);
• I finally tried to customize bduni_convert.sql but without much success and this is where I get stuck. Running r2gg-pivot2pgrouting config.json gets me numerous SQL errors. Completely modifying the SQL conversion script without errors seems to be beyond my skills at the moment.

I have several questions:

• Am I on the right track?
• Do you have any guidance on what to include in the SQL script? I don't need to get the attribute data from road_section, geometry and georeferencing info would be enough.
• Do you know of any other solution, if possible simpler, to achieve what I am trying to do? (I have already tried the Chinese Postman plugin from QGIS but the plugin has outdated dependencies. I filed a ticket a week ago, with no news yet, and I haven't tried to modify the plugin to fix the dependency issues. It could be a a solution but for the moment I'm focused on r2gg).

Thanks in advance for your feedback :)

Je cherche à créer un itinéraire qui passe par les voies contenues au sein d'un polygone (IRIS), pour créer une carte qui servira d'appui à une enquête de terrain (itinéraire d'enquête d'une équipe piétonne).

• L'itinéraire doit passer par chaque voie au moins une fois;
• L'itinéraire doit être le plus court possible.
• L'itinéraire peut éventuellement sortir du polygone et emprunter les voies adjacentes pour revenir dans le polygone par un autre chemin si ça permet d'optimiser.

Ça correspond au problème du postier chinois: https://fr.wikipedia.org/wiki/Probl%C3%A8me_du_postier_chinois

pgRouting semble être en mesure de calculer ces itinéraires: https://docs.pgrouting.org/latest/en/chinesePostmanProblem-family.html.
Les données à partir desquelles je souhaite travailler correspondent à la couche troncon_de_route de la BD TOPO. Si je ne fais pas erreur, il faut donc que je passe par route-graph-generator pour générer les graphes à partir de la BD TOPO.

J'ai suivi le README et déjà avancé. Voilà où j'en suis:

• Les données troncon_de_route sont importées dans une base PostGIS, sur laquelle les extensions postgres_fdw et pgRouting ont été également installées;
• Les procédures https://github.com/IGNF/pgrouting-procedures sont installées elles aussi;
• J'ai adapté le fichier de config générale config.json pour correspondre à mon projet. Ma version est consultable ici: https://mapper.space/tests/r2gg/config.json
• Les autres fichiers de config ont également été personnalisés (pour la gestion des logs, la gestion des connexions aux bases de données, et pour le calcul des coûts);
• J'ai enfin tenté de personnaliser bduni_convert.sql mais en tatonnant, et c'est ici que ça bloque. Lorsque je lance r2gg-pivot2pgrouting config.json, j'ai de nombreuses erreurs lors de l'éxécution du SQL. La modification complète et sans erreur du script de conversion SQL semble être au-delà de mes compétences pour l'instant.

Je me pose donc plusieurs questions:

• Est-ce que je suis sur la bonne voie?
• Auriez-vous des indications sur ce qu'il faut indiquer dans le script SQL? Je n'ai pas besoin de récupérer les données attributaires de troncon_de_route, la géométrie et le géoréférencement suffiraient.
• Auriez-vous connaissance d'une autre solution, si possible plus simple, pour réaliser ce que j'essaye de faire?
  (J'ai déjà essayé le plugin Chinese Postman de QGIS mais le plugin a des problèmes de dépendances obsolètes. J'ai déposé un ticket il y a une semaine, sans nouvelles pour l'instant, et je n'ai pas essayé de modifier le plugin pour en corriger les problèmes de dépendances. C'est une solution mais pour le moment je me suis concentré sur r2gg.)

Merci d'avance pour vos retours :)

Dockerfile docker/debian/Dockerfile should be updated to use requirements.txt

Hello ! I think we will have to use this file inside the dockerfile because the dependancies are outdated now and not sync with those requirements. ^^'

Originally posted by @lgrd in #22 (comment)

Need

r2gg can create a .osm from the pivot. And r2gg can read them to create osrm or valhalla data. But it can't convert an osm data to pivot.

Functionalities

• Download an osm file
• Convert an osm file to the pivot format
• Be able to make an entire generation from osm to pgr, osrm and valhalla

Tasks

• Update the generation file format
• Be sure of the CLI and its IO
• Code it
• Update and run tests
• Update docs (changelog and documentation directory) of this project
• Update contributing.md of this project
• Update contributing.md of Road2
• Update readme.md and docs of Road2

Code

// TODO

Create a new .github/workflows/docker-publish.yml to add docker publication to CI.

Use https://github.com/marketplace/actions/build-and-push-docker-images to build and push docker image

Docker image should be available in GitHub Container Registry

• use docker GitHub Actions to build and push

J'ai pu remplir la base de donnée pivot et je travaille sur la création de la base de données pour pgRouting avec la commande r2gg-pivot2pgrouting. Lorsque je lance cette commande, l'ensemble du script s'exécute correctement jusqu'au moment où la fonction public.clean_graph() est utilisée :

A ce moment, j'ai le message d'erreur suivant :

Traceback (most recent call last):
  File "/home/frozar/.local/bin/r2gg-pivot2pgrouting", line 33, in <module>
    sys.exit(load_entry_point('r2gg', 'console_scripts', 'r2gg-pivot2pgrouting')())
  File "/home/frozar/wk/flaxib/wk/route-graph-generator/r2gg/__init__.py", line 12, in pivot2pgrouting
    pgr_convert(config, resource, db_configs, connection, logger)
  File "/home/frozar/wk/flaxib/wk/route-graph-generator/r2gg/_main.py", line 132, in pgr_convert
    pivot_to_pgr(resource, cost_calculation_file_path, connection, connection_out, logger)
  File "/home/frozar/wk/flaxib/wk/route-graph-generator/r2gg/_pivot_to_pgr.py", line 384, in pivot_to_pgr
    cursor_isolated.execute(clean_graph_query)
psycopg2.errors.UndefinedFunction: function public.clean_graph(unknown) does not exist
LINE 1: SELECT public.clean_graph('pedestrian')

En cherchant dans le projet pour savoir où est défini la fonction public.clean_graph(), je ne trouve rien.

Je pense donc que la définition de cette fonction n'est pas dans ce projet.

Si tel est le cas, pouvez-vous rajouter la définition de cette fonction dans le fichier _pivot_to_pgr.py svp ?

To check cost_calculation file content we should use a JSON-schema to validate file.

First step is to create a JSON-schema describing expected structure.

Create a new .github/workflows/linter.yml to add lint to CI.