Deal with partial diff's about osmborder HOT 1 OPEN

matthijsmelissen commented on July 20, 2024

Deal with partial diff's

from osmborder.

Comments (1)

pnorman commented on July 20, 2024

If I understand it correctly, the tool currently cannot deal with partial diffs. This means that anytime a change involving borders is made, the entire table need to be recalculated. It would be nice (but likely far from trivial) to make the tool so that it can also consume partial (minutely?) diffs.

This would require completely re-architecting the software. Given it takes longer to update a PBF than to generate the boundaries from it. I don't see this an issue.

On the WMF servers, we're looking at this code which is approximately

osmosis --read-replication-interval --write-xml-change
osmium apply-changes --fsync osm-data.osm.pbf changes.osc -o osm-data-new.osm.pbf
mv osm-data-new.osm.pbf osm-data.osm.pbf
osmborder_filter osm-filtered.osm.pbf osm-data.osm.pbf
osmborder -o osmborder_lines.csv osm-filtered.osm.pbf
psql -d ct -Xq <<EOF
DROP TABLE IF EXISTS loading.osmborder_lines;
CREATE TABLE loading.osmborder_lines (
  osm_id bigint,
  admin_level int,
  dividing_line bool,
  disputed bool,
  maritime bool,
  way Geometry(LineString, 3857));
\copy loading.osmborder_lines FROM osmborder_lines.csv
DROP TABLE IF EXISTS public.osmborder_lines;
ALTER TABLE loading.osmborder_lines SET SCHEMA public;
EOF