Here's the error in the log:

[2021-05-31 13:46:09] 27935 Done with import
[2021-05-31 13:50:01] 28938 start import from seq-nr 4564062, replag is 18 hour(s)
[2021-05-31 13:50:01] 28938 downloading diff
[2021-05-31 13:52:22] 28938 filtering diff
Traceback (most recent call last):
File "/home/renderaccount/src/regional/trim_osc.py", line 154, in
for nd in root.xpath('//way[@id={}]/nd'.format(row[0])):
File "src/lxml/etree.pyx", line 1577, in lxml.etree._Element.xpath
File "src/lxml/xpath.pxi", line 307, in lxml.etree.XPathElementEvaluator.call
File "src/lxml/xpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_result
lxml.etree.XPathEvalError: Invalid expression
[2021-05-31 13:52:38] 28938 [error] Trim_osc error
[2021-05-31 13:52:38] 28938 resetting state

I saved the offending file at https://map.atownsend.org.uk/tmp/changes.osc_save.gz (21Mb).
I can't see any obvious problems why it would cause a problem - xmllint seems happy with it (it only changes quotes), and there are no oddities in the file like modified ways with zero nodes.

It's just diffs from this one time (seen on two different machines with very different database contents) - later or earlier diffs do not have a problem.

When trying to run trim_osc.py, I get the following error:

$ ./trim_osc.py -p canada.poly changes.osc changes-canada.osc
Traceback (most recent call last):
  File "shapely/speedups/_speedups.pyx", line 234, in shapely.speedups._speedups.geos_linearring_from_py
AttributeError: 'list' object has no attribute '__array_interface__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./trim_osc.py", line 82, in <module>
    tpoly = poly_parse(options.poly)
  File "./trim_osc.py", line 42, in poly_parse
    result = Polygon(poly)
  File "/usr/lib/python3/dist-packages/shapely/geometry/polygon.py", line 240, in __init__
    ret = geos_polygon_from_py(shell, holes)
  File "/usr/lib/python3/dist-packages/shapely/geometry/polygon.py", line 494, in geos_polygon_from_py
    ret = geos_linearring_from_py(shell)
  File "shapely/speedups/_speedups.pyx", line 319, in shapely.speedups._speedups.geos_linearring_from_py
TypeError: object of type 'map' has no len()

Looks like the problem occurs while parsing this poly file. Shapely comes from the latest Ubuntu 18.04 packages.

Any clue?

Working environment is Ubuntu 18.04 LTS

According to this guide, I'm trying to trim my downloaded OSM changes.

After executing sudo apt-get install -y python-psycopg2 python-shapely python-lxml, there were problems with versions of these libraries, which are solved by these commands:

sudo apt-get install python3-psycopg2
sudo apt-get -y install python3-pip
pip3 install lxml
pip3 install shapely

Now, executing the script ~/src/regional/trim_osc.py -d gis --user postgres --host localhost --port 5432 --password -p "${WORKOSM_DIR}/andorra.poly" osmChange osmChangeTrimmed, we have this error:

Traceback (most recent call last):
File "shapely/speedups/_speedups.pyx", line 234, in shapely.speedups._speedups.geos_linearring_from_py
AttributeError: 'list' object has no attribute 'array_interface'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rosm/src/regional/trim_osc.py", line 82, in
tpoly = poly_parse(options.poly)
File "/home/rosm/src/regional/trim_osc.py", line 42, in poly_parse
result = Polygon(poly)
File "/home/rosm/.local/lib/python3.6/site-packages/shapely/geometry/polygon.py", line 240, in init
ret = geos_polygon_from_py(shell, holes)
File "/home/rosm/.local/lib/python3.6/site-packages/shapely/geometry/polygon.py", line 494, in geos_polygon_from_py
ret = geos_linearring_from_py(shell)
File "shapely/speedups/_speedups.pyx", line 319, in shapely.speedups._speedups.geos_linearring_from_py
TypeError: object of type 'map' has no len()

How can I solve this? Maybe prerequisites for executing this script should be provided, because not all the users who want it are professional Python developers.

I have simple world.zip osm changes file, which I want trim by corresponding script, by specifying Andorra's poly file.

Here is my command for that ~/src/regional/trim_osc.py -d gis --user postgres --host localhost --port 5432 --password -p "${WORKOSM_DIR}/andorra.poly" world.osm worldTrimmed.osm.

After executing that command, worldTrimmed.osm is lesser than world.osm, but after checking coordinates in file, I see that there are still coordinates outside Andorra's bounds.

In fact, sample file:

<?xml version='1.0' encoding='UTF-8'?>
<osmChange version="0.6" generator="Osmosis 0.47">
  <modify>
    <node id="60359977" version="3" timestamp="2019-08-09T08:07:49Z" uid="3909835" user="mysampleuser" changeset="73182495" lat="42.776481" lon="9.4764622">
      <tag k="crossing" v="uncontrolled"/>
      <tag k="highway" v="crossing"/>
    </node>
    <node id="60359982" version="5" timestamp="2019-08-09T08:07:49Z" uid="3909835" user="mysampleuser" changeset="73182495" lat="42.494126" lon="1.585888">
      <tag k="crossing" v="controlled"/>
      <tag k="highway" v="crossing"/>
    </node>
  </modify>
</osmChange>

is filtered well, and only node with id 60359982 lasts.

What's wrong with that scenario and how can I fix that?

The script reads only "1" polygon.

During the import, I get the following error:

[2014-11-14 11:30:57] 14425 filtering diff
Traceback (most recent call last):
  File "/tmp/trim_osc.py", line 131, in <module>
    for nd in root.xpath('//way[@id={}]/nd'.format(row[0])):
  File "lxml.etree.pyx", line 1509, in lxml.etree._Element.xpath (src/lxml/lxml.
  File "xpath.pxi", line 318, in lxml.etree.XPathElementEvaluator.__call__ (src/                                                                                                                     .c:145954)
  File "xpath.pxi", line 238, in lxml.etree._XPathEvaluatorBase._handle_result (                                                                                                                     tree.c:144962)
  File "xpath.pxi", line 224, in lxml.etree._XPathEvaluatorBase._raise_eval_erro                                                                                                                     l.etree.c:144817)

I installed "python-lxml 3.3.3", but on another computer it works with this version. I always work with the same region "Germany".

Many thanks!
Christian

When using a full planet import, the trim_osc.py script isn't necessary. When not using a full planet import, you need the trim to prevent data out of bounds to be added to the database by updates.

But even when not usind a full planet import, depending on the use case, it might be beneficial to use flat nodes (e.g. when using continents).

The problem with the trim_osc.py when using flat nodes is that the database table PREFIX_nodes doesn't exist. But it is always accessed by the script to check if a node exists in the database or not (so that it is not trimmed if it exists, e.g. for partly out of bounds polygons).

Because the database table doesn't exists, the script will always fail with an error:

[2022-05-23 11:45:14] 4708 filtering diff
Traceback (most recent call last):
  File "/home/renderer/src/regional/trim_osc.py", line 129, in <module>
    cur.execute(q1, (nodesM,))
psycopg2.errors.UndefinedTable: relation "planet_osm_nodes" does not exist
LINE 1: select id from planet_osm_nodes where id = ANY(ARRAY[4937988... 

What's needed is an additional parameter for the trim_osc.py script, that let you provide a path to the binary flat nodes file. When provided, the script should no longer try to access the nodes table in the database to check wether a node exists, but instead should look into the flat nodes file.

Note: This might not be an easy change. The flat node file is a very big binary file. But with knowledge of the internal file structure it should be possible to calculate the exact position for any osm_id inside the file and to read out its value directly.

I built an OSM tile server and was in the process of incorporating updates when I ran into two issues (so far). With the help of others we tentatively resolved two issues.

The first resolution involved changing the argparse.FileType to rb vice r. It is totally unclear to me what changed, I assume with the data input file.

The second issue involved python3 vs python2. There was a use of long that I had to change to int.

While tempted to do a PR, I simply don't know the code or system well enough to be confident these are the right fixes? And, I'm still running the update at this time.

The script drops relations which are well within the data extract to keep.