marklit / osm_split Goto Github PK
View Code? Open in Web Editor NEWFeature-specific, named OpenStreetMap (OSM) GeoPackage files
Home Page: https://tech.marksblogg.com/extracting-osm-features.html
License: MIT License
osm_split's People
Contributors
Stargazers
Watchers
osm_split's Issues
Geometry collections of points can't be read by QGIS
Consider turning these into individual point records. Use a flag so this conversion isn't the default behavior.
$ python ~/osm_split/main.py \
sweden-latest.osm.pbf \
--geom-type=other_relations \
--only-h3=820897fffffffff \
--polygon-buildingsselect * from st_read('other_relations/enforcement.gpkg') limit 1;┌─────────────┬──────────┬─────────────┬────────────────────────────────────────────────────────────────────────────────────────────────┐
│ enforcement │ maxspeed │ type │ geom │
│ varchar │ varchar │ varchar │ geometry │
├─────────────┼──────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ maxspeed │ 80 │ enforcement │ GEOMETRYCOLLECTION (POINT (16.4761211 57.7223264), POINT (16.4720156 57.7223297), POINT (16.… │
└─────────────┴──────────┴─────────────┴────────────────────────────────────────────────────────────────────────────────────────────────┘
There are some edge cases like this one where both ends of the bridge are marked with points. Figure out if there is a way to detect these and convert them into sensible geometry.
select * from st_read('other_relations/bridge.gpkg') limit 1;┌─────────┬──────────────────────────────────────────────────────────────────────────────────┐
│ type │ geom │
│ varchar │ geometry │
├─────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ bridge │ GEOMETRYCOLLECTION (POINT (16.503222 57.7692159), POINT (16.5032595 57.7691777)) │
└─────────┴──────────────────────────────────────────────────────────────────────────────────┘
Support converting buildings made of lines into polygons
Only capturing small amount of features in the split
Really nice idea for a script. But I do not manage to get it working as expected when trying it for Sweden.
I ran it first for a small part - Gotland. And I realize only a small portion of the features in the pgf-file are parsed to the gpkg-files, if I am not mistaken.
One example - this is a picture from Visby.
Brown background is the multipolygon-layer in the pgf. The colored polygons above are the gpkg-files from the output folders.
One simple example - almost no buildings are capture in the output. Do I need to go through and set up the layers to capture myself?
This is the area on OSM https://www.openstreetmap.org/relation/2640780#map=16/57.6308/18.3001
Fix H3 Filtering
Support filtering on multiple H3s
Buildings can be made from points and ways
Add support for converting these into multi-polygons.
Example: https://www.openstreetmap.org/relation/14979587
Geojson to gpkg on lines failed
I tried the new release, on the swedish data. It stoped with the following error:
Categorising (lines).. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:22:24
GeoJSON to GPKG (lines).. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:--
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ D:\gen3Data\OSM\osm_split-main\main.py:439 in main │
│ │
│ 436 │ │ │ │ # WIP: free(): invalid pointer │
│ 437 │ │ │ │ # I'm running a dodgy build from GDAL's main branch │
│ 438 │ │ │ │ if 'invalid pointer' not in str(exc): │
│ ❱ 439 │ │ │ │ │ raise exc │
│ 440 │ │ │ │
│ 441 │ │ │ unlink('%s.geojson' % basename) │
│ 442 │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ _geojson = 'lines/forest.geojson' │ │
│ │ _osm_conf = 'osmconf.ini' │ │
│ │ basename = 'lines/highway/unclassified' │ │
│ │ categorisers = { │ │
│ │ │ 'lines': <function lines at 0x0000021D715F6340>, │ │
│ │ │ 'multilinestrings': <function multilinestrings at │ │
│ │ 0x0000021D715F63E0>, │ │
│ │ │ 'multipolygons': <function multipolygons at 0x0000021D715F6480>, │ │
│ │ │ 'points': <function points at 0x0000021D715F6520>, │ │
│ │ │ 'other_relations': <function other_relations at │ │
│ │ 0x0000021D715F65C0> │ │
│ │ } │ │
│ │ category = 'forest' │ │
│ │ cmd = 'ogr2ogr -f GeoJSON lines/highway/unclassified.gpkg │ │
│ │ lines/highway/unclassified.ge'+5 │ │
│ │ file_handles = { │ │
│ │ │ 'lines/highway/unclassified.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/unclassified.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/residential.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/residential.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/tertiary.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/tertiary.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/pedestrian.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/pedestrian.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/motorway.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/motorway.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/secondary.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/secondary.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/footway.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/footway.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/service.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/service.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/cycleway.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/cycleway.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/path.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/path.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ ... +748 │ │
│ │ } │ │
│ │ folder = 'lines' │ │
│ │ geojson_filename = 'lines/highway/unclassified.geojson' │ │
│ │ geom = <LINESTRING (16.03 58.891, 16.03 58.891, 16.03 58.891, 16.029 58.891, │ │
│ │ 16.029...> │ │
│ │ geom_category = 'forest' │ │
│ │ geom_type = None │ │
│ │ geom_type_ = 'lines' │ │
│ │ h3_polys = [] │ │
│ │ num_rows = 7244803 │ │
│ │ only_h3 = None │ │
│ │ osm_file = 'sweden-latest.osm.pbf' │ │
│ │ other_tags = {'landuse': 'forest'} │ │
│ │ other_tags_no_subs = {'landuse': 'forest'} │ │
│ │ override_geom_type_ = None │ │
│ │ polygon_buildings = False │ │
│ │ rec = ( │ │
│ │ │ '"landuse"=>"forest"', │ │
│ │ │ 'LINESTRING (16.0303971 58.8908842, 16.0304777 58.8910277, │ │
│ │ 16.0301692 58.8912336,'+226 │ │
│ │ ) │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ D:\gen3Data\OSM\osm_split-main\main.py:434 in main │
│ │
│ 431 │ │ │ │ │ 'to': quote(basename + '.gpkg')} │
│ 432 │ │ │ │
│ 433 │ │ │ try: │
│ ❱ 434 │ │ │ │ execute(cmd) │
│ 435 │ │ │ except Exception as exc: │
│ 436 │ │ │ │ # WIP: free(): invalid pointer │
│ 437 │ │ │ │ # I'm running a dodgy build from GDAL's main branch │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ _geojson = 'lines/forest.geojson' │ │
│ │ _osm_conf = 'osmconf.ini' │ │
│ │ basename = 'lines/highway/unclassified' │ │
│ │ categorisers = { │ │
│ │ │ 'lines': <function lines at 0x0000021D715F6340>, │ │
│ │ │ 'multilinestrings': <function multilinestrings at │ │
│ │ 0x0000021D715F63E0>, │ │
│ │ │ 'multipolygons': <function multipolygons at 0x0000021D715F6480>, │ │
│ │ │ 'points': <function points at 0x0000021D715F6520>, │ │
│ │ │ 'other_relations': <function other_relations at │ │
│ │ 0x0000021D715F65C0> │ │
│ │ } │ │
│ │ category = 'forest' │ │
│ │ cmd = 'ogr2ogr -f GeoJSON lines/highway/unclassified.gpkg │ │
│ │ lines/highway/unclassified.ge'+5 │ │
│ │ file_handles = { │ │
│ │ │ 'lines/highway/unclassified.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/unclassified.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/residential.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/residential.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/tertiary.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/tertiary.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/pedestrian.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/pedestrian.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/motorway.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/motorway.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/secondary.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/secondary.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/footway.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/footway.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/service.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/service.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/cycleway.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/cycleway.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ 'lines/highway/path.geojson': <_io.TextIOWrapper │ │
│ │ name='lines/highway/path.geojson' mode='a' encoding='utf-8'>, │ │
│ │ │ ... +748 │ │
│ │ } │ │
│ │ folder = 'lines' │ │
│ │ geojson_filename = 'lines/highway/unclassified.geojson' │ │
│ │ geom = <LINESTRING (16.03 58.891, 16.03 58.891, 16.03 58.891, 16.029 58.891, │ │
│ │ 16.029...> │ │
│ │ geom_category = 'forest' │ │
│ │ geom_type = None │ │
│ │ geom_type_ = 'lines' │ │
│ │ h3_polys = [] │ │
│ │ num_rows = 7244803 │ │
│ │ only_h3 = None │ │
│ │ osm_file = 'sweden-latest.osm.pbf' │ │
│ │ other_tags = {'landuse': 'forest'} │ │
│ │ other_tags_no_subs = {'landuse': 'forest'} │ │
│ │ override_geom_type_ = None │ │
│ │ polygon_buildings = False │ │
│ │ rec = ( │ │
│ │ │ '"landuse"=>"forest"', │ │
│ │ │ 'LINESTRING (16.0303971 58.8908842, 16.0304777 58.8910277, │ │
│ │ 16.0301692 58.8912336,'+226 │ │
│ │ ) │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ C:\Users\web\AppData\Local\Programs\Python\Python311\Lib\site-packages\shpyx\runner.py:316 in │
│ run │
│ │
│ 313 │ │ result.return_code = p.returncode │
│ 314 │ │ │
│ 315 │ │ # Verify that the command result is valid, based on the verification configurati │
│ ❱ 316 │ │ self._verify_result( │
│ 317 │ │ │ result=result, │
│ 318 │ │ │ verify_return_code=verify_return_code, │
│ 319 │ │ │ verify_stderr=verify_stderr, │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ args = 'ogr2ogr -f GeoJSON lines/highway/unclassified.gpkg │ │
│ │ lines/highway/unclassified.ge'+5 │ │
│ │ cmd_env = { │ │
│ │ │ 'ALLUSERSPROFILE': 'C:\\ProgramData', │ │
│ │ │ 'APPDATA': 'C:\\Users\\web\\AppData\\Roaming', │ │
│ │ │ 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', │ │
│ │ │ 'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files', │ │
│ │ │ 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files', │ │
│ │ │ 'COMPUTERNAME': 'MASKINEN', │ │
│ │ │ 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe', │ │
│ │ │ 'DOKANLIBRARY1': 'C:\\Program Files\\Dokan\\Dokan Library-1.5.1\\', │ │
│ │ │ 'DOKANLIBRARY1_LIBRARYPATH_X64': 'C:\\Program Files\\Dokan\\Dokan │ │
│ │ Library-1.5.1\\lib\\', │ │
│ │ │ 'DOKANLIBRARY1_LIBRARYPATH_X86': 'C:\\Program Files\\Dokan\\Dokan │ │
│ │ Library-1.5.1\\x86\\lib\\', │ │
│ │ │ ... +37 │ │
│ │ } │ │
│ │ cmd_str = 'ogr2ogr -f GeoJSON lines/highway/unclassified.gpkg │ │
│ │ lines/highway/unclassified.ge'+5 │ │
│ │ env = None │ │
│ │ exec_dir = None │ │
│ │ final_stderr = b'' │ │
│ │ final_stdout = b'' │ │
│ │ log_cmd = None │ │
│ │ log_output = None │ │
│ │ p = <Popen: returncode: 3221225781 args: 'ogr2ogr -f GeoJSON │ │
│ │ lines/highway/uncla...> │ │
│ │ result = ShellCmdResult( │ │
│ │ │ cmd='ogr2ogr -f GeoJSON lines/highway/unclassified.gpkg │ │
│ │ lines/highway/unclassified.ge'+5, │ │
│ │ │ stdout='', │ │
│ │ │ stderr='', │ │
│ │ │ all_output='', │ │
│ │ │ return_code=3221225781 │ │
│ │ ) │ │
│ │ self = <shpyx.runner.Runner object at 0x0000021D712E5C10> │ │
│ │ stderr_data = b'' │ │
│ │ stdout_data = b'' │ │
│ │ tmp_file = <tempfile._TemporaryFileWrapper object at 0x0000021D064ED090> │ │
│ │ unix_raw = False │ │
│ │ use_shell = True │ │
│ │ use_signal_names = None │ │
│ │ verify_return_code = None │ │
│ │ verify_stderr = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ C:\Users\web\AppData\Local\Programs\Python\Python311\Lib\site-packages\shpyx\runner.py:181 in │
│ _verify_result │
│ │
│ 178 │ │ │ │ f"Error output:\n{result.stderr}\n" │
│ 179 │ │ │ │ f"All output:\n{result.all_output}" │
│ 180 │ │ │ ) │
│ ❱ 181 │ │ │ raise ShpyxVerificationError(reason=reason, result=result) │
│ 182 │ │
│ 183 │ def run( │
│ 184 │ │ self, │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ reason = "The command 'ogr2ogr -f GeoJSON lines/highway/unclassified.gpkg │ │
│ │ lines/highway/un"+84 │ │
│ │ result = ShellCmdResult( │ │
│ │ │ cmd='ogr2ogr -f GeoJSON lines/highway/unclassified.gpkg │ │
│ │ lines/highway/unclassified.ge'+5, │ │
│ │ │ stdout='', │ │
│ │ │ stderr='', │ │
│ │ │ all_output='', │ │
│ │ │ return_code=3221225781 │ │
│ │ ) │ │
│ │ return_code_str = '3221225781' │ │
│ │ self = <shpyx.runner.Runner object at 0x0000021D712E5C10> │ │
│ │ success = False │ │
│ │ use_signal_names = None │ │
│ │ verify_return_code = None │ │
│ │ verify_stderr = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ShpyxVerificationError: The command 'ogr2ogr -f GeoJSON lines/highway/unclassified.gpkg lines/highway/unclassified.geojson' failed with return code 3221225781.
Error output:
All output:
I have now a folder lines with geojson-files. No other output.
Geometry Collections in GPKG files unrecognised in QGIS
I've raised qgis/QGIS#56587
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.