tooledesign / pybna Goto Github PK
View Code? Open in Web Editor NEWPython module to implement BNA logic in a PostGIS database
License: GNU General Public License v3.0
Python module to implement BNA logic in a PostGIS database
License: GNU General Public License v3.0
Category | Original BNA | pybna (normal parking) | pybna (low parking) |
---|---|---|---|
Overall | 50 | 38 | 46 |
People | 56 | 43 | 53 |
Core Services | 49 | 27 | 36 |
Dentists | 38 | 15 | 21 |
Doctors | 39 | 18 | 23 |
Grocery | 56 | 33 | 43 |
Hospitals | 49 | 12 | 13 |
Pharmacies | 41 | 22 | 27 |
Social Services | 60 | 27 | 35 |
Opportunity | 50 | 35 | 42 |
Employment | 44 | 31 | 38 |
Higher Education | 54 | 25 | 26 |
K12 Education | 54 | 37 | 46 |
Tech/Vocational College | 42 | 8 | 13 |
Recreation | 65 | 36 | 39 |
Community Centers | 43 | 19 | 25 |
Parks | 75 | 67 | 72 |
Trails | 69 | n/a* | n/a* |
Retail | 56 | 39 | 47 |
Transit | 24 | 6 | 9 |
* In the past we have discussed a better version of the trails score. I've left it out completely here but it would be easy to add a measure for access to trailheads--which is what we've discussed in the past--as long as it can be taken from OSM.
Currently these lines
don't handle bad formulations of numbers when coalescing to text array, e.g. 18'4"
This can be solved with the following code in a WHERE block of that statement
WHERE (NOT (osm."cycleway" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway" ISNULL)
AND (NOT (osm."cycleway:left" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:left" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:left" ISNULL)
AND (NOT (osm."cycleway:right" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:right" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:right" ISNULL)
AND (NOT (osm."oneway:bicycle" ~ '^[0-9]+[''][0-9]+' OR osm."oneway:bicycle" ~ '^[0-9]+[\"][0-9]+') OR osm."oneway:bicycle" ISNULL)
AND (NOT (osm."cycleway:both" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:both" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:both" ISNULL)
AND (NOT (osm."cycleway:buffer" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:buffer" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:buffer" ISNULL)
AND (NOT (osm."cycleway:left:buffer" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:left:buffer" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:left:buffer" ISNULL)
AND (NOT (osm."cycleway:right:buffer" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:right:buffer" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:right:buffer" ISNULL)
AND (NOT (osm."cycleway:both:buffer" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:both:buffer" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:both:buffer" ISNULL)
AND (NOT (osm."cycleway:width" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:width" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:width" ISNULL)
AND (NOT (osm."cycleway:left:width" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:left:width" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:left:width" ISNULL)
AND (NOT (osm."cycleway:right:width" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:right:width" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:right:width" ISNULL)
AND (NOT (osm."cycleway:both:width" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:both:width" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:both:width" ISNULL)
This filters out any records where the tag fields have numbers separated by a ' or ", single quote or double quote.
pybna scores a bike lane adjacent to parking with one travel lane as high stress
as discussed in #48. If there are two travel lanes it still gets a low stress
. This doesn't make sense.
caused by running list_scenarios()
Retrieves network data before building a new network. Must fix!
P Furth's tables have bike lanes on one-ways being treated differently than on two-ways. Current LTS logic includes this. Need to remove so we're matching BNA charts.
Add a check to the process that imports Census blocks that removes blocks covered 100% by water. This should be implemented with an arg to import_census_blocks (e.g. remove_water=True) so that it doesn't have to happen if a user doesn't want it to.
Hi spencerrecneps@ I was using the python pybna 1.0.2 library to calculate the stress for Portland, Oregon. I have overwritten the boundary shape file, the library successfully read the boundary shape file, imported census blocks, and received census jobs data using the below line of codes.
City Boundary is available here: https://gis-pdx.opendata.arcgis.com/datasets/1559e31273654eb9858397861f1fdefa/explore?location=45.629289%2C-122.551444%2C10.44
'''
i = pybna.Importer(config="config.yaml")
i.import_boundary('city_of_portland_boundary.shp', overwrite=True)
i.import_census_blocks(fips=41, overwrite=True)
i.import_census_jobs("received.neighborhood_census_block_jobs",state="OR", overwrite=True)
'''
C:\Users\mxm5116.conda\envs\geo_env\lib\site-packages\pybna\importer.py:58: ResourceWarning: unclosed file <_io.TextIOWrapper name='config.yaml' mode='r' encoding='cp1252'>
self.config = self.parse_config(yaml.safe_load(open(config)))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Connecting to database
Copying boundary to database
Loading boundary
C:\Users\mxm5116.conda\envs\geo_env\lib\site-packages\pandas\io\sql.py:761: UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy
warnings.warn(
Loading data from http://www2.census.gov/geo/tiger/TIGER2010BLKPOPHU/tabblock2010_41_pophu.zip
Filtering blocks to boundary
Filtering out water
Copying blocks to database
Loading data for state OR
No data for state OR for year 2020. Checking previous year.
C:\Users\mxm5116.conda\envs\geo_env\lib\site-packages\pybna\importer.py:351: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
jobs = jobs_main.append(jobs_aux,ignore_index=True)
Copying jobs to database
After the above lines of code, when I was trying to import osm network using the below line of code, I encountered the error-
'''
i.import_osm_network()
i.import_osm_destinations()
'''
Downloading OSM data
Copying OSM ways to database
Copying OSM intersections to database
Processing OSM data in database
'''
InvalidTextRepresentation Traceback (most recent call last)
Input In [13], in <cell line: 1>()
----> 1 i.import_osm_network()
2 i.import_osm_destinations()
File ~.conda\envs\geo_env\lib\site-packages\pybna\importer.py:473, in Importer.import_osm_network(self, roads_table, ints_table, boundary_file, boundary_buffer, osm_file, keep_holding_tables, srid, km, overwrite)
462 self.gdf_to_postgis(
463 nodes,
464 osm_nodes_table,
(...)
468 conn=conn
469 )
471 conn.commit()
--> 473 self._process_osm(
474 roads_table,roads_schema,ints_table,ints_schema,osm_ways_table,
475 osm_ways_schema,osm_nodes_table,osm_nodes_schema,srid,km,overwrite,conn
476 )
478 conn.commit()
479 conn.close()
File ~.conda\envs\geo_env\lib\site-packages\pybna\importer.py:566, in Importer._process_osm(self, roads_table, roads_schema, ints_table, ints_schema, osm_ways_table, osm_ways_schema, osm_nodes_table, osm_nodes_schema, srid, km, overwrite, conn)
564 query = self.read_sql_from_file(fquery)
565 q = sql.SQL(query).format(**subs)
--> 566 cur.execute(q)
567 cur.close()
569 if commit:
InvalidTextRepresentation: malformed array literal: "{6'0"}"
DETAIL: Unexpected array element.
'''
I am using the first time of this library. Any help to solve this issue is greatly appreciated!
Config has been fleshed out and sql script is ready. Just need to hook up inputs from config and run SQL script.
Line 540 in 6110a73
Hey All,
Looks like 2020 blocks are available with pop and housing info:
http://www2.census.gov/geo/tiger/TIGER2020/TABBLOCK20/
Should we consider updating import_census_blocks() from this:
Line 232 in 6110a73
to this:
src = "http://www2.census.gov/geo/tiger/TIGER2020/TABBLOCK20/tl_2020_" + fips + "_tabblock20.zip"
allow to add a scenario from the config file by name. this way a user can easily load one scenario without having to load all
This would allow for absolute or relative scoring of destination access. Current default is relative (points are dependent on the number of high stress destinations, so if there's only one school in the area you get full points if you can get to it on low stress, even though you only have access to one school.)
Allowing relative vs absolute allows to define categories where access is important regardless of what you can do on the low stress network. So you could get punished for not having low stress access to a grocery store, even though there isn't a grocery store anywhere near you.
Can be built as an entry in the config file for each destination.
@beckymasond
This is out of scope for the current slate of BNA changes, but after thinking more about #48 I realized the best solution might be to allow for targeted assumptions to be applied at a neighborhood level.
This could be operationalized by using a standardized shapefile. The idea is that users would draw boundaries within which a certain set of assumptions would apply. For example, I could use neighborhood boundaries from my city and have a different speed limit assumption for each neighborhood. In many cities, the downtown and inner neighborhoods generally have lower speeds while newer areas with wider roads and a more suburban pattern have higher speeds. At present, we apply a single assumption for the entire area.
We could still use city-wide assumptions if no shapefile is provided, but I could imagine this would be an additional input prepared by cities when they provide their other inputs for the online BNA. Those who wish to get the more nuanced treatment would be expected to create boundaries and enter appropriate assumptions for things like speed, number of lanes, parking, etc.
In the process of running the import commands for parker, I got this error/traceback.
Connecting to database
Downloading OSM data
Traceback (most recent call last):
File "bna.py", line 12, in <module>
i.import_osm_network()
File "/home/hkelley/code/B0084_Parker_Ped_Bike_Analysis/pybnavenv/lib/python3.6/site-packages/pybna/importer.py", line 442, in import_osm_network
ways, nodes = self._osm_net_from_osmnx(boundary,osm_file)
File "/home/hkelley/code/B0084_Parker_Ped_Bike_Analysis/pybnavenv/lib/python3.6/site-packages/pybna/importer.py", line 675, in _osm_net_from_osmnx
ox.config(useful_tags_node=node_tags,useful_tags_path=way_tags)
TypeError: config() got an unexpected keyword argument 'useful_tags_path'
The OSMnx changelog says "rename settings useful_tags_path to the more appropriate useful_tags_way" for 0.14.1
I can submit a pr for changing the argument name if you think it's as simple as that. It's this line.
Sort of a weak reason to add a breaking change...
There seems to be a slight discrepancy in the number of destinations being imported between importing from a PBF file and overpass directly. It does not seem to be limited to multiple queries. In general, the overpass import rows seem to be slightly less in number than PBF imports. The number is pretty small and may not affect BNA results much, but something to look into for later. I suspect it has something to do with PBF potentially having historic locations as well as current ones. This is only a guess based on spot checking and will need to be explored more to be certain.
The test code which shows the discrepancy is here
Current LTS settings cause living streets to get LTS 3 because of ADT
Current implementation runs a shortest path for each node connected to a block. Need to investigate improvement by adding a temporary node with edges that connect to each node associated with the block. This lets us run the network search one time instead of multiple times. Could be an order of magnitude improvement on processor time.
Example:
Block A has nodes 1,2,3,4. Testing connectivity to Block B with nodes 4,5,6,7
Current implementation:
Shortest path from 1 to 4, 2 to 4...1 to 5, 2 to 5...4 to 7 and then compare each
Better implementation:
Add temporary node -1 with edges that connect to 1,2,3,4 and cost 0
Add temporary node -2 with edges that connect to 4,5,6,7 and cost 0
Shortest path from -1 to -2
Breaking change: osmid is no longer retained for nodes
One significant driver of different results: I discovered a bug in the old BNA that caused the 25% detour to be ignored.
https://github.com/azavea/pfb-network-connectivity/blob/51dc18ae5b31f9016fd7161abd8b18ce6b015eb2/src/analysis/connectivity/connected_census_blocks.sql#L35
(The issue is It saves the low stress distance for both low and high stress in the connected_blocks table)
I'm calling this as an enhancement in pybna since it fixes a problem in the old BNA.
Nodes have to be updated manually, but the rebuild should automatically happen anytime network is rebuilt.
Mostly complete via #53
Still need to check tables against imperial tables but waiting to finalize LTS checks and make updates to the imperial tables
Current osmnx version is not compatible with geopandas>0.6.3 due to changes in the way projection strings are handled in geopandas. For now, we need to specify geopandas==0.6.3 in the requirements file.
Change travel shed output to something more meaningful with a column like "status" and options
connected
connected but doesn't meet directness requirements
not connected
I think the connectivity matrix encodes this information already with the difference between connected but doesn't meet...
and not connected
being that not connected
is null but doesn't meet
is false.
If a non-standard schema is specified in config.yaml
for the stress lookup_tables, an error is returned when creating a new instance of pybna.Stress()
.
Default:
Lines 211 to 215 in 14a2766
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pybna\pybna\stress.py", line 60, in __init__
self._create_lookup_table(*t)
File "pybna\pybna\stress.py", line 181, in _create_lookup_table
cur.copy_from(f,table,columns=[c[0] for c in columns],sep=";",null="")
psycopg2.errors.UndefinedTable: relation "stress_shared" does not exist
Old BNA: Tertiary road with default parking (8 ft) + bike lane (5 ft) gets low stress
because 13 ft reach is considered low stress. This is consistent with P Furth's tables.
pyBNA: Tertiary road with default parking (8 ft) + bike lane (5 ft) gets high stress
. I'm not convinced 13 ft reach should be considered low stress on a busier street with adjacent parking. A lot depends on context. If parking usage is moderate this can be stressful since you have to be more concerned with door zone conflicts and turnover. If parking usage is low this is low stress. This is something we've introduced in some of our projects for clients. Our modified tables in this case require 14 ft (8 ft parking + 6 ft bike lane--effectively a buffered bike lane) in order to be low stress.
I think this comes down to what we want to assume. The pyBNA tables are likely to render most bike lanes in most cities high stress
since we assume parking if there's no OSM data making it explicit. Based on my experience in Chicago, parking-adjacent bike lanes are high stress
facilities. Based on my experience in Madison, parking-adjacent bike lanes in residential areas are low stress
facilities since parking isn't as heavy and you just needed the bike lane to pass the occasional parked car.
We could also consider introducing a configuration parameter that lets the user set an assumption for parking usage. e.g. you could set city-wide parking usage to low
and get one result or high
to get another result in these locations.
Add methods for QC. Examples:
Shapely version 2 became the latest release back in December.
https://pypi.org/project/shapely/#history
That version does not provide direct access to geos, the way previous versions did, which breaks OSMNx v1.x because of the change needed from this commit
The current version of osmnx in pypi is 1.3
https://pypi.org/project/osmnx/
when I installed pybna with pip install git+ssh://[email protected]/tooledesign/pybna.git@dev
I got shapely v 2.1 and OSMNx v 1.2.2.
The error upon running import pybna
is:
/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/pybna/conf.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
from collections import Mapping
Traceback (most recent call last):
File "07_import_name_ref/02_import.py", line 1, in <module>
import pybna
File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/pybna/__init__.py", line 3, in <module>
from .importer import Importer
File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/pybna/importer.py", line 10, in <module>
import osmnx as ox
File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/osmnx/__init__.py", line 3, in <module>
from ._api import *
File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/osmnx/_api.py", line 17, in <module>
from .geometries import geometries_from_address
File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/osmnx/geometries.py", line 21, in <module>
from shapely.geos import TopologicalError
ImportError: cannot import name 'TopologicalError' from 'shapely.geos' (/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/shapely/geos.py)
I solved this with a new virtual environment and installing with the commands
pip install wheel
pip install shapely==1.8.5.post1
pip install git+ssh://[email protected]/tooledesign/pybna.git@dev
Alternatively, we could try using OSMNx v2.x but I'm assuming there's a good chance of running into other problems we dont' ahve time to debug.
We need to compare pybna results against a comparable BNA run using the old BNA method. I've downloaded results from the website for Madison, WI. The following tables are in the received schema of the bna DB on the Portland server:
Our process should proceed as follows:
We can document any issues or discrepancies we come across here. Ultimately, we need to be able to catalog any differences as either a regression (needs fixing), a neutral change (difference that results in no substantial effect), or an enhancement (fixes a problem with the original)
Test case is in B042-denver_moves_downtown
When it tries to make tiles, calling the ._split_tiles() method isn't working. Cites that it takes exactly 10 arguments and is only getting 8.
Seems to assign defaults even if non-default settings are used in pybna.
bna_settings.zip
Network build chokes if there are values in one_way that don't represent forward or backward.
build a monolithic dictionary of sql substitutions that can be used across all queries
Switch to a select into...where false
to allow for different data types on the block id column (currently requires varchar)
User shouldn't have to calculate that on each destination table
title says it all - annoying having to hand delete when something fails
When there are multiple tags in config.yaml
for a destination category, the import_osm_destinations()
method only seems to import destinations that match just the last tag. This does not seem to be an issue for imports done from an OSM file.
PyBNA still seems to employ graph_tool even in the structural change branch. I can't import pybna as is.
Need to test for detour % when determining connectivity
Reading in the network from DB expects "source_vert" and other standardized column names. Need to be parameterized so we can use attributes described in config.
Need to verify that lane data on oneway multi-lane streets is not being divided in half.
Early on in the BNA we decided to include crosswalks in the network to handle cases where a trail crossing across a road was represented with the OSM tags highway=crossing
and crossing=footway
. I think this ultimately causes for confusion for people who view the LTS scores because they end up seeing all the little crosswalk stubs.
I propose that we require an additional bicycle=yes
or bicycle=designated
tag for a crosswalk to be imported from OSM. This will reduce crosswalks in the network only to those that actually serve the bicycle network.
Thoughts, @beckymasond ?
should check for
I agree, this is something I have been meaning to improve for the BNA's importer module. Feel free to open an issue on the BNA repo for this.
Originally posted by @theja in https://github.com/tooledesign/50793_Anne_Arundel_Bike_Ped_Master_Plan/pull/2#discussion_r822811508
A street table with null geoms gums up the network build. Check for nulls and kick an error if any are found.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.