tooledesign / pybna Goto Github PK

* In the past we have discussed a better version of the trails score. I've left it out completely here but it would be easy to add a measure for access to trailheads--which is what we've discussed in the past--as long as it can be taken from OSM.

Currently these lines

don't handle bad formulations of numbers when coalescing to text array, e.g. 18'4"

This can be solved with the following code in a WHERE block of that statement

WHERE (NOT (osm."cycleway" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway" ISNULL)
    AND (NOT (osm."cycleway:left" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:left" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:left" ISNULL)
    AND (NOT (osm."cycleway:right" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:right" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:right" ISNULL)
    AND (NOT (osm."oneway:bicycle" ~ '^[0-9]+[''][0-9]+' OR osm."oneway:bicycle" ~ '^[0-9]+[\"][0-9]+') OR osm."oneway:bicycle" ISNULL)
    AND (NOT (osm."cycleway:both" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:both" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:both" ISNULL)
    AND (NOT (osm."cycleway:buffer" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:buffer" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:buffer" ISNULL)
    AND (NOT (osm."cycleway:left:buffer" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:left:buffer" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:left:buffer" ISNULL)
    AND (NOT (osm."cycleway:right:buffer" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:right:buffer" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:right:buffer" ISNULL)
    AND (NOT (osm."cycleway:both:buffer" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:both:buffer" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:both:buffer" ISNULL)
    AND (NOT (osm."cycleway:width" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:width" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:width" ISNULL)
    AND (NOT (osm."cycleway:left:width" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:left:width" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:left:width" ISNULL)
    AND (NOT (osm."cycleway:right:width" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:right:width" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:right:width" ISNULL)
    AND (NOT (osm."cycleway:both:width" ~ '^[0-9]+[''][0-9]+' OR osm."cycleway:both:width" ~ '^[0-9]+[\"][0-9]+') OR osm."cycleway:both:width" ISNULL)

This filters out any records where the tag fields have numbers separated by a ' or ", single quote or double quote.

pybna scores a bike lane adjacent to parking with one travel lane as high stress as discussed in #48. If there are two travel lanes it still gets a low stress. This doesn't make sense.

caused by running list_scenarios()

Retrieves network data before building a new network. Must fix!

P Furth's tables have bike lanes on one-ways being treated differently than on two-ways. Current LTS logic includes this. Need to remove so we're matching BNA charts.

Add a check to the process that imports Census blocks that removes blocks covered 100% by water. This should be implemented with an arg to import_census_blocks (e.g. remove_water=True) so that it doesn't have to happen if a user doesn't want it to.

Hi spencerrecneps@ I was using the python pybna 1.0.2 library to calculate the stress for Portland, Oregon. I have overwritten the boundary shape file, the library successfully read the boundary shape file, imported census blocks, and received census jobs data using the below line of codes.

City Boundary is available here: https://gis-pdx.opendata.arcgis.com/datasets/1559e31273654eb9858397861f1fdefa/explore?location=45.629289%2C-122.551444%2C10.44

'''
i = pybna.Importer(config="config.yaml")
i.import_boundary('city_of_portland_boundary.shp', overwrite=True)
i.import_census_blocks(fips=41, overwrite=True)
i.import_census_jobs("received.neighborhood_census_block_jobs",state="OR", overwrite=True)
'''

C:\Users\mxm5116.conda\envs\geo_env\lib\site-packages\pybna\importer.py:58: ResourceWarning: unclosed file <_io.TextIOWrapper name='config.yaml' mode='r' encoding='cp1252'>
self.config = self.parse_config(yaml.safe_load(open(config)))
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Connecting to database
Copying boundary to database
Loading boundary

C:\Users\mxm5116.conda\envs\geo_env\lib\site-packages\pandas\io\sql.py:761: UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy
warnings.warn(

Loading data from http://www2.census.gov/geo/tiger/TIGER2010BLKPOPHU/tabblock2010_41_pophu.zip
Filtering blocks to boundary
Filtering out water
Copying blocks to database
Loading data for state OR
No data for state OR for year 2020. Checking previous year.

C:\Users\mxm5116.conda\envs\geo_env\lib\site-packages\pybna\importer.py:351: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
jobs = jobs_main.append(jobs_aux,ignore_index=True)

Copying jobs to database

After the above lines of code, when I was trying to import osm network using the below line of code, I encountered the error-
'''
i.import_osm_network()
i.import_osm_destinations()
'''

Downloading OSM data

Copying OSM ways to database

Copying OSM intersections to database

Processing OSM data in database

'''
InvalidTextRepresentation Traceback (most recent call last)
Input In [13], in <cell line: 1>()
----> 1 i.import_osm_network()
2 i.import_osm_destinations()

File ~.conda\envs\geo_env\lib\site-packages\pybna\importer.py:473, in Importer.import_osm_network(self, roads_table, ints_table, boundary_file, boundary_buffer, osm_file, keep_holding_tables, srid, km, overwrite)
462 self.gdf_to_postgis(
463 nodes,
464 osm_nodes_table,
(...)
468 conn=conn
469 )
471 conn.commit()
--> 473 self._process_osm(
474 roads_table,roads_schema,ints_table,ints_schema,osm_ways_table,
475 osm_ways_schema,osm_nodes_table,osm_nodes_schema,srid,km,overwrite,conn
476 )
478 conn.commit()
479 conn.close()

File ~.conda\envs\geo_env\lib\site-packages\pybna\importer.py:566, in Importer._process_osm(self, roads_table, roads_schema, ints_table, ints_schema, osm_ways_table, osm_ways_schema, osm_nodes_table, osm_nodes_schema, srid, km, overwrite, conn)
564 query = self.read_sql_from_file(fquery)
565 q = sql.SQL(query).format(**subs)
--> 566 cur.execute(q)
567 cur.close()
569 if commit:

InvalidTextRepresentation: malformed array literal: "{6'0"}"
DETAIL: Unexpected array element.
'''

I am using the first time of this library. Any help to solve this issue is greatly appreciated!

Config has been fleshed out and sql script is ready. Just need to hook up inputs from config and run SQL script.

Hey All,

Looks like 2020 blocks are available with pop and housing info:
http://www2.census.gov/geo/tiger/TIGER2020/TABBLOCK20/

Should we consider updating import_census_blocks() from this:

to this:
src = "http://www2.census.gov/geo/tiger/TIGER2020/TABBLOCK20/tl_2020_" + fips + "_tabblock20.zip"

allow to add a scenario from the config file by name. this way a user can easily load one scenario without having to load all

This would allow for absolute or relative scoring of destination access. Current default is relative (points are dependent on the number of high stress destinations, so if there's only one school in the area you get full points if you can get to it on low stress, even though you only have access to one school.)

Allowing relative vs absolute allows to define categories where access is important regardless of what you can do on the low stress network. So you could get punished for not having low stress access to a grocery store, even though there isn't a grocery store anywhere near you.

Can be built as an entry in the config file for each destination.

@beckymasond
This is out of scope for the current slate of BNA changes, but after thinking more about #48 I realized the best solution might be to allow for targeted assumptions to be applied at a neighborhood level.

This could be operationalized by using a standardized shapefile. The idea is that users would draw boundaries within which a certain set of assumptions would apply. For example, I could use neighborhood boundaries from my city and have a different speed limit assumption for each neighborhood. In many cities, the downtown and inner neighborhoods generally have lower speeds while newer areas with wider roads and a more suburban pattern have higher speeds. At present, we apply a single assumption for the entire area.

We could still use city-wide assumptions if no shapefile is provided, but I could imagine this would be an additional input prepared by cities when they provide their other inputs for the online BNA. Those who wish to get the more nuanced treatment would be expected to create boundaries and enter appropriate assumptions for things like speed, number of lanes, parking, etc.

In the process of running the import commands for parker, I got this error/traceback.

Connecting to database
Downloading OSM data
Traceback (most recent call last):
  File "bna.py", line 12, in <module>
    i.import_osm_network()
  File "/home/hkelley/code/B0084_Parker_Ped_Bike_Analysis/pybnavenv/lib/python3.6/site-packages/pybna/importer.py", line 442, in import_osm_network
    ways, nodes = self._osm_net_from_osmnx(boundary,osm_file)
  File "/home/hkelley/code/B0084_Parker_Ped_Bike_Analysis/pybnavenv/lib/python3.6/site-packages/pybna/importer.py", line 675, in _osm_net_from_osmnx
    ox.config(useful_tags_node=node_tags,useful_tags_path=way_tags)
TypeError: config() got an unexpected keyword argument 'useful_tags_path'

The OSMnx changelog says "rename settings useful_tags_path to the more appropriate useful_tags_way" for 0.14.1

I can submit a pr for changing the argument name if you think it's as simple as that. It's this line.

Sort of a weak reason to add a breaking change...

There seems to be a slight discrepancy in the number of destinations being imported between importing from a PBF file and overpass directly. It does not seem to be limited to multiple queries. In general, the overpass import rows seem to be slightly less in number than PBF imports. The number is pretty small and may not affect BNA results much, but something to look into for later. I suspect it has something to do with PBF potentially having historic locations as well as current ones. This is only a guess based on spot checking and will need to be explored more to be certain.

The test code which shows the discrepancy is here

• Remove 100% water blocks via #37
• Allow default state and city residential speed limits (already enabled by the config file)
• Remove hiking and other pedestrian trails that prohibit bikes via #41
• Correct handling of one-way multilane streets #58
• Expand bicycle permissions for footways and service roads #42
• Update LODES data 6afcb2c
• Allow for metric measurements being handled in #54
• Expand retail tagging options via fe3a009

Current LTS settings cause living streets to get LTS 3 because of ADT

Current implementation runs a shortest path for each node connected to a block. Need to investigate improvement by adding a temporary node with edges that connect to each node associated with the block. This lets us run the network search one time instead of multiple times. Could be an order of magnitude improvement on processor time.

Example:
Block A has nodes 1,2,3,4. Testing connectivity to Block B with nodes 4,5,6,7
Current implementation:
Shortest path from 1 to 4, 2 to 4...1 to 5, 2 to 5...4 to 7 and then compare each
Better implementation:
Add temporary node -1 with edges that connect to 1,2,3,4 and cost 0
Add temporary node -2 with edges that connect to 4,5,6,7 and cost 0
Shortest path from -1 to -2

Breaking change: osmid is no longer retained for nodes

attempted fix with 706ef97
but needed to revert (b2e9b88) due to extreme slowness in db operations.

One significant driver of different results: I discovered a bug in the old BNA that caused the 25% detour to be ignored.
https://github.com/azavea/pfb-network-connectivity/blob/51dc18ae5b31f9016fd7161abd8b18ce6b015eb2/src/analysis/connectivity/connected_census_blocks.sql#L35

(The issue is It saves the low stress distance for both low and high stress in the connected_blocks table)

I'm calling this as an enhancement in pybna since it fixes a problem in the old BNA.

Nodes have to be updated manually, but the rebuild should automatically happen anytime network is rebuilt.

Mostly complete via #53
Still need to check tables against imperial tables but waiting to finalize LTS checks and make updates to the imperial tables

Current osmnx version is not compatible with geopandas>0.6.3 due to changes in the way projection strings are handled in geopandas. For now, we need to specify geopandas==0.6.3 in the requirements file.

Change travel shed output to something more meaningful with a column like "status" and options

• connected
• connected but doesn't meet directness requirements
• not connected

I think the connectivity matrix encodes this information already with the difference between connected but doesn't meet... and not connected being that not connected is null but doesn't meet is false.

If a non-standard schema is specified in config.yaml for the stress lookup_tables, an error is returned when creating a new instance of pybna.Stress().

Default:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pybna\pybna\stress.py", line 60, in __init__
    self._create_lookup_table(*t)
  File "pybna\pybna\stress.py", line 181, in _create_lookup_table
    cur.copy_from(f,table,columns=[c[0] for c in columns],sep=";",null="")
psycopg2.errors.UndefinedTable: relation "stress_shared" does not exist

Old BNA: Tertiary road with default parking (8 ft) + bike lane (5 ft) gets low stress because 13 ft reach is considered low stress. This is consistent with P Furth's tables.

pyBNA: Tertiary road with default parking (8 ft) + bike lane (5 ft) gets high stress. I'm not convinced 13 ft reach should be considered low stress on a busier street with adjacent parking. A lot depends on context. If parking usage is moderate this can be stressful since you have to be more concerned with door zone conflicts and turnover. If parking usage is low this is low stress. This is something we've introduced in some of our projects for clients. Our modified tables in this case require 14 ft (8 ft parking + 6 ft bike lane--effectively a buffered bike lane) in order to be low stress.

I think this comes down to what we want to assume. The pyBNA tables are likely to render most bike lanes in most cities high stress since we assume parking if there's no OSM data making it explicit. Based on my experience in Chicago, parking-adjacent bike lanes are high stress facilities. Based on my experience in Madison, parking-adjacent bike lanes in residential areas are low stress facilities since parking isn't as heavy and you just needed the bike lane to pass the occasional parked car.

We could also consider introducing a configuration parameter that lets the user set an assumption for parking usage. e.g. you could set city-wide parking usage to low and get one result or high to get another result in these locations.

Add methods for QC. Examples:

1. Create GIS layers centered in the study area that show buffered distances taken from the config (e.g. max_distance, detour_agnostic_threshold)
2. Create GIS layers that show unusually short or long segments, high numbers of intersection legs, etc.

Shapely version 2 became the latest release back in December.
https://pypi.org/project/shapely/#history

That version does not provide direct access to geos, the way previous versions did, which breaks OSMNx v1.x because of the change needed from this commit

The current version of osmnx in pypi is 1.3
https://pypi.org/project/osmnx/

when I installed pybna with pip install git+ssh://[email protected]/tooledesign/pybna.git@dev I got shapely v 2.1 and OSMNx v 1.2.2.

The error upon running import pybna is:

/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/pybna/conf.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
  from collections import Mapping
Traceback (most recent call last):
  File "07_import_name_ref/02_import.py", line 1, in <module>
    import pybna
  File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/pybna/__init__.py", line 3, in <module>
    from .importer import Importer
  File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/pybna/importer.py", line 10, in <module>
    import osmnx as ox
  File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/osmnx/__init__.py", line 3, in <module>
    from ._api import *
  File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/osmnx/_api.py", line 17, in <module>
    from .geometries import geometries_from_address
  File "/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/osmnx/geometries.py", line 21, in <module>
    from shapely.geos import TopologicalError
ImportError: cannot import name 'TopologicalError' from 'shapely.geos' (/home/hkell/code/pybna_testing/venv/lib/python3.8/site-packages/shapely/geos.py)

I solved this with a new virtual environment and installing with the commands

pip install wheel
pip install shapely==1.8.5.post1
pip install git+ssh://[email protected]/tooledesign/pybna.git@dev

Alternatively, we could try using OSMNx v2.x but I'm assuming there's a good chance of running into other problems we dont' ahve time to debug.

We need to compare pybna results against a comparable BNA run using the old BNA method. I've downloaded results from the website for Madison, WI. The following tables are in the received schema of the bna DB on the Portland server:

• website_neighborhood_census_blocks
• website_neighborhood_ways
• website_connected_census_blocks

Our process should proceed as follows:

1. Import destinations from OSM using pybna
2. Manually calculate hs/ls destination counts for each block using website_connected_census_blocks (this represents the website condition, but using updated destination data since we don't have the destination sets used in the website BNA run)
3. Reconstruct the intersection layer using the intersection_from/intersection_to columns in website_neighborhood_ways.
4. Run pybna on website_neighborhood_ways to create a connected blocks table for pybna.
5. Calculate hs/ls destination counts for each block and compare with counts generated in Step 2.
6. Import new OSM network using pybna and run LTS
7. Compare LTS in Step 6 with LTS from website in website_neighborhood_ways

We can document any issues or discrepancies we come across here. Ultimately, we need to be able to catalog any differences as either a regression (needs fixing), a neutral change (difference that results in no substantial effect), or an enhancement (fixes a problem with the original)

Test case is in B042-denver_moves_downtown

When it tries to make tiles, calling the ._split_tiles() method isn't working. Cites that it takes exactly 10 arguments and is only getting 8.

Seems to assign defaults even if non-default settings are used in pybna.
bna_settings.zip

Network build chokes if there are values in one_way that don't represent forward or backward.

build a monolithic dictionary of sql substitutions that can be used across all queries

Switch to a select into...where false to allow for different data types on the block id column (currently requires varchar)

User shouldn't have to calculate that on each destination table

title says it all - annoying having to hand delete when something fails

When there are multiple tags in config.yaml for a destination category, the import_osm_destinations() method only seems to import destinations that match just the last tag. This does not seem to be an issue for imports done from an OSM file.

@spencerrecneps

PyBNA still seems to employ graph_tool even in the structural change branch. I can't import pybna as is.

Need to test for detour % when determining connectivity

Reading in the network from DB expects "source_vert" and other standardized column names. Need to be parameterized so we can use attributes described in config.

Need to verify that lane data on oneway multi-lane streets is not being divided in half.

Early on in the BNA we decided to include crosswalks in the network to handle cases where a trail crossing across a road was represented with the OSM tags highway=crossing and crossing=footway. I think this ultimately causes for confusion for people who view the LTS scores because they end up seeing all the little crosswalk stubs.

I propose that we require an additional bicycle=yes or bicycle=designated tag for a crosswalk to be imported from OSM. This will reduce crosswalks in the network only to those that actually serve the bicycle network.

Thoughts, @beckymasond ?

should check for

• empty geoms
• short geoms (< 1 unit)
• from/to not in ints table
• empty from/to
• high number of shared from or to (>10?)

I agree, this is something I have been meaning to improve for the BNA's importer module. Feel free to open an issue on the BNA repo for this.

Originally posted by @theja in https://github.com/tooledesign/50793_Anne_Arundel_Bike_Ped_Master_Plan/pull/2#discussion_r822811508

A street table with null geoms gums up the network build. Check for nulls and kick an error if any are found.