ishiland / python-geosupport Goto Github PK

Hello,

I just came across this repo and it looks like a great project. I haven't had a chance to test drive python-geosupport yet but I am looking forward to it.

I am the author/maintainer of geoclient which is a Java-based library for calling Geosupport in-process or through a remote REST API. I am posting this because I have been thinking about adding different language bindings to geoclient for a while -- python, ruby, go, etc. I'm glad I came across your work before attempting my own, undoubtedly less idiomatic python implementation.

If you are ever interested in collaborating, sharing ideas or similar, please let me know.

Thanks,
@mlipper

Hi,

I have python 3.8.10 64-BIT and GeoSupport Desktop 64-BIT (gde_23d_x64) installed. I have been trying to quickstart code but I keep getting the following error:

geosupport.error.GeosupportError: Could not find module 'NYCGEO.dll' (or one of its dependencies). Try using the full path with constructor syntax.
You are currently using a 64-bit Python interpreter. Is the installed version of Geosupport 64-bit?

I have tried specifying the path to my GeoSupport as seen in the below code but results in the same error. I also tried specifying the path to NYCGEO.dll directly and the same thing occured. Any idea on what the issue may be?

# Import the library and create a `Geosupport` object.
from geosupport import Geosupport
g = Geosupport(geosupport_path="C:\\Program Files\\Geosupport Desktop Edition")

# Call the address processing function by name
result = g.address(house_number=125, street_name='Worth St', borough_code='Mn')

While running appveyor CI tests, this error occurs on all builds with python 2.7:

ERROR: test_print (tests.unit.test_help.TestHelp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\projects\python-geosupport\tests\unit\test_help.py", line 32, in test_print
    o = self.geosupport.help()
  File "c:\projects\python-geosupport\geosupport\geosupport.py", line 146, in help
    print(return_val)
TypeError: unicode argument expected, got 'str'
----------------------------------------------------------------------

Thanks for building this library - it will be extremely valuable to me once I can get it working!

When I try to run the first example in the docs, I get the following error. I am running Ubuntu 16.04.

root@0b3992e9466b:/# python
Python 3.6.6 |Anaconda custom (64-bit)| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from geosupport import Geosupport
>>> g = Geosupport()
>>> result = g.address(house_number=125, street_name='Worth St', borough_code='Mn')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda/lib/python3.6/site-packages/geosupport/geosupport.py", line 115, in call
    result
geosupport.error.GeosupportError: ERROR ACCESSING GEOSUPPORT FILE: GRID1. NOTIFY SYSTEM SUPPORT ERROR ACCESSING GEOSUPPORT FILE: PAD. NOTIFY SYSTEM SUPPORT

The relevant part of the Dockerfile where I install Geosupport is as follows:

# Lots of stuff before this point ...
RUN apt-get update && \
    apt-get install -y unzip && \
    wget https://www1.nyc.gov/assets/planning/download/zip/data-maps/open-data/gdelx_18b.zip && \
    unzip gdelx_18b.zip -d /var/geosupport/ && \
    rm gdelx_18b.zip
ENV GEOFILES=/var/geosupport/version-18b_18.2/fls
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/geosupport/version-18b_18.2/lib/
RUN pip install python-geosupport

I took a look at your Dockerfile and so I ran:

git clone https://github.com/ishiland/python-geosupport.git
cd ./python-geosupport
pip install -e .[dev]
invoke test all

The output of the tests was a segmentation fault:

root@0b3992e9466b:/# git clone https://github.com/ishiland/python-geosupport.git
Cloning into 'python-geosupport'...
remote: Enumerating objects: 602, done.
remote: Total 602 (delta 0), reused 0 (delta 0), pack-reused 602
Receiving objects: 100% (602/602), 510.80 KiB | 0 bytes/s, done.
Resolving deltas: 100% (311/311), done.
Checking connectivity... done.
root@0b3992e9466b:/# cd ./python-geosupport
root@0b3992e9466b:/python-geosupport# pip install -e .[dev]
Obtaining file:///python-geosupport
Requirement already satisfied: coverage in /usr/local/anaconda/lib/python3.6/site-packages (from python-geosupport==1.0.3) (4.5.2)
Collecting invoke>=1.1.1 (from python-geosupport==1.0.3)
  Downloading https://files.pythonhosted.org/packages/be/9f/8508712c9cad73ac0c8eeb2c3e51c9ef65136653dda2b512bde64109f023/invoke-1.2.0-py3-none-any.whl (206kB)
    100% |████████████████████████████████| 215kB 8.8MB/s
Requirement already satisfied: nose in /usr/local/anaconda/lib/python3.6/site-packages (from python-geosupport==1.0.3) (1.3.7)
Installing collected packages: invoke, python-geosupport
  Found existing installation: python-geosupport 1.0.3
    Uninstalling python-geosupport-1.0.3:
      Successfully uninstalled python-geosupport-1.0.3
  Running setup.py develop for python-geosupport
Successfully installed invoke-1.2.0 python-geosupport
root@0b3992e9466b:/python-geosupport# invoke test all
EEEEEE.EEEFEFEEEEEEEE......EEEE.Segmentation fault

Pytest gives the following output:

root@0b3992e9466b:/python-geosupport# pytest
======================================================== test session starts ========================================================
platform linux -- Python 3.6.6, pytest-3.8.0, py-1.6.0, pluggy-0.7.1
rootdir: /python-geosupport, inifile:
plugins: remotedata-0.3.0, openfiles-0.3.0, doctestplus-0.1.3, cov-2.5.1, arraydiff-0.2, celery-4.2.1
collected 51 items

tests/functional/test_call.py FFFFFF.FFFFFFFFFFFFFF......FFFF.                                                                [ 62%]
tests/functional/test_call_alternate.py Segmentation fault
root@0b3992e9466b:/python-geosupport#

Any sense what might be the cause of the problem?

Hi there,

Thank you for developing this package, it's been super helpful for my geocoding process.
I have 1 question tho. After I finished geocoding using function 1B (the g.address function, which only contains PAD), I tried to use the same function with mode='tpad' and mode='regular+tpad', which for some reason resulted more entries that couldn't find a lon and lat.

Also mode 'tpad' and 'regular+tpad' contains the exact same numbers of entries that couldn't find lon and lat. I know in your README, 'regular+tpad' is not even an option, but somehow it works?

It would be super helpful if you can explain this curious case, thanks!

Per the Geosupport System User Bulletin 19-03, a new field "Truck Route Type" has been added.
This field should be added in WA2 outputs as defined here:

Full details attached here
Geosupport System User Bulletin 19-03.pdf

Move documentation to Read the Docs in an effort to better organize and structure the documentation. The readme for this repo can contain a "quick start" section while Read the Docs can provide a more in-depth guide on the installation and functionality of python-geosupport.

Hi, thank you for this library, I've set this up in Windows but trying to get the Linux version of Geosupport running w/ this library on CentOS8.

I ran the following:

from geosupport import Geosupport
g = Geosupport()

# Call the address processing function by name
result = g.address(house_number=125, street_name='Worth St', borough_code='Mn')

I get the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/geosupport/geosupport.py", line 63, in __init__
    self.geolib = cdll.LoadLibrary("libgeo.so")
  File "/usr/lib64/python3.6/ctypes/__init__.py", line 421, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib64/python3.6/ctypes/__init__.py", line 343, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libgeo.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/geosupport/geosupport.py", line 75, in __init__
    e, self.py_bit, self.py_bit
geosupport.error.GeosupportError: libgeo.so: cannot open shared object file: No such file or directory
You are currently using a 64-bit Python interpreter. Is the installed version of Geosupport 64-bit?

Should I use a 32-bit Python interpreter? If so, is there a set of directions you'd recommend?

Should we add FIPS in the BOROUGH look up so people who work with census data can have an easier time?

36005 - Bronx
36047 - Kings (Brooklyn)
36061 - New York (Manhattan)
36081 - Queens
36085 - Richmond (Staten Island)

Also proposing to add the strange DOE public school Borough prefixes:
K = Brooklyn (Kings County)
M = Manhattan
Q = Queens
R = Staten Island (Richmond County)
X = Bronx

Line breaks are not working in the interactive help function.

Issue is reproducible using Win7/Python 3.4 in both Powershell and the Command prompt.

Hi there,
I noticed some problems with census tract handling issues, for example:

>>> g['1B'](street_name='Broadway', house_number='120', borough_code='MN')['2010 Census Tract']
'7'

Which creates some headaches in terms of padding them with 0s (BoroCT, e.g. the amove record would be recorded as '1000700', a 7 digit number with first digit as boro code).
When dealing with census tracts that has a decimal, e.g. CT 5.01 and 501, the returns would be the same here, which creates confusion.
After consulting with some of the mainframe programmers, I was told that in the case above. the mainframe output would be:

'   7  ' # '000700' 

So that we just have to do a ' 7 '.replace(' ', '0') to get '000700
accordingly.

'   501' #000501 --> 5.01  
' 501  ' #050100 --> 501

Is there anyway you can implement this when translating output to python?

As came up in #11, if possible we want to make sure that changes to the underlying data doesn't break our tests. But we also need to make sure that the tests could possibly catch changes to the data format.

The issue is: we want to make sure that the data coming out is being parsed correctly, which means we need to care about specific values. For instance, if the work area were expanded and the location of a field were changed, we would want the tests to fail, which necessarily means testing the values coming out and making sure they're being mapped to the correct place.

I wonder if the geosupport developers have any dummy data in there that they use to test it and ensure forward compatibility. If there's data we know won't change, then we could use that in the test cases.

Hello,

I have both Python 3.12 (64 bit) and Geosupport version 23d (64 bit) for Window on my computer. I have the FLs directory at C:\Program Files\Geosupport Desktop Edition\fls but I am getting the following error.

Original Code:

directory = r'C:\Program Files\Geosupport Desktop Edition\Fls'
from geosupport import Geosupport
g = Geosupport()
result = g.address(house_number=125, street_name='Worth St', borough_code='Mn')

Error

Traceback (most recent call last):
  File "C:\Users\kagnas\PycharmProjects\Geocoding\Geocoding Lesson 1\check.py", line 5, in <module>
    result = g.address(house_number=125, street_name='Worth St', borough_code='Mn')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\kagnas\PycharmProjects\Geocoding\Geocoding Lesson 1\venv\Lib\site-packages\geosupport\geosupport.py", line 117, in call
    raise GeosupportError(
geosupport.error.GeosupportError: ERROR ACCESSING GEOSUPPORT FILE: GRID1. NOTIFY SYSTEM SUPPORT ERROR ACCESSING GEOSUPPORT FILE: PAD. NOTIFY SYSTEM SUPPORT

I checked if the directory exists as well as GRID1 and PAD. Then, checked if it will initialize to the Geosupport library but I see that those two files are present in the directory, but the error persists attempting to initialize Geosupport. I was wondering if you please assist with this, thank you!

Setup CI using Python 2.7 & 3.6 on Windows x86, Windows x64 and Linux. Appveyor seems like a good approach.

From the Geosupport System User Bulletin 19-03:

Function 3 with Mode E returns more B7SCs (Borough and 7-digit street codes) instead of
B5SCs, e.g. in the list of cross streets at the low address end and the high address end of the
segment. This gives the user easier access to a finer definition of the streets.
The mode E functions are referred to as Function 3 Enhanced (or 3E) and Function 3C
Enhanced (or 3CE).

This should be easier to address once the Geosupport UPG is updated to 19.3 with a WA layout for these new functions.

First off, thanks for creating this library.

There is an issue with file encoding that occurs when running on linux (or at least on debian).

The package installs fine - both using git and pypi -- but running import geosupport raises this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/geosupport/venv/lib/python3.6/site-packages/geosupport/__init__.py", line 1, in <module>
    from .geosupport import Geosupport
  File "/tmp/geosupport/venv/lib/python3.6/site-packages/geosupport/geosupport.py", line 5, in <module>
    from .function_info import FUNCTIONS, function_help, list_functions, input_help
  File "/tmp/geosupport/venv/lib/python3.6/site-packages/geosupport/function_info.py", line 205, in <module>
    FUNCTIONS = load_function_info()
  File "/tmp/geosupport/venv/lib/python3.6/site-packages/geosupport/function_info.py", line 35, in load_function_info
    for row in csv:
  File "/usr/lib/python3.6/csv.py", line 111, in __next__
    self.fieldnames
  File "/usr/lib/python3.6/csv.py", line 98, in fieldnames
    self._fieldnames = next(self.reader)
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 496: invalid start byte

The CSV files located in function_info/ are encoded with ISO-8859-1, but python on Linux appaers to assume by default that they are UTF-8 and raises an error.

I did fix the issue by converting the files to UTF-8 via this shell command:

 find geosupport/function_info/ -type f -name '*.csv' | xargs -I FILE sh -c "iconv -t UTF-8 -f ISO-8859-1 FILE  > tmp && mv tmp FILE"

however, perhaps a solution should be incorporated into the program?

Is there an argument for the g.address() method that allows the input to include both house_number and street_name to be part of one variable? In the desktop version, it is referred to as freeform.

Thanks for the assistance.

I have both Python 3.11 (64 bit) and Geosupport version 19b (64 bit) for windows on my computer. My path the environmental variables appear to be correct, I have the fls directory at C:\Program Files\Geosupport Desktop Edition\fls and the bin directory at C:\Program Files\Geosupport Desktop Edition\bin.

However, when running the code to initialize Geosupport, I get following error message:
geosupport.error.GeosupportError: Could not find module 'NYCGEO.dll' (or one of its dependencies). Try using the full path with constructor syntax.
You are currently using a 64-bit Python interpreter. Is the installed version of Geosupport 64-bit?

Is there a way to see what the similiar names are for the below code? My goal is to see the similiar names and use fuzzy matching to potentially find the correct street name.

from geosupport import GeosupportError

try:
    g.get_street_code(borough='MN', street='Wort Street')
except GeosupportError as e:
    print(e) # 'WORT STREET' NOT RECOGNIZED. THERE ARE 010 SIMILAR NAMES.

just an observation here I noticed that when geocoding using zipcode, the output zipcode is not always the same as the input zipcode, any idea why this is happening?

>>>geo=g['1b'](house_number='33-33', street_name='broadway', zip_code='10031')
>>>geo.get('ZIP Code', '')
'10006'

In this case, the house number 33-33 has display format 33, and 33 Broadway is indeed in lower Manhattan with a zipcode 10006.

So I started thinking if it's because geosupport translates zipcode to borough, then geocode. so I tried to use a brooklyn zipcode 11238 to see if it's still returning 10006

>>> geo=g['1b'](house_number='33-33', street_name='broadway', zip_code='11238')
>>> geo.get('ZIP Code', '')
'11249'
>>>

Indeed this time it returned a different Brooklyn zipcode.
I guess it's just something people should keep in mind

Tests are failing for Geosupport 18D. This appears to be due to Geosupport data changes not matching the expected test value.

It seems one approach would be to write data-agnostic unit tests, but not sure that is the right direction and/or possible.

From Geosupport System User Bulletin 19-02 / Release 19B:

hi there,

I would like to use browse flag for picking primary addresses, but I don't know the syntax for it, can you provide more info how to use it?>

Thanks