avanetten / cresi Goto Github PK
View Code? Open in Web Editor NEWRoad network extraction from satellite imagery, with speed and travel time estimates
License: Apache License 2.0
Road network extraction from satellite imagery, with speed and travel time estimates
License: Apache License 2.0
There are 2 errors while I tried to build the image. I walk-around Error1 by comment that line and then I encounter Erro2...
Error1:
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
Log:
Step 11/30 : ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
ADD failed: Get https://github-production-release-asset-2e65be.s3.amazonaws.com/31432573/f039db52-89a3-11e7-8e08-44ac01cb80d5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200522%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200522T015128Z&X-Amz-Expires=300&X-Amz-Signature=f9a9f4a90b0c526c0502c9c7aae5497b0b316155b1d651b47e5c8febe8330d69&X-Amz-SignedHeaders=host&actor_id=0&repo_id=31432573&response-content-disposition=attachment%3B%20filename%3Dtini&response-content-type=application%2Foctet-stream: x509: certificate signed by unknown authority
Error2:
RUN conda install -n cresi pytorch torchvision cuda90 -c pytorch && conda clean -p && conda clean -t && conda clean --yes --all
Log
Downloading and Extracting Packages
pytorch-1.4.0 | 432.9 MB | | 0%
intel-openmp-2020.1 | 780 KB | ########## | 100%
ninja-1.10.0 | 1.9 MB | ########## | 100%
cuda90-1.0 | 3 KB | | 0%
mkl-2020.1 | 129.0 MB | ########## | 100%
torchvision-0.5.0 | 9.1 MB | | 0%
cudatoolkit-10.1.243 | 347.4 MB | ########## | 100%
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pytorch/linux-64/pytorch-1.4.0-py3.6_cuda10.1.243_cudnn7.6.3_0.tar.bz2>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pytorch/linux-64/cuda90-1.0-h6433d27_0.tar.bz2>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pytorch/linux-64/torchvision-0.5.0-py36_cu101.tar.bz2>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
The command '/bin/bash -c conda install -n cresi pytorch torchvision cuda90 -c pytorch && conda clean -p && conda clean -t && conda clean --yes --all' returned a non-zero code: 1
Hi,
first of all, thank you for this great project.
I tried to use the 04_skeletonize.py with images that I have created. This is one example image:
The make_skeleton()
method runs perfectly and produces an excellent ske image as result.
However, the graph can't be generated, the output is:
len G.nodes(): 1
len G.edges(): 0
What am I doing wrong?
Thanks in advance for help.
Best,
Jan
PS: My configuration (main method). Values are taken from the sn5_baseline.json:
log_file = '../results/skeleton.log'
console, logger1 = make_logger.make_logger(log_file, logger_name='log')
im_dir = "../data/"
outfile_csv = "../results/out.csv"
out_ske_dir = "../results/ske/"
out_gdir = "../results/gdir/"
debug = False
add_small = True
fix_borders = True
img_shape = () # (1300, 1300)
skel_replicate = 5
skel_clip = 2
img_mult = 255
hole_size = 300
cv2_kernel_close = 7
cv2_kernel_open = 7
# max_out_size=(16000, 8000) # works fine
# max_out_size=(8003, 16009) # works fine
max_out_size = (2000000, 2000000)
spacenet_naming_convention = False
# from config
min_subgraph_length_pix = 20
skeleton_band = 0
num_classes= 1
im_prefix = ''
thresh = 0.3
t0 = time.time()
df = build_wkt_dir(im_dir, outfile_csv, out_ske_dir, out_gdir, thresh, #threshes={'2': .3, '3': .3, '4': .3, '5': .2},
debug=debug, add_small=add_small, fix_borders=fix_borders,
img_shape=img_shape,
skel_replicate=skel_replicate, skel_clip=skel_clip,
img_mult=img_mult, hole_size=hole_size,
min_subgraph_length_pix=min_subgraph_length_pix,
cv2_kernel_close=cv2_kernel_close, cv2_kernel_open=cv2_kernel_open,
max_out_size=max_out_size,
skeleton_band=skeleton_band,
num_classes=num_classes,
im_prefix=im_prefix,
spacenet_naming_convention=spacenet_naming_convention)
print ("len df:", len(df))
print ("outfile:", outfile_csv)
t1 = time.time()
logger1.info("Total time to run build_wkt_dir: {} seconds".format(t1-t0))
PPS: The complete output:
1 / 1 : prediction8.png
img_loc: ../data/prediction8.png
im_root: prediction8
out_ske_file: ../results/ske/prediction8.png
Executing make_skeleton...
log : INFO 1 / 1 : prediction8.png
make_skeleton(), input img_shape: ()
make_skeleton(), img.shape: (256, 256)
make_skeleton(), img.size: 65536
make_skeleton(), img dtype: uint8
run preprocess()...
Run preprocess() with skimage
Time to run preprocess(): 0.002991199493408203 seconds
skeletonize...
Time to run skimage.skeletonize(): 0.0019943714141845703 seconds
fix_borders...
Time fix borders: 0.0 seconds
ske.shape: (256, 256)
Time to run make_skeleton: 21.24360680580139 seconds
Execute sknw...
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:20: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "mark" failed type inference due to: Untyped global name 'neighbors': cannot determine Numba type of <class 'function'>
File "utils\sknw.py", line 22:
def mark(img): # mark the array use (0, 1, 2)
nbs = neighbors(img.shape)
^
@jit # my mark
numba.transforms: INFO finding looplift candidates
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:20: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "mark" failed type inference due to: Untyped global name 'neighbors': cannot determine Numba type of <class 'function'>
File "utils\sknw.py", line 22:
def mark(img): # mark the array use (0, 1, 2)
nbs = neighbors(img.shape)
^
@jit # my mark
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:178: NumbaWarning: Function "mark" was compiled in object mode without forceobj=True, but has lifted loops.
File "utils\sknw.py", line 22:
def mark(img): # mark the array use (0, 1, 2)
nbs = neighbors(img.shape)
^
state.func_ir.loc))
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:188: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.
For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit
File "utils\sknw.py", line 22:
def mark(img): # mark the array use (0, 1, 2)
nbs = neighbors(img.shape)
^
state.func_ir.loc))
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:92: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "parse_struc" failed type inference due to: Untyped global name 'neighbors': cannot determine Numba type of <class 'function'>
File "utils\sknw.py", line 94:
def parse_struc(img):
nbs = neighbors(img.shape)
^
@jit # parse the image then get the nodes and edges
numba.transforms: INFO finding looplift candidates
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:92: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "parse_struc" failed type inference due to: Untyped global name 'neighbors': cannot determine Numba type of <class 'function'>
File "utils\sknw.py", line 94:
def parse_struc(img):
nbs = neighbors(img.shape)
^
@jit # parse the image then get the nodes and edges
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:178: NumbaWarning: Function "parse_struc" was compiled in object mode without forceobj=True, but has lifted loops.
File "utils\sknw.py", line 94:
def parse_struc(img):
nbs = neighbors(img.shape)
^
state.func_ir.loc))
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:188: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.
For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit
File "utils\sknw.py", line 94:
def parse_struc(img):
nbs = neighbors(img.shape)
^
state.func_ir.loc))
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:92: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "parse_struc" failed type inference due to: non-precise type pyobject
[1] During: typing of argument at C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py (101)
File "utils\sknw.py", line 101:
def parse_struc(img):
<source elided>
nodes = []
for p in pts:
^
@jit # parse the image then get the nodes and edges
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:178: NumbaWarning: Function "parse_struc" was compiled in object mode without forceobj=True.
File "utils\sknw.py", line 101:
def parse_struc(img):
<source elided>
nodes = []
for p in pts:
^
state.func_ir.loc))
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:188: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.
For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit
File "utils\sknw.py", line 101:
def parse_struc(img):
<source elided>
nodes = []
for p in pts:
^
state.func_ir.loc))
C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py:92: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "parse_struc" failed type inference due to: non-precise type pyobject
[1] During: typing of argument at C:\Users\Jan.noessner\Documents\01_road_detection\resources\centerline\cresi\cresi\utils\sknw.py (108)
File "utils\sknw.py", line 108:
def parse_struc(img):
<source elided>
edges = []
for p in pts:
^
@jit # parse the image then get the nodes and edges
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:178: NumbaWarning: Function "parse_struc" was compiled in object mode without forceobj=True.
File "utils\sknw.py", line 108:
def parse_struc(img):
<source elided>
edges = []
for p in pts:
^
state.func_ir.loc))
C:\Apps\Anaconda3\envs\ml\lib\site-packages\numba\object_mode_passes.py:188: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.
For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit
File "utils\sknw.py", line 108:
def parse_struc(img):
<source elided>
edges = []
for p in pts:
^
state.func_ir.loc))
log : INFO Time to build graph: 1018.8935737609863 seconds
log : INFO Total time to run build_wkt_dir: 1018.8965332508087 seconds
len G.nodes(): 1
len G.edges(): 0
len df: 1
outfile: ../results/out.csv
Process finished with exit code 0
Dear avanetten,
I am trying to reproduce the continuous speed mask extraction algorithm from your codes. There are get_contours_complex() function and get_mask() function which are unreferenced. Could you share with me the link to the road_raster.py?
Kind Regards,
Shicheng Zu
If I use my own .tif image, it's fine during step 2~4 but failed to generate graph in step 5.
Guess there exists some dependencies about SpaceNet dataset.
Is there any simple way to walk around that?
python 05_wkt_to_G.py /opt/cresi/cresi/configs/sn5_baseline.json
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
ERROR 1: No PROJ.4 translation for source SRS, coordinate transformation initialization has failed.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/conda/envs/cresi/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/opt/conda/envs/cresi/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "05_wkt_to_G.py", line 354, in pixelToGeoCoord
geom.Transform(coord_trans)
File "/opt/conda/envs/cresi/lib/python3.6/site-packages/osgeo/ogr.py", line 7371, in Transform
return _ogr.Geometry_Transform(self, *args)
TypeError: in method 'Geometry_Transform', argument 2 of type 'OSRCoordinateTransformationShadow *'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "05_wkt_to_G.py", line 1283, in <module>
main()
File "05_wkt_to_G.py", line 1274, in main
wkt_to_G(params[0])
File "05_wkt_to_G.py", line 887, in wkt_to_G
verbose=verbose)
File "05_wkt_to_G.py", line 439, in get_node_geo_coords
coords_dict_list = pool.map(pixelToGeoCoord, params)
File "/opt/conda/envs/cresi/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/opt/conda/envs/cresi/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
TypeError: in method 'Geometry_Transform', argument 2 of type 'OSRCoordinateTransformationShadow *'
are there any instructions to run on the windows machine without docker?
I tried to download the datasets according to your notebook but it failed.
$ aws s3 cp --recursive s3://spacenet-dataset/spacenet/SN5_roads/tarballs/ .
download: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_test_public_AOI_8_Mumbai.tar.gz to ./SN5_roads_test_public_AOI_8_Mumbai.tar.gz
download failed: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_test_public_AOI_9_San_Juan.tar.gz to ./SN5_roads_test_public_AOI_9_San_Juan.tar.gz [Errno 22] Invalid argument
download failed: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_test_public_AOI_7_Moscow.tar.gz to ./SN5_roads_test_public_AOI_7_Moscow.tar.gz [Errno 22] Invalid argument
download failed: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_train_AOI_7_Moscow.tar.gz to ./SN5_roads_train_AOI_7_Moscow.tar.gz [Errno 22] Invalid argument
download failed: s3://spacenet-dataset/spacenet/SN5_roads/tarballs/SN5_roads_train_AOI_8_Mumbai.tar.gz to ./SN5_roads_train_AOI_8_Mumbai.tar.gz [Errno 22] Invalid argument
I run ./train.sh configs/sn5_baseline.json But I got a error at print (" Missing file:", os.path.join(self.save_path, checkpoint_name))
How to i can create checkpoint file or how to i download checkpoint file?
I have been trying to build the cresi image according to the instructions on GitHub, and rooting the docker located in "gpu folder", but I am encountering several errors stopping the process. I wish it was possible to pull the image automatically which prevents such issues. Anyway, has anyone have any ideas? Any help would be appreciated.
cresi/cresi/configs/sn5_baseline.json
Line 7 in b996f2d
should be:
"speed_conversion_file": "/path/to/data/cresi_data/cresi_train/SN5_roads_train_speed_conversion_binned.csv",
I'm interested in the routing performance of large road networks in the SpaceNet dataset,
so getting larger imagery or the way to match small patches up to single large imagery is necessary.
In the blog Large Road Networks section 4, that is the same thing that I want to do.
However, I'm unfamiliar with the details.
I notice that patches in SpaceNet road datasets (AOI_2_Vegas, AOI_3_Paris, AOI_4_Shanghai, AOI_5_Khartoum) are 1300x1300,
and they should be cropped from larger imagery like the picture below.
Could you help me with how to figure out the relation between cropped ones and the larger one?
BTW, is it possible to connect roads in different patches, and how?
Thank you very much!
When running %run -i 05_wkt_to_G.py configs/dar_tutorial_cpu.json
in dar_tutorial_cpu/cresi_cpu_part2.ipynb, I get the following error (the key 'geometry' is not present within G_projected.edges"
KeyError Traceback (most recent call last)
~/Desktop/cresi/cresi/05_wkt_to_G.py in <module>
1281 ###############################################################################
1282 if __name__ == "__main__":
-> 1283 main()
~/Desktop/cresi/cresi/05_wkt_to_G.py in main()
1272 pool.map(wkt_to_G, params)
1273 else:
-> 1274 wkt_to_G(params[0])
1275
1276 tf = time.time()
~/Desktop/cresi/cresi/05_wkt_to_G.py in wkt_to_G(params)
918 for i,(u,v,attr_dict) in enumerate(G_projected.edges(data=True)):
919 print (attr_dict)
--> 920 attr_dict['geometry_wkt'] = attr_dict['geometry'].wkt
921
922 if verbose:
KeyError: 'geometry'
I've also printed the keys/values in G_projected.edges here for example until it threw me that error:
{'start': 10000, 'start_loc_pix': (7350.0, 2.0), 'end': 10001, 'end_loc_pix': (7322.0, 53.0), 'length_pix': 58.180752831155424, 'wkt_pix': 'LINESTRING (7350 2, 7322 53)', 'geometry_pix': 'LINESTRING (7350 2, 7322 53)', 'osmid': 0, 'geometry_latlon_wkt': 'LINESTRING (39.27236399997931 -6.7580028000011, 39.27228839997933 -6.758140500001112)', 'geometry_utm_wkt': 'LINESTRING (530098.2259186266 9252994.531477312, 530089.8629878408 9252979.31387781)', 'length_latlon': 0.0001570880326449568, 'length_utm': 17.36415693108535, 'length': 17.36415693108535, 'utm_zone': 37, 'utm_letter': 'M'}
cresi/cresi/utils/get_paths.py
Line 9 in b996f2d
cresi/cresi/configs/sn5_baseline.json
Line 4 in b996f2d
TypeError: __new__() got an unexpected keyword argument 'path_data_root'
found using python 04_skeletonize.py configs/sn5_baseline.json
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.