drprojects / point_geometric_features Goto Github PK
View Code? Open in Web Editor NEWPython wrapper around C++ utility to compute local geometric features of a point cloud
License: MIT License
Python wrapper around C++ utility to compute local geometric features of a point cloud
License: MIT License
Packaging pgeof
as a pip package, free from the conda dependency of eigen3
would be great.
@rjanvier I think you were thinking of working on this at some point. Is this still something you are considering ?
I am running into trouble installing the library using pip. It looks like an issue with the nanobind library. The last time I installed pgeof it went fine (a couple of months ago). The installation ends with:
Failed to build pgeof
ERROR: Could not build wheels for pgeof, which is required to install pyproject.toml-based projects
× Building wheel for pgeof (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [110 lines of output]
*** scikit-build-core 0.9.4 using CMake 3.27.7 (wheel)
*** Configuring CMake...
2024-05-31 11:06:57,601 - scikit_build_core - WARNING - Can't find a Python library, got libdir=/home/matic/anaconda3/envs/test-env/lib, ldlibrary=libpython3.8.a, multiarch=x86_64-linux-gnu, masd=None
loading initial cache file build/cp38-cp38-linux_x86_64/CMakeInit.txt
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python: /home/matic/anaconda3/envs/test-env/bin/python (found suitable version "3.8.19", minimum required is "3.8") found components: Interpreter Development.Module
-- Configuring done (0.4s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/build/cp38-cp38-linux_x86_64
*** Building project with Ninja...
[1/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/nb_static_property.cpp.o
[2/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/implicit.cpp.o
[3/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/trampoline.cpp.o
[4/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/nb_internals.cpp.o
[5/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/error.cpp.o
[6/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/nb_ndarray.cpp.o
[7/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/common.cpp.o
[8/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/nb_enum.cpp.o
[9/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/nb_func.cpp.o
[10/13] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/src/nb_type.cpp.o
[11/13] Linking CXX static library libnanobind-static.a
[12/13] Building CXX object CMakeFiles/pgeof_ext.dir/src/pgeof_ext.cpp.o
FAILED: CMakeFiles/pgeof_ext.dir/src/pgeof_ext.cpp.o
/usr/bin/c++ -Dpgeof_ext_EXPORTS -I/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include -I/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/third_party/eigen -I/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/third_party/nanoflann/include -I/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/third_party/taskflow -I/home/matic/anaconda3/envs/test-env/include/python3.8 -I/tmp/pip-build-env-34w23q9k/overlay/lib/python3.8/site-packages/nanobind/include -O3 -DNDEBUG -std=gnu++17 -flto -fno-fat-lto-objects -fPIC -fvisibility=hidden -fno-stack-protector -ffunction-sections -fdata-sections -MD -MT CMakeFiles/pgeof_ext.dir/src/pgeof_ext.cpp.o -MF CMakeFiles/pgeof_ext.dir/src/pgeof_ext.cpp.o.d -o CMakeFiles/pgeof_ext.dir/src/pgeof_ext.cpp.o -c /tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/src/pgeof_ext.cpp
In file included from /tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/src/pgeof_ext.cpp:8:
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:76:58: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
76 | static nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, feature_count>> compute_geometric_features(
| ^~~~~~~~~~~~~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:76:58: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:76:71: error: template argument 3 is invalid
76 | static nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, feature_count>> compute_geometric_features(
| ^~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp: In function ‘int pgeof::compute_geometric_features(pgeof::RefCloud<real_t>, nanobind::ndarray<const unsigned int, nanobind::shape<-1> >, nanobind::ndarray<const unsigned int, nanobind::shape<-1> >, size_t, bool)’:
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:114:62: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
114 | return nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, feature_count>>(features, 2, shape, owner_features);
| ^~~~~~~~~~~~~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:114:62: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:114:75: error: template argument 3 is invalid
114 | return nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, feature_count>>(features, 2, shape, owner_features);
| ^~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp: At global scope:
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:158:67: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
158 | static nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, nb::any, feature_count>> compute_geometric_features_multiscale(
| ^~~~~~~~~~~~~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:158:67: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:158:67: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:158:67: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:158:80: error: template argument 3 is invalid
158 | static nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, nb::any, feature_count>> compute_geometric_features_multiscale(
| ^~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp: In function ‘int pgeof::compute_geometric_features_multiscale(pgeof::RefCloud<real_t>, nanobind::ndarray<const unsigned int, nanobind::shape<-1> >, nanobind::ndarray<const unsigned int, nanobind::shape<-1> >, const std::vector&, bool)’:
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:206:71: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
206 | return nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, nb::any, feature_count>>(
| ^~~~~~~~~~~~~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:206:71: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:206:71: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:206:71: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:206:84: error: template argument 3 is invalid
206 | return nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, nb::any, feature_count>>(
| ^~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp: At global scope:
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:241:58: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
241 | static nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, feature_count>> compute_geometric_features_optimal(
| ^~~~~~~~~~~~~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:241:58: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:241:71: error: template argument 3 is invalid
241 | static nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, feature_count>> compute_geometric_features_optimal(
| ^~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp: In function ‘int pgeof::compute_geometric_features_optimal(pgeof::RefCloud<real_t>, nanobind::ndarray<const unsigned int, nanobind::shape<-1> >, nanobind::ndarray<const unsigned int, nanobind::shape<-1> >, uint32_t, uint32_t, uint32_t, bool)’:
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:303:62: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
303 | return nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, feature_count>>(features, 2, shape, owner_features);
| ^~~~~~~~~~~~~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:303:62: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:303:75: error: template argument 3 is invalid
303 | return nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, feature_count>>(features, 2, shape, owner_features);
| ^~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp: At global scope:
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:320:62: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
320 | static nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, nb::any>> compute_geometric_features_selected(
| ^~~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:320:62: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:320:62: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:320:62: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:320:65: error: template argument 3 is invalid
320 | static nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, nb::any>> compute_geometric_features_selected(
| ^~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp: In function ‘int pgeof::compute_geometric_features_selected(pgeof::RefCloud<real_t>, real_t, uint32_t, const std::vectorpgeof::EFeatureID&)’:
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:367:66: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
367 | return nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, nb::any>>(features, {static_cast<size_t>(n_points), feature_count}, owner_features);
| ^~~
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:367:66: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:367:66: error: type/value mismatch at argument 1 in template parameter list for ‘template<long int ...Is> struct nanobind::shape’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:367:66: note: expected a constant of type ‘long int’, got ‘nanobind::any’
/tmp/pip-install-i_isslon/pgeof_16416ab81e6a4431b1b2dc0f09770a72/include/pgeof.hpp:367:69: error: template argument 3 is invalid
367 | return nb::ndarray<nb::numpy, real_t, nb::shape<nb::any, nb::any>>(features, {static_cast<size_t>(n_points), feature_count}, owner_features);
| ^~
ninja: build stopped: subcommand failed.
*** CMake build failed
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pgeof
Failed to build pgeof
ERROR: Could not build wheels for pgeof, which is required to install pyproject.toml-based projects
I tried installing inside a fresh conda environment using python3.8 and 3.9, both unsuccessful. Installation using install.sh inside the SPT repo also failed with the same error. Googling I found it could also be an issue with non-upgraded pip, wheel, but trying that didn't fix my issue.
I am using Ubuntu 20.04.4 LTS, CUDA Version 12.4
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
ld_impl_linux-64 2.40 hf3520f5_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h77fa898_7 conda-forge
libgomp 13.2.0 h77fa898_7 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libsqlite 3.45.3 h2797004_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.2.13 h4ab18f5_6 conda-forge
ncurses 6.5 h59595ed_0 conda-forge
openssl 3.3.0 h4ab18f5_3 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
python 3.9.19 h0755675_0_cpython conda-forge
readline 8.2 h8228510_1 conda-forge
setuptools 70.0.0 pyhd8ed1ab_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tzdata 2024a h0c530f3_0 conda-forge
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
Any help is appreciated!
Hi Damien,
For a bit of context, I currently use @drprojects/superpoint_transformer to make some forest segmentation for the @3DFin project. It works pretty well but we want to lower the technical cost of using it (in an inference context) for scientist interested in the field of Forest management.
I started to review compiled dependency and evaluate if we can offer pre build wheels (via https://github.com/pypa/cibuildwheel). I first noticed that all your C++ based python module/dependencies use distutils
as a build system. distutils
is deprecated and I think it did not even work anymore with python 3.12
. So future users could have hard time to install your dependencies. Would you agree if I try to port pgeof
to setuptools
build system (which is the "drop in" replacement for distutils
)?
Hello,
as mentionned in this issue I think pgeof has a memory leaks too:
If we take the demo.py script with a slightly faster knn :
from pgeof import pgeof
import numpy as np
import tracemalloc
tracemalloc.start()
num_iter=200
marqueur=0
snapshot=tracemalloc.take_snapshot()
import gc
from scipy.spatial import cKDTree
def get_x(n_points=4000000):
import torch
x_min=[-15,46]
y_min=[-45,30]
z_min=[0,7]
num_class=9
x=torch.rand((n_points,3))
x[:,0]=x[:,0]*(x_min[1]-x_min[0])+x_min[0]
x[:,1]=x[:,1]*(y_min[1]-y_min[0])+y_min[0]
x[:,2]=x[:,2]*(z_min[1]-z_min[0])+z_min[0]
return x
def Query_CPU(
xyz_query, xyz_search, K,r):
kdtree=cKDTree(xyz_search)
distances, neighbors = kdtree.query(xyz_query, k=K, distance_upper_bound=r,workers=-1)
neighbors[distances==float('inf')]=-1
return distances, neighbors
for j in range(num_iter):
# Generate a random synthetic point cloud
num_points = int(1e5)
xyz=get_x(num_points).numpy()
# Manually generating random neighbors in CSR format
nn_ptr = np.r_[0, np.random.randint(low=0, high=30, size=num_points).cumsum()]
nn = np.random.randint(low=0, high=num_points, size=nn_ptr[-1])
# Converting k-nearest neighbors to CSR format
from sklearn.neighbors import NearestNeighbors
k = 20
kneigh = Query_CPU(xyz, xyz, k,20)
nn_ptr = np.arange(num_points + 1) * k
nn = kneigh[1].flatten()
# Converting radius neighbors to CSR format
# from sklearn.neighbors import NearestNeighbors
# radius = 0.1
# rneigh = NearestNeighbors(radius=radius).fit(xyz).radius_neighbors(xyz)
# nn_ptr = np.r_[0, np.array([x.shape[0] for x in rneigh[1]]).cumsum()]
# nn = np.concatenate(rneigh[1])
# Make sure xyz are float32 and nn and nn_ptr are uint32
xyz = xyz.astype('float32')
nn_ptr = nn_ptr.astype('uint32')
nn = nn.astype('uint32')
# Make sure arrays are contiguous (C-order) and not Fortran-order
xyz = np.ascontiguousarray(xyz)
nn_ptr = np.ascontiguousarray(nn_ptr)
nn = np.ascontiguousarray(nn)
geof = pgeof(
xyz, nn, nn_ptr, k_min=10, k_step=1, k_min_search=15,
verbose=True)
marqueur+=1
gc.collect()
if marqueur>1e1:
snapshot2=tracemalloc.take_snapshot()
marqueur=0
Top_stats=snapshot2.compare_to(snapshot,'lineno')
print("TOP 10 differneces")
for stat in Top_stats[:10]:
print(stat)
current,peak=tracemalloc.get_traced_memory()
print(f"Current memory usage is {current/10**6}MB; Peak was {peak/10**6}MB")
top_stats=snapshot2.statistics('traceback')
stat=top_stats[0]
print("%s memory blocks: %.1f in MiB: " % (stat.count,stat.size/1024**2))
for line in stat.traceback.format():
print(line)
the memory is slightly increasing each iteration and I tried to run the script with valgrind and I indeed go some memory definitely lsot
Hi Damien,
I have a little window to try to implement multi scale feature computation we talked about (see #4).
I would like to know how do you see the thing? would you want to rely on an aggregation scheme (I don't think a mean makes sense for all features) or output the whole multi scale feature map?
Thank in advance,
Romain
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.