Comments (7)
Hmm, definitely still some issues -- things got brought over from umap, but apparently there are some issues. I don't have time to dig into this right now, but hopefully in a week or two I will have time.
from pynndescent.
Thanks! Let me know if I've missed out any useful information you might need.
from pynndescent.
I finally got some time and dug into this. The short answer was that not everything got fully ported over from umap, and there were some issues with sparse support in threaded mode (is as much as that wasn't yet implemented). I think I've got that all fixed now. If you can try cloning the current master and let me know if that resolves the issue for you I would appreciate it.
from pynndescent.
Thanks so much for looking into this! Both of the problems I mentioned seem to be fixed; I can use it on sparse just like I would expect.
Unfortunately, building and querying the index now raises several numba warnings, even when I run it on dense data e.g.
/home/pavlin/dev/pynndescent/pynndescent/sparse_nndescent.py:121: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
File "pynndescent/utils.py", line 459:
@numba.njit(parallel=True)
def new_build_candidates(
^
False,
/home/pavlin/dev/pynndescent/pynndescent/sparse_nndescent.py:82: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
File "pynndescent/utils.py", line 79:
@numba.njit(parallel=True)
def rejection_sample(n_samples, pool_size, rng_state):
^
indices = rejection_sample(n_neighbors, n_vertices, rng_state)
when building the index and
/home/pavlin/dev/pynndescent/pynndescent/sparse_nndescent.py:273: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
File "pynndescent/sparse_nndescent.py", line 175:
@numba.njit(parallel=True)
def sparse_init_from_random(
^
dist_args,
/home/pavlin/dev/pynndescent/pynndescent/sparse_nndescent.py:288: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
File "pynndescent/sparse_nndescent.py", line 207:
@numba.njit(parallel=True)
def sparse_init_from_tree(
^
dist_args,
/home/pavlin/miniconda3/envs/tmp/lib/python3.7/site-packages/numba-0.44.0-py3.7-linux-x86_64.egg/numba/ir_utils.py:1958: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'forest' of function 'sparse_initialise_search'.
For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types
File "pynndescent/sparse_nndescent.py", line 248:
@numba.njit()
def sparse_initialise_search(
^
warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
when querying the index.
from pynndescent.
It looks like the warnings have started appearing from Numba 0.44. I've opened #68 to remove the ones related to parallel=True
. There are two more to do with 'reflective list' being deprecated, but those can't be easily fixed until the replacement appears in Numba 0.45. In the meantime the warnings can be ignored, or Numba 0.43 (or earlier) can be specified explicitly when installing.
from pynndescent.
Thanks for #68 @tomwhite, that shoudl take care of a lot of them. This is popping up in umap as well, so I will have to take care of it there soon.
from pynndescent.
There are two more to do with 'reflective list' being deprecated, but those can't be easily fixed until the replacement appears in Numba 0.45.
@tomwhite I haven't seen those warnings myself, but I think Numba 0.45 is out already. Do you know what's the status of those warnings?
from pynndescent.
Related Issues (20)
- uint8 as internal data HOT 1
- Cosine metric - error "Negative values in data passed to precomputed distance matrix" HOT 2
- Question about covariance matrix used when using Mahalanobis distance
- Tests fail: E SystemError: initialization of _internal failed without raising an exception
- Newest version breaks with UMAP HOT 3
- Slice error using mac M1-max ARM HOT 6
- Exceedingly large amount of memory usage
- Very high memory usage HOT 7
- Specifying threshold in distance metrics
- API to save and load index from disk
- Minor bias in split selection? HOT 1
- true_angular is not a distance?
- TSSS missing a factor of 2
- Reverse diversification is actually forward diversification again
- pynndescent might break with next numba release HOT 3
- How to navigate pyinstaller HOT 1
- Querying the training set: runtime tradeoff for large k HOT 2
- 1 test fails: ZeroDivisionError: division by zero
- `np.infty` replacement
- Speed-up Indexing with custom metrics HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pynndescent.