Comments (5)
I believe the error is different since it was a new bug, introduced when I did some data structure refactoring and apparently missed a few spots.
In good news a lot of this should be made a lot more sane in version 0.5 where everything moves to numba parallelism controlled with numba's new threading control features. This will mean that there will be only a single approach that covers both the serial and threaded cases rather than the current split of approaches that exists now. That should also fix some of the memory use issues with threaded use that you encountered earlier.
from pynndescent.
Hey Leland, unfortunately I am still getting an error with pynndescent 0.4.7 (latest from conda). Looks like the error is different though. So just to state it again, the fist line here works, but the second does not:
nn = NNDescent(X, metric='cosine')
nn = NNDescent(X, metric='cosine', n_jobs=-1)
The error dump log is pretty long. I am not sure which part is the most relevant so here it goes:
~/anaconda3/envs/sonic/lib/python3.7/site-packages/pynndescent/sparse_threaded.py in sparse_init_random(current_graph, inds, indptr, data, dist, n_neighbors, chunk_size, rng_state, parallel, seed_per_row)
95
96 # run map functions
---> 97 for index, status in parallel(parallel_calls(current_graph_map, n_tasks)):
98 if status is False:
99 raise ValueError("Failed in random initialization")
~/anaconda3/envs/sonic/lib/python3.7/site-packages/joblib/parallel.py in __call__(self, iterable)
1015
1016 with self._backend.retrieval_context():
-> 1017 self.retrieve()
1018 # Make sure that we get a last message telling us we are done
1019 elapsed_time = time.time() - self._start_time
~/anaconda3/envs/sonic/lib/python3.7/site-packages/joblib/parallel.py in retrieve(self)
907 try:
908 if getattr(self._backend, 'supports_timeout', False):
--> 909 self._output.extend(job.get(timeout=self.timeout))
910 else:
911 self._output.extend(job.get())
~/anaconda3/envs/sonic/lib/python3.7/multiprocessing/pool.py in get(self, timeout)
655 return self._value
656 else:
--> 657 raise self._value
658
659 def _set(self, i, obj):
~/anaconda3/envs/sonic/lib/python3.7/multiprocessing/pool.py in worker(inqueue, outqueue, initializer, initargs, maxtasks, wrap_exception)
119 job, i, func, args, kwds = task
120 try:
--> 121 result = (True, func(*args, **kwds))
122 except Exception as e:
123 if wrap_exception and func is not _helper_reraises_exception:
~/anaconda3/envs/sonic/lib/python3.7/site-packages/joblib/_parallel_backends.py in __call__(self, *args, **kwargs)
606 def __call__(self, *args, **kwargs):
607 try:
--> 608 return self.func(*args, **kwargs)
609 except KeyboardInterrupt:
610 # We capture the KeyboardInterrupt and reraise it as
~/anaconda3/envs/sonic/lib/python3.7/site-packages/joblib/parallel.py in __call__(self)
254 with parallel_backend(self._backend, n_jobs=self._n_jobs):
255 return [func(*args, **kwargs)
--> 256 for func, args, kwargs in self.items]
257
258 def __len__(self):
~/anaconda3/envs/sonic/lib/python3.7/site-packages/joblib/parallel.py in <listcomp>(.0)
254 with parallel_backend(self._backend, n_jobs=self._n_jobs):
255 return [func(*args, **kwargs)
--> 256 for func, args, kwargs in self.items]
257
258 def __len__(self):
~/anaconda3/envs/sonic/lib/python3.7/site-packages/pynndescent/sparse_threaded.py in current_graph_map(index)
90 rng_state_threads[index],
91 seed_per_row=seed_per_row,
---> 92 sparse_dist=dist,
93 ),
94 )
~/anaconda3/envs/sonic/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
399 e.patch_message(msg)
400
--> 401 error_rewrite(e, 'typing')
402 except errors.UnsupportedError as e:
403 # Something unsupported is present in the user code, add help info
~/anaconda3/envs/sonic/lib/python3.7/site-packages/numba/dispatcher.py in error_rewrite(e, issue_type)
342 raise e
343 else:
--> 344 reraise(type(e), e, None)
345
346 argtypes = []
~/anaconda3/envs/sonic/lib/python3.7/site-packages/numba/six.py in reraise(tp, value, tb)
666 value = tp()
667 if value.__traceback__ is not tb:
--> 668 raise value.with_traceback(tb)
669 raise value
670
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (Heap(array(int32, 2d, C), array(float32, 2d, C), array(uint8, 2d, C)), (Literal[int](0), int64, Literal[int](0)))
* parameterized
In definition 0:
All templates rejected with literals.
In definition 1:
All templates rejected without literals.
In definition 2:
All templates rejected with literals.
In definition 3:
All templates rejected without literals.
In definition 4:
All templates rejected with literals.
In definition 5:
All templates rejected without literals.
In definition 6:
All templates rejected with literals.
In definition 7:
All templates rejected without literals.
In definition 8:
All templates rejected with literals.
In definition 9:
All templates rejected without literals.
In definition 10:
All templates rejected with literals.
In definition 11:
All templates rejected without literals.
In definition 12:
All templates rejected with literals.
In definition 13:
All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at /home/dkobak/anaconda3/envs/sonic/lib/python3.7/site-packages/pynndescent/sparse_threaded.py (40)
File "../../../anaconda3/envs/sonic/lib/python3.7/site-packages/pynndescent/sparse_threaded.py", line 40:
def sparse_current_graph_map_jit(
<source elided>
seed(rng_state_local, i)
if heap[0, i, 0] < 0.0:
^
This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.
from pynndescent.
I copied your entire sparse_threaded.py
into my conda file but unfortunately now instead of a Python error I get a kernel crash:
(sonic) dkobak@lisboa:~$ python
Python 3.7.7 (default, Mar 26 2020, 15:48:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy
>>> from pynndescent import NNDescent
>>> import numpy as np
>>> X = scipy.sparse.csr_matrix(np.random.randn(1000,100)>1)
>>> X
<1000x100 sparse matrix of type '<class 'numpy.bool_'>'
with 15906 stored elements in Compressed Sparse Row format>
>>> nn = NNDescent(X, metric='cosine')
/home/dkobak/anaconda3/envs/sonic/lib/python3.7/site-packages/pynndescent/sparse_nndescent.py:502: NumbaTypeSafetyWarning: unsafe cast from uint32 to int32. Precision may be lost.
init_rp_tree(inds, indptr, data, dist, current_graph, leaf_array)
>>> nn = NNDescent(X, metric='cosine')
>>> nn = NNDescent(X, metric='cosine', n_jobs=-1)
corrupted size vs. prev_size
Aborted (core dumped)
(sonic) dkobak@lisboa:~$
In good news a lot of this should be made a lot more sane in version 0.5 where everything moves to numba parallelism controlled with numba's new threading control features. This will mean that there will be only a single approach that covers both the serial and threaded cases rather than the current split of approaches that exists now. That should also fix some of the memory use issues with threaded use that you encountered earlier.
This sounds great! Looking forward.
from pynndescent.
Blech; kernel crashes inside numba are a pain to track down. I'll see what I can figure out.
from pynndescent.
Hi Leland, I just noticed that this seems to be resolved in 0.5 (at least everything works fine for me), and moreover the default behaviour (n_jobs=None
) seems to be using n_jobs=-1
(btw, this was not entirely clear to me from the docstring). So I am closing this issue!
from pynndescent.
Related Issues (20)
- Sample identifiers for semantic search HOT 2
- uint8 as internal data HOT 1
- Cosine metric - error "Negative values in data passed to precomputed distance matrix" HOT 2
- Question about covariance matrix used when using Mahalanobis distance
- Tests fail: E SystemError: initialization of _internal failed without raising an exception
- Newest version breaks with UMAP HOT 3
- Slice error using mac M1-max ARM HOT 6
- Exceedingly large amount of memory usage
- Very high memory usage HOT 7
- Specifying threshold in distance metrics
- API to save and load index from disk
- Minor bias in split selection? HOT 1
- true_angular is not a distance?
- TSSS missing a factor of 2
- Reverse diversification is actually forward diversification again
- pynndescent might break with next numba release HOT 3
- How to navigate pyinstaller HOT 1
- Querying the training set: runtime tradeoff for large k HOT 2
- 1 test fails: ZeroDivisionError: division by zero
- `np.infty` replacement
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pynndescent.