I've been trying to get this to work with sparse matrices. The setup

Thanks for <a class="issue-link js-issue-link" data-error-text="Failed to load title"

Problems with sparse about pynndescent HOT 7 OPEN

lmcinnes commented on August 27, 2024 1

Problems with sparse

from pynndescent.

Comments (7)

lmcinnes commented on August 27, 2024

Hmm, definitely still some issues -- things got brought over from umap, but apparently there are some issues. I don't have time to dig into this right now, but hopefully in a week or two I will have time.

from pynndescent.

pavlin-policar commented on August 27, 2024

Thanks! Let me know if I've missed out any useful information you might need.

from pynndescent.

lmcinnes commented on August 27, 2024

I finally got some time and dug into this. The short answer was that not everything got fully ported over from umap, and there were some issues with sparse support in threaded mode (is as much as that wasn't yet implemented). I think I've got that all fixed now. If you can try cloning the current master and let me know if that resolves the issue for you I would appreciate it.

from pynndescent.

pavlin-policar commented on August 27, 2024

Thanks so much for looking into this! Both of the problems I mentioned seem to be fixed; I can use it on sparse just like I would expect.

Unfortunately, building and querying the index now raises several numba warnings, even when I run it on dense data e.g.

/home/pavlin/dev/pynndescent/pynndescent/sparse_nndescent.py:121: NumbaPerformanceWarning: 
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.

File "pynndescent/utils.py", line 459:
@numba.njit(parallel=True)
def new_build_candidates(
^

  False,
/home/pavlin/dev/pynndescent/pynndescent/sparse_nndescent.py:82: NumbaPerformanceWarning: 
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.

File "pynndescent/utils.py", line 79:
@numba.njit(parallel=True)
def rejection_sample(n_samples, pool_size, rng_state):
^

  indices = rejection_sample(n_neighbors, n_vertices, rng_state)

when building the index and

/home/pavlin/dev/pynndescent/pynndescent/sparse_nndescent.py:273: NumbaPerformanceWarning: 
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.

File "pynndescent/sparse_nndescent.py", line 175:
@numba.njit(parallel=True)
def sparse_init_from_random(
^

  dist_args,
/home/pavlin/dev/pynndescent/pynndescent/sparse_nndescent.py:288: NumbaPerformanceWarning: 
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.

File "pynndescent/sparse_nndescent.py", line 207:
@numba.njit(parallel=True)
def sparse_init_from_tree(
^

  dist_args,
/home/pavlin/miniconda3/envs/tmp/lib/python3.7/site-packages/numba-0.44.0-py3.7-linux-x86_64.egg/numba/ir_utils.py:1958: NumbaPendingDeprecationWarning: 
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'forest' of function 'sparse_initialise_search'.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "pynndescent/sparse_nndescent.py", line 248:
@numba.njit()
def sparse_initialise_search(
^

  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))

when querying the index.

from pynndescent.

tomwhite commented on August 27, 2024

It looks like the warnings have started appearing from Numba 0.44. I've opened #68 to remove the ones related to parallel=True. There are two more to do with 'reflective list' being deprecated, but those can't be easily fixed until the replacement appears in Numba 0.45. In the meantime the warnings can be ignored, or Numba 0.43 (or earlier) can be specified explicitly when installing.

from pynndescent.

lmcinnes commented on August 27, 2024

Thanks for #68 @tomwhite, that shoudl take care of a lot of them. This is popping up in umap as well, so I will have to take care of it there soon.

from pynndescent.

dkobak commented on August 27, 2024

There are two more to do with 'reflective list' being deprecated, but those can't be easily fixed until the replacement appears in Numba 0.45.

@tomwhite I haven't seen those warnings myself, but I think Numba 0.45 is out already. Do you know what's the status of those warnings?

from pynndescent.

Problems with sparse about pynndescent HOT 7 OPEN

Comments (7)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent