pairs.sandbag crashes about pypairs HOT 10 CLOSED

Hi @fidelram ,

Numba is the module used for the jit-compiling. Probably system specific. Can you tell me OS, and the versions of PyPairs and Numba (- was numba installed via PyPi)? Then I will search along, have not seen this error yet.

In the meantime, you could try lowering the number of cores used with settings.n_jobs to 1. This way numba does not try to parallelize.

from pypairs.

Hi,

I tried reinstalling numba (using the numba channel of conda) but the problems persists.

We are using scientific linux and I have installed all packages using conda. Versions are:

numba: 0.42
pyPairs: 3.0.11

Now is running in single core mode, but I assume this will take a while.

from pypairs.

So both numba versions PyPi and Conda fail, right? After some googling it seems to be an issue with the threading layer of numba and some mislinked libraries. It might be worth a shot trying a different one..

The threading layers available are tbb, omp and workqueue, where it seems tbb is installed on your machine and was used. You can set it to workqueue with the enviroment variable NUMBA_THREADING_LAYER, that is build-in, or use omp with OpenMPI installed. See:
https://numba.pydata.org/numba-doc/latest/user/threading-layer.html

I will try to add a scientific linux instance for debugging.

Please let me know if you have some more details, or changing the threading layer works.

Yes, this might take a while depending on the dataset size.

from pypairs.

Thanks, I tried to re-install tbb but didn't improve. Y also shifted to workqueue but still does not work.

Since the backtrace points to /lib64/libc.so.6 I found this issue: pytorch/pytorch#2205 could be related?

from pypairs.

Yeah this seems fairly close. How much memory did you have / made available and how big is your dataset? Then I will investigate further. And maybe try to decrease the memory footprint of sandbag even further.

from pypairs.

I should have enough memory (over 50Gb). The dataset I am using is the test dataset leng15

from pypairs.

Still working on getting Scientific Linux 7.6 (Nitrogen) up and running on travis. Then I can investigate further. I'll let you know once I have some news.

from pypairs.

It seems I can't recreate your bug.

Running pytest with:

NAME="Scientific Linux"
VERSION="7.6 (Nitrogen)"
ID="scientific"
ID_LIKE="rhel centos fedora"
VERSION_ID="7.6"
PRETTY_NAME="Scientific Linux 7.6 (Nitrogen)"

passes all tests.

See: https://travis-ci.org/rfechtner/pypairs/jobs/496672905

from pypairs.

@fidelram FYI, you can also use the default marker pairs to skip the sandbag part and only use cyclone:

Either by loading the markers from datasets.default_cc_marker() to have a look at them

import pypairs import pairs, datasets
adata = datasets.leng15('unsorted')
marker_pairs = datasets.default_cc_marker()
scores = pairs.cyclone(adata, marker_pairs)

(see: https://pypairs.readthedocs.io/en/latest/pypairs.datasets.default_cc_marker.html#pypairs.datasets.default_cc_marker)
or simply call pairs.cyclone() without the markers parameter, that will also use the default marker pairs as above:

import pypairs import pairs, datasets
adata = datasets.leng15('unsorted')
scores = pairs.cyclone(adata)

Cheers,

Ron

from pypairs.

It seems this issue was caused by bad memory access. This should now be solved by the last two PRs.
New release is on the way

from pypairs.