ir_dist alignment stuck about scirpy HOT 4 CLOSED

Hi Rachel,

thanks for reporting this. I'm not entirely sure what could be causing this issue.

How many cells does your mdata have? And do you have the same problem when you use a small example dataset (e.g. scirpy.datasets.wu2020_3k()?

from scirpy.

My mdata has 71706 cells in the airr modality and iedb database has 34509 TCRs.
I don't have the same problem if I use a small dataset.
I think something happened at the end of VJ dist mat calculation, where a thread fails to shut down and then it gets stuck in lock indefinitely, suggested by this traceback:

File ~/mambaforge/envs/compbio/lib/python3.10/threading.py:1096, in Thread.join(self, timeout)
   1093     raise RuntimeError("cannot join current thread")
   1095 if timeout is None:
-> 1096     self._wait_for_tstate_lock()
   1097 else:
   1098     # the behavior of a negative timeout isn't documented, but
   1099     # historically .join(timeout=x) for x<0 has acted as if timeout=0
   1100     self._wait_for_tstate_lock(timeout=max(timeout, 0))

File ~/mambaforge/envs/compbio/lib/python3.10/threading.py:1116, in Thread._wait_for_tstate_lock(self, block, timeout)
   1113     return
   1115 try:
-> 1116     if lock.acquire(block, timeout):
   1117         lock.release()
   1118         self._stop()

KeyboardInterrupt:

Could it be a generic threading problem? open-telemetry/opentelemetry-python#2284

from scirpy.

You could try to increase the block size, which will result in less threads that could be stuck :D The only downside is that the progressbar doesn't update as continuously:

dist_calc = ir.ir_dist.metrics.AlignmentDistanceCalculator(cutoff, block_size=5000, n_jobs=32, ...)
sc.pp.ir_dist(adata, metric=dist_calc)

from scirpy.

Hi @racng,

I believe I found a solution to this in #473.
joblib.Parallel is allegedly more robust than the multiprocessing module from the standard library.

An added benefit is that you can also use the dask backend via joblib.parallel_config to distribute the workload across multiple machines. The version from that PR also has an improved version of the alignment distance that is faster.

Would be great if you can give it a try! You can install it with

pip install [email protected]:icbi-lab/scirpy.git@ir-dist-parallelism

from scirpy.