Large memory use when loading file with np.memmap in recent dask versions about dask HOT 8 CLOSED

Thank you all for the minimal reproducer and the information about versions. If folks wanted to take this further, the next step here is probably to use git bisect to find the change that caused this. My guess is that that makes it much easier to diagnose.

from dask.

I also tested this on a Windows computer, with a similar python install: same issue.

• 2024.1.1 (almost) no memory use
• 2024.5.1: About 13 GB memory use, and takes about 1 minute

from dask.

I tested this on my Windows computer and I get the same behaviour:

• 2024.5.1 and 2024.4.1: about 21 GB memory use, taking about 30 secs
• 2024.1.1: Instant and no memory use

from dask.

I ran git bisect to locate the bad commit, testing through about 80 revisions. And here is the final output:

f51fa77f4cbba7a92a54a760da65a4b6e712e4ad is the first bad commit
commit f51fa77f4cbba7a92a54a760da65a4b6e712e4ad
Author: crusaderky <[email protected]>
Date:   Tue Feb 6 17:50:42 2024 +0000

    Make tokenization more deterministic (#10876)

 dask/array/ufunc.py          |   4 +-
 dask/base.py                 | 176 ++++++------
 dask/tests/test_base.py      |  12 +-
 dask/tests/test_delayed.py   |  12 +-
 dask/tests/test_highgraph.py |   6 +-
 dask/tests/test_tokenize.py  | 670 +++++++++++++++++++++++++++----------------
 dask/utils.py                |  18 +-
 7 files changed, 534 insertions(+), 364 deletions(-)

The commit is this one here. There seems to be a lot of changes to the base.py file, where the problem could have arisen.

from dask.

Thanks for doing that work!

Looking more closely at the commit it looks like some code was removed that used to handle memmap files.

f51fa77#diff-10422b02c591d63ee295724faa14f7698b4a742c98ba20771c5f70d1a6926d06L1309-L1327

cc @fjetter when he gets back (on PTO currently) or maybe @jrbourbeau if he wants a quick fix

from dask.

Excellent! Would this be a regression? Or is there some other way of "lazily" loading binary data like with dask?

from dask.

Thing used to work. Thing no longer works. Sounds like a regression to me 🙂

I'd say most people doing lazy loading of array data tend to use more established formats like Zarr or HDF today. memmapped numpy files are rare these days (probably why this issue went unreported for so long).

If someone was starting a new project today I'd point them to https://zarr.dev/ which seems to be becoming the standard surprisingly quickly across a variety of fields.

from dask.

For most of our work with large array data, we do use Zarr. The reason for having to work with binary data here, is to load data generated by a type of very fast camera. Thus, the raw data itself comes out as binary files.

For all of my own data processing needs it is zarr all the way :)

from dask.