use sparse arrays from xarray about atlite HOT 6 CLOSED

@euronion could you point me to the parts where atlite uses the flattened sparse arrays you talked about?

from atlite.

Thanks @FabianHofmann for pointing this out, that sounds really promising. Especially as sparse is from the pydata community aiming to replace scipy.sparse.

The lines in question are basically the operations around the layoutmatrix and aggregation method in

from atlite.

As long as

It lacks layouts that are not easily generalized like CSR/CSC and depends on scipy.sparse for some computations.

is true, it will not help our use case which derives its speed from CSR-based matrix multiplication.

from atlite.

Conversion to CSR is simple COO(...).tocsr(), as sparse is scipy.sparse compatible. I am trying a few things over at PyPSA-EUR with the xarray sparse support, significantly improving code readability.

from atlite.

So a quick verdict based on the status quo (sparse only supporting COO-format and many matrix operations not yet fully supported between non-sparse and sparse xarray DataArrays):

• Suitable for use in scripts, like in PyPSA-EUR
• Not suitable in general for atlite

It boils down to being usable where only small sparse arrays (<= ~ 1Mio. NNZs) or few sparse-dense operations are involved. The performance impact is measureable but acceptable. For time-series aggregation in atlite I did a quick test on a common PyPSA-EUR system (189x157 grid, ca. 3000 buses/regions):

• Necessary operations for bringing the time-series data into the right format (conversion to sparse array or index-stacking and reordering): Using sparse ca. 4x slower (~15-20s for sparse / one weather year)
• Aggregation (matrix broadcasting and multiplication): sparse ca. 200x slower (~5-10 minutes for sparse / one weather year)

Timings did vary since I didn't do a really clean performance test (- didn't consider it necessary).

Also the memory overhead was noticable due to manual conversion becoming necessary between non-sparse and sparse xarrays.

I did not test the "dirty" option of converting to scipy.sparse arrays. I don't think it's worth investing time at the moment but rather wait till sparse get's support for more formats. That should be the next big feature according to their roadmap.

@nworbmot : The numbers you were interested in earlier.

from atlite.

I'm closing this due to the above evaluation.

from atlite.