Suggestion for stack size when calling par_sort_unstable about rayon HOT 5 OPEN

I don't there can be a standard suggestion, because it depends on both the number of elements and the number of threads in the pool, at least. In theory it will have logarithmic depth, but work-stealing will lengthen that further.

I wonder if we need to be smarter about MAX_SEQUENTIAL here:

i.e. rather than just a fixed constant, maybe that should also consider num_elem / num_threads and some additional factor to account for imbalance. Then it should use a higher limit when you're starting with billions, and you won't get so much stack use due to work stealing.

Similarly, the stable sort has two constants, CHUNK_SIZE while splitting and MAX_SEQUENTIAL while merging:

from rayon.

Well, what if I know the number of elements, their size, and the number of threads in the pool? Just ballpark.

from rayon.

I don't know. You'll get a better answer if you run a few experiments yourself.

from rayon.

I'm running into the same issue and have found that around 5-10B elements the default of 2MB seems to break, so I'm testing out a heuristic of using max[1, log2(N / 5e9)] * 2 MB right now.
(this is sorting a memory mapped slice of u64 on a machine with 48 physical cores)

from rayon.

Update: I actually ran into a bus error (core dumped) when using the logarithmic scaling. Linear scaling would sort of defeat the purpose of memory mapping the data, so I might try square root scaling.

from rayon.