H3Core.nonZeroLongArrayToList takes too much memory about h3-java HOT 3 OPEN

Hi, sorry for the delayed response! Yes, I think counting the number of non-zero indexes is a reasonable change. Off hand I don't know the performance of the other option, but I assume you want to make sure it does not keep the entire result array alive. Alternately we could do something like allowing the ArrayList to dynamically resize, but I suspect that would always be slower.

I'm curious if the result array itself already takes up roughly 8GB memory - is that being shown in your memory profiler with the 8GB of memory already being used?

from h3-java.

Hi @isaacbrodsky

I just stumbled across this issue when trying to calculate all H3 indexes on a 2.5km long street. (red line in the image at the bottom)

maxPolyfillSize returns 222'668'225. But the actual number of non-zero elements is 4'889. So the memory overhead is huge 🥲

What's your preferred solution for the problem? I cannot really make a statement on which one's the best regarding performance and memory consumption.

One could also solve it using streams:

return Arrays.stream(out)
        .filter(idx -> idx > 0)
        .boxed()
        .collect(Collectors.toList());

In terms of readability I'd probably prefer this solution.

from h3-java.

In my case the size of the ArrayList is only one small part of the problem. The actual problem is, that the number returned by maxPolyfillSize is way too big. I opened another issue here: uber/h3#708

from h3-java.