Possible 'overallocation' of memory about java-merge-sort HOT 5 CLOSED

Sounds plausible, I'll have a look. Thanks!

from java-merge-sort.

Quick comment on memory size calculations: assumption is that usually most memory is not used for Object[] itself, but rather for entries contained. Heuristics are approximate of course.

from java-merge-sort.

you are correct, but if the estimated size is grossly off (we originally estimated 24 bytes/object, then sat down a bit and really looked at it, this was a good reference: http://www.ibm.com/developerworks/java/library/j-codetoheap/index.html) we realised for an Object wrapping a String and an long, it actually adds up to 106 bytes or something, so we were 4x wrong! this then makes java-merge-sort choose to use 4x as many to fit the buffer.

If we combine that silly original estimate, with the fact that we were running Java 6 U31 64bit which used Compressed OOPs and_then had to downgrade to a JVM prior to the Compressed OOPS support (long story on the why, irrelevant here), plus the maybe-doubling of the Object[] above, our 2Gb heap pretty much vanished! :)

The 2nd copy of the array is a problem still I believe, because the user of the library will allocate memory based on their understanding of the JVM environment they're running in, and if it's now doubling it, that may well be a penalty they can't afford (particularly if their object size calcs are pretty dodgy.. :) )

from java-merge-sort.

Yes, forgot to mention that the estimate obviously has to be close enough -- and Strings are expensive for multiple reasons.

I also think you may well be right that there is unintended reference retention, and that would be good to eliminate.
Especially if there are duplicate Strings from previous rounds.

from java-merge-sort.

Ok, yes. Some of things were not getting cleared as they should have, including one you pointed out. I added two smaller clean up fixes; I hope those help a bit too.

I released a new version (actually, well, two, finding one more thing to fix), see if 0.7.1 behaves any better for you?
Further suggestions for improvements are gladly accepted. :-)

from java-merge-sort.