ExpandableDirectByteBuffer maximum capacity only 1GB about agrona HOT 6 CLOSED

ByteBuffer and byte arrays cannot be larger be larger than int capacity.

from agrona.

ByteBuffer and byte arrays cannot be larger be larger than int capacity.

That is, if Argon would like to support a ring buffer of this size, then the only option is to work with files directly (memory-mapped files, dev/shm, etc.)?

Mapped files in Java still return a ByteBuffer so still not possible. There are techniques that can be used which are unsupported in the JDK. We can provide commercial consulting on that but we do not provide it as free support in Agrona.

from agrona.

First of all as pointed out @mjpt777 the underlying size cannot exceed int (an actual value is Integer.MAX_VALUE - 8). That being said we could allow ExpandableDirectByteBufferto grow until that max value as oppose to 1GB.

If you need more than 2GB of storage you can use org.agrona.concurrent.MappedResizeableBuffer.

from agrona.

This would require changing all of the accessor methods that take in a index value and switch it from a int to a long. This in turn would break compatibility with the array backed implementations of the DirectBuffer.

from agrona.

This would require changing all of the accessor methods that take in a index value and switch it from a int to a long. This in turn would break compatibility with the array backed implementations of the DirectBuffer.

What do you think, are there other pitfalls besides compatibility that prevent the increase in the maximum size? Only this limitation does not allow using Aeron in our project, where the buffer size must be at least 60 gigabytes.

from agrona.

ByteBuffer and byte arrays cannot be larger be larger than int capacity.

That is, if Argon would like to support a ring buffer of this size, then the only option is to work with files directly (memory-mapped files, dev/shm, etc.)?

from agrona.