Drop original capacity and implement truncate_front about bytes HOT 10 CLOSED

The intention of tracking the original allocation size was to handle a regression when transitioning to tokio-io. Part of the problem is that Decoder don't know the size of the originally allocated buffer. So Decoder is expected to reserve only as much as is needed for the current frame, but then, you can get into a situation where tiny reallocations are required.

Being able to truncate_front would be nice... but I am unsure what the best trade off is... At this point, removing the original capacity tracking would require either figuring out how to do so w/o it being a breaking change OR figure out a way to migrate off for tokio-io 0.2.

from bytes.

Shouldn't FramedRead store initial capacity explicitly, and use it in reserve, and do not rely on Bytes property of preserving capacity in split_* and take operations?

It should be a mostly backward-compatible change of tokio-io (it will be source-compatible, but behavior will change slighly), no need to wait for tokio-io 0.2.

from bytes.

FramedRead does not call reserve. The Decoder impl calls reserve as needed.

from bytes.

Even if Decoder impl calls reserve, FramedRead could also call reserve before read_buf with initial capacity.

from bytes.

How does FramedRead know when to call reserve? If it calls reserve w/ the initial capacity each "tick" then there is no point in using Bytes because every iteration will allocate a brand new chunk.

Decoder is the thing that knows when to call reserve correctly, but it doesn't know the "right" amount... FramedRead knows this.

from bytes.

OK, I think I got it.

FramedRead could call reserve(initial_capacity) right before read_buf. However, in that case it will be one unnecessary reallocation per read_buf if last reserve call in Decoder reserved less than initial_capacity.

So it is not possible to make this change without breaking/changing tokio-io.

from bytes.

Somewhat related, what's the cheapest way of "consuming" the first N bytes of Bytes right now?
I'm using split_off(..)

from bytes.

So, tokio-io Decoder::decode could be changed to something like:

// Ok(Ok(item)) means it was decoded successfully
// Ok(Err(44)) means that decoder needs 44 bytes to decode something (e. g. a header or an item)
fn decode(&mut self, src: &mut BytesMut) -> Result<Result<Self::Item, usize>, Self::Error>;

And FramedRead::poll should do reserve with error value of inner result.

Right?

from bytes.

Technically, yes. However, the goal of Encoder and Decoder is to be as simple as possible. I think we already have strayed from that goal too much.

from bytes.

Closing this issue in favor of tokio-io issue I've just created.

from bytes.