[Feature] In memory LZ4Frame decompression about k4os.compression.lz4 HOT 7 CLOSED

Currently I don't provide ContentLength as Stream.Length but this might be relatively easy thing to do (not trivial though, as it is expected to be available right after construction, so reading of inner file would need to happen before reading lz4 stream, kind of non-obvious side effect).
Once you have frame length you could use https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream.

I know it is still not ideal, but much more plausible in short term.

from k4os.compression.lz4.

In version 1.0.2 you can now read content length (if compressed with content length).

from k4os.compression.lz4.

1.3.0-beta has a lot of new streaming methods and some of them are very memory friendly, let's say.

from k4os.compression.lz4.

Isn't LZ4Pickler.Pickle and LZ4Pickler.Unpickle do (almost) exactly that?

from k4os.compression.lz4.

@MiloszKrajewski do Pickle and Unpickle handle Frames?

What I am trying to do is to read and play ROS-Bag files... they're essentially video files, where every frame can be compressed using LZ4 or BZ2... I usually need to decompress between 60 and 90 images per second, so I'm trying to avoid any unnecesary byte copy. I am also using ArrayPool to reuse buffers.

from k4os.compression.lz4.

That's fair. From what I've seen every frame is ROS bag file is a mini LZ4Stream. I guess all the magic is in https://github.com/ros/ros_comm/blob/melodic-devel/utilities/roslz4/src/lz4s.c but it also seems to me like customized lz4 implementation so I'm not exactly sure if it would be easy to implement.

I understand LZ4Stream.Decode(new MemoryStream(lz4bytes)).ReadToEnd() is what you need but with less memory allocations?

from k4os.compression.lz4.

I understand LZ4Stream.Decode(new MemoryStream(lz4bytes)).ReadToEnd() is what you need but with less memory allocations?

Exactly. Right now it's what I am using, and it's decoding the LZ4 miniblocks just fine.

But in order to improve performance, any memory reallocation is a bottleneck. I am using Spans, ArraySegments and ArrayPools everywhere...

I also would like to try with realtime recording/encoding; right now I am recording raw frames at 500mb/second throughput, and it's already at the limit. So my plan is that if realtime encoding is not fast enough, I would do it in a post process pass.

from k4os.compression.lz4.