Question about random access, i want to make something similar to the original dictionaryRandoimAccess.c example about k4os.compression.lz4 HOT 5 CLOSED

I'm not sure if I grasp what code you mentioned really does and how it does it, but I do have some thoughts:

• initial block sizes are known, so when you seek you can quickly skip blocks without decompressing them
• ...unless they are chained / dependent blocks

So Seek / Read is kind of possible with caveats:

• independent blocks only
• skip blocks without decompression
• decompress whole block containing bytes you need

from k4os.compression.lz4.

To plug Seek into stream you would need to start somewhere in LZ4FrameReader.vs but then a lot of work would be in LZ4BlockDecoder.cs.

This is kind of doable but all abstractions are built with no Seek in mind at all, see:
IStreamReader.cs

from k4os.compression.lz4.

Just a little clarification, i dont really need to seek the compressed stream in the lib, altrought that will be nice to have too and i may look into that too. Im looking to make a .net compressor/decompressor program that can produce the files in this format, the decompress process in the lib should be done normally (the entire file from start to finish). A open source C++ game called Freespace Open is the one that will read and use those files using random access.

You are right about the block sizes and their position on the stream, a offset table should not be needed, thats a good point, i made that implementation over two years ago and im reading back what i did and i cant understand it either, i mostly followed the official example, maybe im missing something, but it really seems to contain the block position in the stream.

If im correct, if i create a stream right now, with independient blocks, it will have the lz4 frame format https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md?

First i want to produce a file with the structure i used following the random access example + the two extra ints as thats is the structure that the game uses, then i want to see if i can make work with the lz4 frame format, if this is possible i can look into adding the seeks here too.

Do you think is possible to change the output format to the one i need? (im not looking to PR that, just to use it for this expecific implementation).

If you are curious, this is my original work with the random access example when i made that implementation over two years ago: (yes i posted this issue from the wrong account)
Compression:
https://github.com/Shivansps/FS2-VP-Compression-Concept/blob/main/VPCompressionTest/VPCompressionTest.cpp#L435
Random Access Read:
https://github.com/Shivansps/FS2-VP-Compression-Concept/blob/main/VPCompressionTest/VPCompressionTest.cpp#L285

from k4os.compression.lz4.

Hi, i just re-checked the C code i did 2 years ago, it works like this:
Since each block contains BLOCK_SIZE of uncompressed data, except maybe the last, the trailing offset table contains the seek position of each block in the stream, and the position you are looking for % BLOCK_SIZE gives the block number were that data is.

Not sure if this can be done whiout the offset table what makes it incompatible with the lz4 frame format i belive? If it can be done with the lz4 frame format it would be also be great to know because i can certanly change the game to accept that too.

But if you can at least help me to point me to were i need to look in order to change the format in order to produce something like this i would be very grateful.

Thank you.

from k4os.compression.lz4.

Hi, sorry, in the end i found another lib were i was able to add what i needed very quickly. Thanks for you help anyway. Ill close this issue when i get the chance to log in with my other account.

from k4os.compression.lz4.