Comments (5)
I'm not sure if I grasp what code you mentioned really does and how it does it, but I do have some thoughts:
- initial block sizes are known, so when you seek you can quickly skip blocks without decompressing them
- ...unless they are chained / dependent blocks
So Seek / Read is kind of possible with caveats:
- independent blocks only
- skip blocks without decompression
- decompress whole block containing bytes you need
from k4os.compression.lz4.
To plug Seek
into stream you would need to start somewhere in LZ4FrameReader.vs but then a lot of work would be in LZ4BlockDecoder.cs.
This is kind of doable but all abstractions are built with no Seek in mind at all, see:
IStreamReader.cs
from k4os.compression.lz4.
Just a little clarification, i dont really need to seek the compressed stream in the lib, altrought that will be nice to have too and i may look into that too. Im looking to make a .net compressor/decompressor program that can produce the files in this format, the decompress process in the lib should be done normally (the entire file from start to finish). A open source C++ game called Freespace Open is the one that will read and use those files using random access.
You are right about the block sizes and their position on the stream, a offset table should not be needed, thats a good point, i made that implementation over two years ago and im reading back what i did and i cant understand it either, i mostly followed the official example, maybe im missing something, but it really seems to contain the block position in the stream.
If im correct, if i create a stream right now, with independient blocks, it will have the lz4 frame format https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md?
First i want to produce a file with the structure i used following the random access example + the two extra ints as thats is the structure that the game uses, then i want to see if i can make work with the lz4 frame format, if this is possible i can look into adding the seeks here too.
Do you think is possible to change the output format to the one i need? (im not looking to PR that, just to use it for this expecific implementation).
If you are curious, this is my original work with the random access example when i made that implementation over two years ago: (yes i posted this issue from the wrong account)
Compression:
https://github.com/Shivansps/FS2-VP-Compression-Concept/blob/main/VPCompressionTest/VPCompressionTest.cpp#L435
Random Access Read:
https://github.com/Shivansps/FS2-VP-Compression-Concept/blob/main/VPCompressionTest/VPCompressionTest.cpp#L285
from k4os.compression.lz4.
Hi, i just re-checked the C code i did 2 years ago, it works like this:
Since each block contains BLOCK_SIZE of uncompressed data, except maybe the last, the trailing offset table contains the seek position of each block in the stream, and the position you are looking for % BLOCK_SIZE gives the block number were that data is.
Not sure if this can be done whiout the offset table what makes it incompatible with the lz4 frame format i belive? If it can be done with the lz4 frame format it would be also be great to know because i can certanly change the game to accept that too.
But if you can at least help me to point me to were i need to look in order to change the format in order to produce something like this i would be very grateful.
Thank you.
from k4os.compression.lz4.
Hi, sorry, in the end i found another lib were i was able to add what i needed very quickly. Thanks for you help anyway. Ill close this issue when i get the chance to log in with my other account.
from k4os.compression.lz4.
Related Issues (20)
- Unity Support HOT 3
- Implicit reference to System.Runtime.CompilerServices.Unsafe appears to cause wrong version to be selected HOT 3
- Chained encoder does not produce the same result as the lz4 cli when chaining is enabled HOT 4
- Migrating from L4zNet to new implementation
- InvalidDataException: LZ4 frame magic number expected HOT 10
- Implement frame format with K4os.Compression.LZ4 HOT 2
- why Length return 7 HOT 1
- Question regarding Lz4net archived repo HOT 6
- LZ4 Legacy format support HOT 2
- LZ4EncoderStream Dispose Issue HOT 3
- Compression Issue? HOT 3
- API: LZ4Codec.Decode/Encode(Stream input, Stream output) HOT 12
- LZ4 with Maui HOT 7
- Stream API CopyTo got NotSupportedException HOT 4
- LZ4DecoderStream ReadAsync calls original stream's sync Read internally HOT 2
- [Question] Block vs Stream vs Pickler to store compressed bitmaps in memory HOT 10
- Adding non compressed data at the end of the stream HOT 2
- The `int LZ4Frame#Encode` overloads don't close the frame HOT 2
- The repository contains the code with the incompatible licenses HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k4os.compression.lz4.