Comments (7)
So definitely such thing does not exist (a stream which can read both formats).
I was working on assumption that you (the user) can tell them apart yourself (like ProtocolVersion
field in database, or different extension in filename).
First 4 bytes of NEW stream is a magic number '0x184D2204' maybe this might help? (see:
)if magic number is not there, than InvalidDataException is thrown (see:
)I know this is not ideal, as you would still need to open stream, read 4 bytes, and open stream again, but that's all I can offer you at the moment.
from k4os.compression.lz4.
Ok, thanks for the feedback.
Closing the issue as you said:
I was working on assumption that you (the user) can tell them apart yourself
from k4os.compression.lz4.
Actually, @MiloszKrajewski there's one more thing I'd like to know.
In the old LZ4.NET library we used to have Wrap
and Unwrap
functions that accept byte arrays and compress/decompress them returning another byte array.
I see these methods in this repository as well in Legacy
module. Is there any other method that does the same thing outside of Legacy
. I found LZ4Pickle
class that has functions with signatures byte[] -> byte[]
. However, when using the Pickle
method, the output byte array doesn't begin with the MagicNumber
, which suggests it doesn't do the same thing as LZ4Stream
.
Most likely I'm missing a single piece here in order to understand everything :).
from k4os.compression.lz4.
So Pickle/Unpickle has the same purpose as Wrap/Unwrap (thus same signature) but is not compatible,
I was not planning backwards compatibility so Pickle/Unpickle does not any magic number. It is forward compatible (I reserved 3 bits for version) but not backwards.
You can use Legacy assembly one to read old ones (Unwrap) and write them in new format (Pickle), but knowing which ones are old/new is on you.
For example, I had a cache with lots of blobs packed with old LZ4. On migration I've just added new column (let's call it CompressionAlgorithm
at set it to 0
top to bottom).
Now every time new entry is written then I use Pickle
and CompressionAlgorithm
is stored as 1
. On read, I read CompressionAlgorithm
first and decide to use Unpickle or Unwrap depending if it is 0 or 1.
You could also use Pickle with IBufferWriter overload to do you own prefixing with magic number.
I do understand this is suboptimal and it would be much better if it was backwards compatible, but it isn't...
Any legacy support was not even planned at first and added much later (see: #20)
from k4os.compression.lz4.
Most likely I'm missing a single piece here in order to understand everything :).
Unfortunately you are asking very legitimate questions and I'm sorry that answers are most of the time: "you have to work around it yourself".
from k4os.compression.lz4.
Thanks for responding once again.
This all sounds reasonable and saves me a lot of time guessing. Much appreciated!
I found an easy way to implement Wrap
and Unwrap
using Streams, so if I need them, I know what to do, no worries.
from k4os.compression.lz4.
Stream come with quite large overhead. It is fine if we are talking about megabytes of data, but for short messages pickle is much better.
Try code below. It will, of course depend on size of your messages but if they are below 64k it will be much much (much) quicker than stream:
public static byte[] MyPickleMagic =
BitConverter.GetBytes(0x13371234);
public static void MyPickle(
ReadOnlySpan<byte> source, IBufferWriter<byte> target)
{
target.Write(MyPickleMagic);
LZ4Pickler.Pickle(source, target);
}
public static void MyUnpickle(
ReadOnlySpan<byte> source, IBufferWriter<byte> target)
{
if (!source.SequenceEqual(MyPickleMagic))
throw new ArgumentException(
"Pickle magic does not match");
LZ4Pickler.Unpickle(
source.Slice(MyPickleMagic.Length), target);
}
from k4os.compression.lz4.
Related Issues (20)
- h264 nal unit invalid HOT 2
- Implicit reference to System.Runtime.CompilerServices.Unsafe appears to cause wrong version to be selected HOT 3
- Chained encoder does not produce the same result as the lz4 cli when chaining is enabled HOT 4
- Migrating from L4zNet to new implementation
- InvalidDataException: LZ4 frame magic number expected HOT 10
- Implement frame format with K4os.Compression.LZ4 HOT 2
- Question about random access, i want to make something similar to the original dictionaryRandoimAccess.c example HOT 5
- why Length return 7 HOT 1
- Question regarding Lz4net archived repo HOT 6
- LZ4 Legacy format support HOT 2
- LZ4EncoderStream Dispose Issue HOT 3
- Compression Issue? HOT 3
- API: LZ4Codec.Decode/Encode(Stream input, Stream output) HOT 12
- LZ4 with Maui HOT 7
- Stream API CopyTo got NotSupportedException HOT 4
- LZ4DecoderStream ReadAsync calls original stream's sync Read internally HOT 2
- [Question] Block vs Stream vs Pickler to store compressed bitmaps in memory HOT 10
- Adding non compressed data at the end of the stream HOT 2
- The `int LZ4Frame#Encode` overloads don't close the frame HOT 2
- The repository contains the code with the incompatible licenses HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k4os.compression.lz4.