Comments (5)
Is there something specific you are looking for comments on?
from aircompressor.
I'm wondering if the test result is OK since this project claimed that
it faster than other implementation by 10%~40%
But for snappy decompression, it is about 30% slower.
from aircompressor.
After a quick look, I'd guess that either there is a regression/bug in aircompressor, the native snappy decompressor got a lot better, or the native snappy decompressor is running without bounds checks (very unsafe). BTW, we switched all of our uses from Snappy to LZ4, because was better in all of our use cases.
from aircompressor.
em... also the test shown that the airlift version of LZ4 is not obviously better than the jni version.
from aircompressor.
Ah, think I understand now. I believe you are asking the question "why would I use this project when I can use the JNI wrappers?" We created this project to avoid JNI for usability, portability issues and the effect is has on the the GC. In Hotspot based JVMs, you have two choices for JNI operating on heap data, you can copy it to native memory, or you can lock the heap and operate on it without copy. The fist mode has a computational cost, but in my mind the bigger problem is the complexity cost in the buffer/resource management. The second mode has a nasty problem that the locks prevent the GC from running regularly, which can result in early OOMs if you are running highly concurrent software like Presto with a highly concurrent GC like G1. Another benefit of Java for compression algorithms is that the JVM can inline the compression code directly into the uses, which can result in pretty big speedups as the compression code is adapted to the actual inlined use case. You can't see this thin isolated benchmarks, and instead you have to test your actual uses. Finally, there are other benefits like debugging and profiling just working like normal Java code.
The original goal of this project was to create compatible compression algorithms that were on par with the performance of the JNI wrappers. What we found in the initial versions where that they were actually more performant, typically in the range listed in the readme. The performance changes over time as the JNI implementations change, and as the JVM changes. For example, you ran the benchmark on Java 8, but we run on Java 10. We also likely run on different CPUs, and most importantly, the performance of compression algorithms is totally dependent on the data you feed it (take a look at the high variance of the different benchmark corpuses).
If none of this appeals to you, or you don't have the same concurrency issues or portability/ergonomics concerns, use the JNI implementations. They are generally excellent.
from aircompressor.
Related Issues (20)
- Snappy Decompress throw MalformedInputException HOT 2
- Zstd Decompress does support files compressed by newer C version? HOT 6
- Implement other ZSTD compression levels
- Lz4 double buffering
- compile and generate class files for the given Java implementaion. HOT 1
- ZstdOutputStream writes empty stream HOT 2
- Support Zstd seekable format
- Hive fails opening split for zst compressed files
- ZStd JNI vs Aircompressor pure java performance question HOT 3
- The result is different HOT 1
- Add `SECURITY.md` file and enable vulnerability reporting HOT 2
- Set up OSS-Fuzz
- Decompression: How to determine the size of the output buffer? HOT 1
- Remove dependency on sun.misc.Unsafe? HOT 1
- May I ask if Android is usable? HOT 1
- Security vulnerability: Snappy decompressor can be made to crash JVM HOT 5
- Compression ratio is different in ZSTD algorithm between ZstdOutputStream and ZstdCompressor.compress(Bytebuffer) HOT 2
- Support lz4 framing decompression HOT 1
- ZSTD : Drive the end of decompression with both "inputLimit" and "outputLimit"
- Request: Generic Decompressor for all algos based on magic number header
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aircompressor.