Comments (14)
If you could email me the test file, that would be great, since this probably depends on data. I have tested this with a few files, including jvm-compressor-benchmark files, but obviously not yet with enough exemplars. :)
from compress.
Ok I see the files, thanks!
from compress.
Quick question: I can't actually test the issue with the code and files -- but I was wondering if this is sporadic, as there's number of repetitions? If so, how often do you see failure?
from compress.
Btw, it is also possible to pass UnsafeChunkDecoder as the ChunkDecoder to LZFInputStream, to force use of particular one. But even with that tests pass for me on Mac... maybe I need to find a Ubuntu box.
from compress.
I pushed in elasticsearch an upgrade to 0.9 as well, and a user reported JVM crashing when running elasticsearch tests (I did not see it...). Here is more info: elastic/elasticsearch@4bbf298#commitcomment-714854.
from compress.
I think the issue is caused by a buffer overflow in the copyLong method. When I put the following assert here 3 tests start to fail.
assert(destIndex + 8 <= dest.length);
from compress.
That sounds plausible: there are checks in 'copyUpTo32' to avoid this problem, but not in 'copyLong'. I wish I could easily reproduce this on my system, but since I can add asserts perhaps I can figure out an efficient way to avoid this corner case.
It is interesting that this does not seem to affect all platforms; issues with 'copyUpTo32' were very obvious. Perhaps it also depends on memory allocation details.
Thank you for pointing this out -- it gives one lead, should be able to figure out possible patch.
from compress.
I think you should be able to reproduce this on your system. Just add the assert and run ning/compress tests. testIncrementalWithFullReads(com.ning.compress.lzf.TestLZFInputStream), testIncrementalWithMinimalReads(com.ning.compress.lzf.TestLZFInputStream) and testUnsafeCodec(com.ning.compress.lzf.TestLZFRoundTrip) fail every time I run tests on Snow Leopard (64 bit java 1.6.0_29) and Ubuntu (32 bit java 1.6.0_23).
from compress.
Ok: I added boundary check which should prevent write past end of allowed buffer. I would greatly appreciate if anyone with "succesfully crashing" platform could see if this takes care of the issue.
from compress.
@imotov: yes, I was able to verify that. What I mean that I can verify that this aspect gets properly handled, and hopefully that is enough. I just can not yet verify that this is the only issue, i.e. before declaring this solved I hope it can be independently verified.
from compress.
It fixed the issue for elasticsearch. Thanks!
from compress.
Ok good! I will do 0.9.1 build then. I hope Wolfgang can also verify issue on his end, I hope this can be closed.
from compress.
I pulled 0.9.1 and all tests pass. Congrats! Thanks for everyone for fixing this so fast.
from compress.
Great! Big thank you to everyone who helped.
from compress.
Related Issues (20)
- Implement skip() efficiently, without needing to decode if possible HOT 1
- Deserialize directly into a ByteBuffer HOT 6
- Add a method to encode directly into given output buffer (of guaranteed size) HOT 1
- Implement encoder (compressor) that makes use of sun.misc.Unsafe HOT 1
- Unsafe-based decompressor of 0.9.7 fails on 2 sample files from 'maxcomp' data set HOT 1
- Improve 'DataHandler` callback to allow early termination HOT 1
- Expose number of bytes read from `InputStream`, via `LZFInputStream` HOT 1
- Unsafe clean up of Thread Local Value
- Add new variants for "compress only if comp rate at least N" HOT 1
- Incorrect de-serialization leading to stream corruption in Big Endian systems HOT 9
- Add convenience method(s) for GZIP read/write
- Document parallel compression task
- OptimizedGZIPInputStream fails on chunked stream HOT 2
- estimateMaxWorkspaceSize() is too small HOT 9
- did not start with 'ZV' signature bytes HOT 5
- API changes report for Compress LZF
- Fix issues outlined by "lgtm.com"'s static analysis HOT 1
- Add Java 9+ module info using Moditect HOT 1
- `Unsafe` needs support in `module-info.java` HOT 1
- Mistaken Code in k8s HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from compress.