ende76 / brotli-rs Goto Github PK
View Code? Open in Web Editor NEWA Brotli implementation in pure and safe Rust
License: Apache License 2.0
A Brotli implementation in pure and safe Rust
License: Apache License 2.0
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x12\x1b\x00\x1e\x11\x00\x05\x09\x21\x00\x05\x04\x43\x05\xf5\x21\x1e\x11\x00\x05\xf5\x21\x00\x05\x04\x43".to_vec() as &[u8]).read_to_end(&mut input);
}
coreyf@frewbook-pro /t/meow (master)> cargo run
Running `target/debug/meow`
thread '<main>' panicked at 'arithmetic operation overflowed', /Users/coreyf/.cargo/git/checkouts/brotli-rs-33811c30552ad7ba/master/src/transformation/mod.rs:145
Process didn't exit successfully: `target/debug/meow` (exit code: 101)
Crash discovered using afl.rs
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\xb1".to_vec() as &[u8]).read_to_end(&mut input);
}
coreyf@frewbook-pro /t/meow (master)> cargo run
Running `target/debug/meow`
thread '<main>' panicked at '"Encountered unexpected EOF"', /Users/coreyf/.cargo/git/checkouts/brotli-rs-33811c30552ad7ba/master/src/lib.rs:2280
Process didn't exit successfully: `target/debug/meow` (exit code: 101)
Panic discovered using afl.rs
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x30\x30\x40\x00\x00\x00\x00\x00".to_vec() as &[u8]).read_to_end(&mut input);
}
coreyf@frewbook-pro /t/meow (master)> cargo run
Running `target/debug/meow`
thread '<main>' panicked at 'index out of bounds: the len is 18 but the index is 18', ../src/libcollections/vec.rs:1110
Process didn't exit successfully: `target/debug/meow` (exit code: 101)
Crash discovered using afl.rs
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x15\x3f\x60\x00\x15\x3f\x60\x00\x27\xb0\xdb\xa8\x80\x25\x27\xb0\xdb\x40\x80\x12".to_vec() as &[u8]).read_to_end(&mut input);
}
coreyf@frewbook-pro /t/meow (master) [101]> cargo run
Running `target/debug/meow`
thread '<main>' panicked at 'shift operation overflowed', /Users/coreyf/.cargo/git/checkouts/brotli-rs-33811c30552ad7ba/master/src/lib.rs:1162
Process didn't exit successfully: `target/debug/meow` (exit code: 101)
Crash discovered using afl.rs
I tried this on the bro binary from google and it fails to decompress with brotli-rs when compressed using -q 4 -w 16
https://www.dropbox.com/s/il9uhagaaawxl6i/bro?dl=0
https://www.dropbox.com/s/2lb4dzpjczbv8ll/bro.br?dl=0
but this is true for almost every binary I checked...
in fact I just wrote a checker script that simply, in a loop, compresses with bro -q X -w Y
where x = {1..11} and y = {16..22}
That's how I found this glitch.
I've been timing the C implementation versus the rust implementation and I generally notice about a factor of 8-10x difference.
Do you know offhand any obvious performance tradeoffs that were made in the design of this version?
Do you have any ideas about various strategies we could employ to bring it within a factor of two, or ideally to the same speed as the C version especially for multi-megabyte files?
I noticed no inline assembly in the C version, so I'm hoping it is possible to bring the rust version to parity.
Have you done any profiling of the existing code or compared it with the C code?
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x51\xac\x00\x48\x2f\x73\x14\x01\x14\x00\x00\x01\x00\x14\x14\xff\x00\x02\x00\x00\x00\x00\x00\x64\x14\x24\x14\x14\x14\x14\x14\x80\x00\x00\x14\xff\xff\x00\x00\x14\x14\x14\x14\x14\x14\x80\x00\x80".to_vec() as &[u8]).read_to_end(&mut input);
}
Crash discovered using afl.rs
Hi--
I tried running the rust brotli decompressor--and if I use -q 2 on the encoding for big binary files, it fails to decode
here's an example
https://www.dropbox.com/s/nzvkt9pl94qb3b8/origin.br?dl=0
https://www.dropbox.com/s/m35k1udfe56b1wp/origin.pdf?dl=0
Any idea what the issue is? The output is only 128KB and doesn't match the original
extern crate brotli;
use std::io::{self, Read};
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x1b\x3f\xff\xff\xdb\x4f\xe2\x99\x80\x12".to_vec() as &[u8]).read_to_end(&mut input);
}
Crash discovered using afl.rs
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x5b\xff\x00\x01\x40\x0a\x00\xab\x16\x7b\xac\x14\x48\x4e\x73\xed\x01\x92\x03".to_vec() as &[u8]).read_to_end(&mut input);
}
Crash discovered using afl.rs
https://crates.io/crates/brotli
Servo has ownership of this name right now, but we can grant you access if you want. You just need to log in to (and by doing so 'creating') your crates.io account.
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x11\x3f\x00\x00\x24\xb0\xe2\x99\x80\x12".to_vec() as &[u8]).read_to_end(&mut input);
}
coreyf@frewbook-pro /t/meow (master) [101]> cargo run
Running `target/debug/meow`
thread '<main>' panicked at 'index out of bounds: the len is 255 but the index is 391', ../src/libcollections/vec.rs:1110
Process didn't exit successfully: `target/debug/meow` (exit code: 101)
Crash discovered using afl.rs
Compiling brotli v0.3.19 (file:///Users/coreyf/Development/rust/brotli-rs)
src/lib.rs:686:18: 686:33 error: const indexing is an unstable feature
src/lib.rs:686 sum += 32 >> code_lengths[i];
^~~~~~~~~~~~~~~
src/lib.rs:686:18: 686:33 help: in Nightly builds, add `#![feature(const_indexing)]` to the crate attributes to enable
src/lib.rs:1512:10: 1512:60 error: const indexing is an unstable feature
src/lib.rs:1512 1 << BROTLI_DICTIONARY_SIZE_BITS_BY_LENGTH[copy_length]
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/lib.rs:1512:10: 1512:60 help: in Nightly builds, add `#![feature(const_indexing)]` to the crate attributes to enable
src/lib.rs:1518:34: 1518:84 error: const indexing is an unstable feature
src/lib.rs:1518 let transform_id = word_id >> BROTLI_DICTIONARY_SIZE_BITS_BY_LENGTH[copy_length];
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/lib.rs:1518:34: 1518:84 help: in Nightly builds, add `#![feature(const_indexing)]` to the crate attributes to enable
error: aborting due to 3 previous errors
Could not compile `brotli`.
To learn more, run the command again with --verbose.
coreyf@frewbook-pro ~/D/r/brotli-rs (master) [101]> rustc -Vv
rustc 1.6.0-nightly (d49e36552 2015-12-05)
binary: rustc
commit-hash: d49e365528e026df6f56fe5eb001e81e2383fbf5
commit-date: 2015-12-05
host: x86_64-apple-darwin
release: 1.6.0-nightly
https://www.dropbox.com/s/ubfjbfe0oowfvtl/svg.zip?dl=0
when compressed to brotli is
https://www.dropbox.com/s/9ifkxzaxblfhz50/svg.zip.compressed?dl=0
and it fails to roundtrip with the following message
"data/svg.zip.compressed":
output length = 65703
res = Err(Error { repr: Custom(Custom { kind: InvalidData, error: StringError("More uncompressed bytes than expected in meta-block") }) })
===========
However the copy and insert lengths appear to be the same as with the google-provided decompressor
https://www.dropbox.com/s/gc6rjqtca4mi4cx/svg.zip.cerrlog.txt?dl=0
brotli's log:
https://www.dropbox.com/s/vptya6rxmks5f97/svg.zip.errlog.txt?dl=0
Here's the tail end of the log from the brotli-rs binary
Insert And Copy Length = 137
(m_len, self.meta_block.count_output, insert_length, copy_length) = (87897, 87883, 1, 3)
btype = 0
[p1, p2] = RingBuffer { buf: [0, 0], pos: 0, cap: 2 }
Context Mode = 3
Lit = 0 0
Count output 87884
Count output 87887
Insert And Copy Length = 182
(m_len, self.meta_block.count_output, insert_length, copy_length) = (87897, 87887, 6, 8)
Fatal (m_len, a, b, c) = (87897, 87887, 87893, 87901)
output length = 65703
res = Err(Error { repr: Custom(Custom { kind: InvalidData, error: StringError("More uncompressed bytes than expected in meta-block") }) })
===========
OK ()
Brotli expects the literal_buf to be prepopulated with uncompressed_data, if such data is encountered before something needing context bytes.
Currently the literal_buf is left to be 0,0 despite an uncompressed meta-block being encountered
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x1b\x3f\x00\xff\xff\xb0\xe2\x99\x80\x12".to_vec() as &[u8]).read_to_end(&mut input);
}
coreyf@frewbook-pro /t/meow (master)> cargo run
Running `target/debug/meow`
thread '<main>' panicked at 'arithmetic operation overflowed', /Users/coreyf/.cargo/git/checkouts/brotli-rs-33811c30552ad7ba/master/src/lib.rs:577
Process didn't exit successfully: `target/debug/meow` (exit code: 101)
Crash discovered using afl.rs
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x1b\x3f\x01\xf0\x24\xb0\xc2\xa4\x80\x54\xff\xd7\x24\xb0\x12".to_vec() as &[u8]).read_to_end(&mut input);
}
coreyf@frewbook-pro /t/meow (master)> cargo run
Compiling brotli v0.3.11 (https://github.com/ende76/brotli-rs#3540aa18)
Compiling meow v0.1.0 (file:///private/tmp/meow)
Running `target/debug/meow`
thread '<main>' panicked at 'internal error: entered unreachable code', /Users/coreyf/.cargo/git/checkouts/brotli-rs-33811c30552ad7ba/master/src/transformation/mod.rs:35
Process didn't exit successfully: `target/debug/meow` (exit code: 101)
Crash discovered using afl.rs
extern crate brotli;
use std::io::Read;
use brotli::Decompressor;
fn main() {
let mut input = vec![];
let _ = Decompressor::new(&b"\x1b\x30\x30\x30\x24\x30\xe2\xd9\x30\x30".to_vec() as &[u8]).read_to_end(&mut input);
}
coreyf@frewbook-pro /t/meow (master)> cargo run
Running `target/debug/meow`
thread '<main>' panicked at 'internal error: entered unreachable code', /Users/coreyf/.cargo/git/checkouts/brotli-rs-33811c30552ad7ba/master/src/lib.rs:1274
Process didn't exit successfully: `target/debug/meow` (exit code: 101)
Crash discovered using afl.rs
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.