nickbabcock / bitter Goto Github PK
View Code? Open in Web Editor NEWExtract bits from a byte slice
Home Page: https://crates.io/crates/bitter
Extract bits from a byte slice
Home Page: https://crates.io/crates/bitter
Currently "unaligned" bytes are read with "read and refill", but I think we could do quite a bit better.
The idea is to take the buffer as is and bitshift the all bytes in one loop with SIMD.
The below produces nice auto-vectorized code that is >10x faster on bigger reads.
For example here we are "4 bits into a byte":
pub fn read_n_bytes(&mut self, n: usize, input: &[u8], out: &mut [u8]) {
let left_mask = 0b00001111;
let right_mask = 0b11110000;
for i in 0..n {
let left_part = input[i] & left_mask;
let right_part = input[i + 1] & right_mask;
out[i] = left_part | right_part;
}
}
I gave it a shot myself but couldn't quite get the state correct after this operation.
Bitter's 56 bit read limitation can be annoying to workaround. Imagine a use case of a length prefixed value of up to 64 bits is read.
let mut bits = LittleEndianReader::new(&[]);
let len = bits.read_bits(6).unwrap() as u32;
if len <= bitter::MAX_READ_BITS {
bits.read_bits(len).unwrap();
} else {
let lo = bits.read_bits(bitter::MAX_READ_BITS).unwrap();
let hi = bits.read_bits(len - bitter::MAX_READ_BITS).unwrap();
let result = (hi << bitter::MAX_READ_BITS) + lo;
}
Not only is it annoying, there could be faster ways to structure the reads (eg: consume the entire lookahead buffer, refill it, and continue reading). It makes sense for bitter to have this responsibility.
This is probably best suited for only the read_bits
function, and peek
and consume
can keep their limitation (but probably update their assertions to give better error messages).
EDIT: Shim zero bit reads too ๐ค
In working on some older code using your library, I accidentally called BigEndianReader::read_bits(0)
and got this panic:
thread '...' panicked at 'attempt to shift right with overflow', .../bitter-0.6.0/src/lib.rs:793:9
It wasn't immediately obvious what caused this without reading the source and checking the docs. Perhaps *EndianReader::peek
or read_bits
could use another debug_assert!
to make the error message clearer?
If you'd like, I can submit a PR.
Thank you for the crate! Supporting big-endian bitstreams would be a useful feature.
The number of bits to peek or consume is capped at the number of bits left in the lookahead buffer, but the number of bits left isn't exposed. The only workaround for the caller is to invoke refill_lookahead
themselves (perhaps unnecessarily) and do manual bookkeeping which duplicates bitter's internal bookkeeping.
Looking at a solution probably like:
fn lookahead_bits(&self) -> u32 {
}
I've not needed any new features or API changes in over a year (coming on two), if this continues for another 6 months, I will cut the current version (0.6.2) as (1.0).
There are nice to haves like the IO adapter, but since I don't personally need this feature I've not taken this feature to the finish line. I don't suspect it'd be a breaking change anyways.
Other tasks like exploring removing unsafe
(#23) and transitioning away from basing everything on a large macro won't cause any API or behavior changes and can be done after 1.0
Interested in using this crate but it is a bit unergonomic to having to resort to a different crate to write the bit patterns and then switch to this to read them
I am trying to migrate a library that uses bitter from 0.6.2 to 0.7.0.
Specifically there is this function:
pub fn refill(&mut self) {
let refilled = self.reader.refill_lookahead();
if refilled > 0 {
self.bits = self.reader.peek(refilled);
}
self.bits_left = refilled;
}
which now fails to compile with:
error[E0308]: mismatched types
--> src\first_pass\read_bits.rs:55:23
|
55 | if refilled > 0 {
| -------- ^ expected `()`, found integer
| |
| expected because this is `()`
error[E0308]: mismatched types
--> src\first_pass\read_bits.rs:56:42
|
56 | self.bits = self.reader.peek(refilled);
| ---- ^^^^^^^^ expected `u32`, found `()`
| |
| arguments to this method are incorrect
|
note: method defined here
--> C:\Users\Jan-Eric\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bitter-0.7.0\src\lib.rs:387:8
|
387 | fn peek(&self, count: u32) -> u64;
| ^^^^
error[E0308]: mismatched types
--> src\first_pass\read_bits.rs:58:26
|
58 | self.bits_left = refilled;
| -------------- ^^^^^^^^ expected `u32`, found `()`
| |
| expected due to the type of this binding
What is the intended way to migrate this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.