dekkonot / bitbuffer Goto Github PK
View Code? Open in Web Editor NEWA binary stream module for packing binary data in pure Lua
Home Page: https://dekkonot.github.io/bitbuffer/
License: MIT License
A binary stream module for packing binary data in pure Lua
Home Page: https://dekkonot.github.io/bitbuffer/
License: MIT License
All Lua numbers are doubles, which are only accurate to 52 bits. Writing or reading an integer greater than that (either signed or unsigned) should error.
The source of the module looks like it's filled with magic numbers, and hex will make it look less like that since numbers like 0x7f
and 0xff
are obvious to people who know what they're doing
Modifications made to the Selene workflow only prevent it from triggering on non-init commits. This will still happen.
Solution is to change the glob pattern Selene uses to only include init.lua
Should include things like Int32 and UInt32. Should also shorthand some like writeByte -> writeUInt8.
Unclear how to handle assertions and testing. Seems like it may add a lot of overhead to run assertions twice in cases like writeFloat32
.
Given that writeSetLengthString doesn't throw when the string passed is 0 bytes, it doesn't make sense to have its read function throw.
Seems self-explanatory
At the moment, writeUnsigned calls writeByte several times followed by writeBits. It should instead call writeBits with however many bits is needed to make the buffer aligned, call writeByte with whatever is left, then call writeBits after that.
Benchmarks forthcoming, but I assume this will be substantially faster than the current implementation.
After rethinking it, I believe adler32 is too niche to keep in the BitBuffer -- it's easy to implement on its own and is only useful for zlib.
The module currently has several functions that operate on a bit level that utterly thrash the performance of the module, to the point that there's a recommendation to not use them and functions that avoid them entirely. We should remove these entirely in the future.
This is rather far reaching because it involves removing some rather fundamental parts of the module, but it's a small price to pay for performance and futureproofing.
Creating several medium sized strings and then using table.concat
on an array of them is faster than using table.concat on a bunch of small strings.
In some cases (such as writeFloat) the assertion messages use the format bitBuffer.X
and in others (such as readBit) they use X
(n both cases, X is the name of the function)
Ideally, the format would be bitBuffer.X
since it makes debugging easier.
It seems interesting that a data storage system for Roblox can't even store Roblox IDs which have been over 32-bits for over a year.
It would great to see a read/writeUInt64 function added to this library!
Thanks!
It would be beneficial to have the ability to output an X length chunk from the current pointer, as very large streams may cause problems if exported all at once.
Ideally in the form of dumpString(length: number?)
Early in development the plan was to have the buffer return false
instead of throwing when something went wrong. Artifacts of that exist throughout the base functions but the documentation says they all return nil. Should be corrected at some point.
Any calls to writeBit that we can avoid should be avoided -- writeSigned(32, 1) currently makes the internal call to writeUnsigned non-aligned, which is a performance concern at scale. The same concern applies to writeFloat.
There's an illusion that the methods are 'static' when using a period, which isn't ideal. Not sure if it's worth the effort of fixing it though.
Might be worth taking a look at other popular Lua libraries to figure out if it is.
This request comes from the fact that I can't seem to assign the value I get from buffer.dumpString()
into a StringValue object Value
property. I tried using a ModuleScript's Source
property to store the string but didn't have any luck with that neither.
My solution for this is to store the dumpBase64()
string, but it would be nicer to have a way to make BitBuffer read it back ๐
It could be done by changing the API to something like this:
BitBuffer(stream: str, options?: { format = "byte" | "base64" | "binary" | "hex" })
Let me know what you think about that!
In dumpBase64
for the roblox version, to create the table of output bytes the script will do table.create(math.ceil(byteCount * 0.333))
. This is incorrect, as Base64 does not make the file size 1/3. it should be * 1.333
.
The Roblox specific version of the module should:
table.create
where possibleAnd long-term should have support for the Roblox CLI when it comes out.
For things like LZ4 this is a must. Might be good to have it happen when the passed value to setPointer
is negative?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.