Comments (16)
pako does NOT support node streams, and there are no practical reasons to add such support. Because that will work only with node, and node itself is ok with native zlib. Also, this lib is not recommended for node (at least in master thread), because it locks event loop.
If you have practical cases from real life - let me know.
from pako.
Probably you can try inflator.push([], true);
or inflator.push(new Uint8Array([]), true);
Don't know if it work.
from pako.
I did try inflator.push([], true)
and it resulted in a -3 error code. Not sure what the problem was there.
My real world example was actually in the browser. I wanted to inflate a file that was gzipped, and then do some transformations on it line by line (many of which would be discarded because of dirty data), without buffering the entire thing in memory before inflating. Unless zlib needs the whole thing in memory to inflate? I was under the impression that it didn't.
My example above is basically what I was trying to do.
from pako.
Ok. I understand concept, but not sure that it's optimal. You will have troubles with this method anyway. Because if your data is text, you will need to convert encoding into utf8. That will cause problems, if chunk border will cut multibyte char. At least, you will need additional transformer, that will do reencoding with care about each 2 neibor chunks.
The most easy way is to put chunks into Blob object for big data, until compleete and process all at once. That will not eat browser heap. Using streams will work, but will require more efforts & more code.
Let me know what you think about.
from pako.
Use pako from master until next release happens. It already works with dummy push 4c37612
from pako.
Awesome work on the dummy push!
I hadn't thought about multibyte chars, that's definitely a problem I'm not sure how to handle perfectly. Although a transform stream in the middle should be able to check for escape chars near the end of the string and to know to split the chunk. But yeah, that's messy.
from pako.
#11 now collects use cases, to understand if strings/utf8 should be supported directly in pako or not.
I can suggest this way to reencode string blocks < 100k:
function encode_utf8(s) {
return unescape(encodeURIComponent(s));
}
function decode_utf8(s) {
return decodeURIComponent(escape(s));
}
Then write intermediate transcoder stream, that will check last 3 bytes of incoming chunk, fix size and remember tail until next chunk come.
In FF you can also use TextEncode/TextDecode.
Also you can use FileReader.readAsText() - it's async, but should be ok for your case.
Other utf8 info can be found @ stackovervlow, also some modules available in npm. Note, that npm modules use buffers, that takes a lot of useles/slow code for browserification. It's better to write your own transformer.
from pako.
Just curious, but where did you pull the 100k value from?
Now that browserify uses Uint8Arrays as Buffers (https://npmjs.org/package/native-buffer-browserify), perhaps it wouldn't be that slow to use https://www.npmjs.org/package/utf8-stream in the browser (and thus buffers).
from pako.
from pako.
I don't understand.
from pako.
You asked about source of lorem fixtures?
from pako.
I was wondering how you deduced that 100k was some sort of limit for string encoding. Is it a limit you have discovered, or are you just using it as an arbitrary value?
from pako.
You will need to do String.prototype.fromCharCode(null, Array)
there to quickly convert Array -> String, and it will crash with stack overflow for big values.
If you don't care about performance too much - then there are no needs at all for such hacks with blocks operations. Just use any lib that scan chars one-by-one and be happy.
I'd recommend to study sources of JSzip (this one and others) + stackoverflow. Utf-8 reencoding is already well covered with all possible ways, and i have nothing genious to add about. Just do it :)
from pako.
Ahh, you mean the max arguments limit / overflow condition when calling a function, especially when using .apply
.
from pako.
If you want Node's API (including streaming) on top of pako, I made a module for that https://github.com/devongovett/browserify-zlib
from pako.
@devongovett Nice work! And wow it's already in browserify! Awesome.
from pako.
Related Issues (20)
- "Error: -3" when inflating chunks of data HOT 2
- How to decompress blob? HOT 2
- Deflate() creating broken zip file
- Use classes to improve tree-shaking HOT 8
- No way to read Deflate data in chunks. HOT 4
- CI: set minimal permissions on GitHub Workflow HOT 2
- Cannot install pako HOT 1
- library initialization slowing down app start
- TypeError: input.subarray is not a function HOT 1
- Docs update: mention what `inflate` and `deflate` actually return
- Incorrect JSDoc type information for optional parameters
- Expecting a new version release HOT 1
- Different 10th byte gzip header between zlib and pako on Mac OS 13 HOT 1
- License declaration in package.json and LICENSE are not the same
- GZIP pako compressed string results in "Not in GZIP format" in JAVA on decompressing HOT 1
- Inconsistency and `undefined` return in `inflate` method between versions v1 and v2
- Using static dictionary, inflateRaw "Uncaught invalid stored block lengths"
- how to gzip multiple files? HOT 1
- Not compatible with Angular v17
- Getting corrupted data on decompressing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pako.