Node stream support? about pako HOT 16 CLOSED

pako does NOT support node streams, and there are no practical reasons to add such support. Because that will work only with node, and node itself is ok with native zlib. Also, this lib is not recommended for node (at least in master thread), because it locks event loop.

If you have practical cases from real life - let me know.

from pako.

Probably you can try inflator.push([], true); or inflator.push(new Uint8Array([]), true); Don't know if it work.

from pako.

I did try inflator.push([], true) and it resulted in a -3 error code. Not sure what the problem was there.

My real world example was actually in the browser. I wanted to inflate a file that was gzipped, and then do some transformations on it line by line (many of which would be discarded because of dirty data), without buffering the entire thing in memory before inflating. Unless zlib needs the whole thing in memory to inflate? I was under the impression that it didn't.

My example above is basically what I was trying to do.

from pako.

Ok. I understand concept, but not sure that it's optimal. You will have troubles with this method anyway. Because if your data is text, you will need to convert encoding into utf8. That will cause problems, if chunk border will cut multibyte char. At least, you will need additional transformer, that will do reencoding with care about each 2 neibor chunks.

The most easy way is to put chunks into Blob object for big data, until compleete and process all at once. That will not eat browser heap. Using streams will work, but will require more efforts & more code.

Let me know what you think about.

from pako.

Use pako from master until next release happens. It already works with dummy push 4c37612

from pako.

Awesome work on the dummy push!

I hadn't thought about multibyte chars, that's definitely a problem I'm not sure how to handle perfectly. Although a transform stream in the middle should be able to check for escape chars near the end of the string and to know to split the chunk. But yeah, that's messy.

from pako.

#11 now collects use cases, to understand if strings/utf8 should be supported directly in pako or not.

I can suggest this way to reencode string blocks < 100k:

function encode_utf8(s) {
  return unescape(encodeURIComponent(s));
}

function decode_utf8(s) {
  return decodeURIComponent(escape(s));
}

Then write intermediate transcoder stream, that will check last 3 bytes of incoming chunk, fix size and remember tail until next chunk come.

In FF you can also use TextEncode/TextDecode.

Also you can use FileReader.readAsText() - it's async, but should be ok for your case.

Other utf8 info can be found @ stackovervlow, also some modules available in npm. Note, that npm modules use buffers, that takes a lot of useles/slow code for browserification. It's better to write your own transformer.

from pako.

Just curious, but where did you pull the 100k value from?

Now that browserify uses Uint8Arrays as Buffers (https://npmjs.org/package/native-buffer-browserify), perhaps it wouldn't be that slow to use https://www.npmjs.org/package/utf8-stream in the browser (and thus buffers).

from pako.

http://lipsum.com/

from pako.

I don't understand.

from pako.

You asked about source of lorem fixtures?

from pako.

I was wondering how you deduced that 100k was some sort of limit for string encoding. Is it a limit you have discovered, or are you just using it as an arbitrary value?

from pako.

You will need to do String.prototype.fromCharCode(null, Array) there to quickly convert Array -> String, and it will crash with stack overflow for big values.

If you don't care about performance too much - then there are no needs at all for such hacks with blocks operations. Just use any lib that scan chars one-by-one and be happy.

I'd recommend to study sources of JSzip (this one and others) + stackoverflow. Utf-8 reencoding is already well covered with all possible ways, and i have nothing genious to add about. Just do it :)

from pako.

Ahh, you mean the max arguments limit / overflow condition when calling a function, especially when using .apply.

from pako.

If you want Node's API (including streaming) on top of pako, I made a module for that https://github.com/devongovett/browserify-zlib

from pako.

@devongovett Nice work! And wow it's already in browserify! Awesome.

from pako.