Please provide (minimal) client-side examples about pako HOT 27 CLOSED

@ixti I've already read the Examples and read the API you mention. Where do you think I've been those two hours before reopening? Clearly, they aren't helping me.

It's typical how certain developers often want to share their code, but not their knowledge by systematically refusing to symphatize with less seasoned developers who run into a (small) problem that can easily be avoided by a simple(r) example.

They rather see a thousand users waste half a day, than improve documentation with a simple usecase. Every developer knows other developers are clever enough to interpolate from there to the darkest depths of the API.

I'm guessing it has something to do with the fact that they once had to waste time themselves, and they don't want others to get off easy. Either that, or something about maintaining the status-quo as it's benefacting the size of their e-penis which is directly proportional to their technological superiority.

Sorry, I let myself go a bit. This is the difference between a friendly project (like socket.io and node.js and Linux Mint) and a hostile project (like Filezilla and Debian), and I just hate to see promising projects be hostile. I don't understand what drives people to go there.

Anyway, pako itself is very sweet and powerful, so let me give my fellow devvers some friendly pointers on it's usage, so they don't have to spend half a day:

Newbie-mistakes

1. pako needs an uInt8 array. They might seem the same as normal Array or ArrayBuffer from other frameworks or uInt16Array JavaScript uses internally or String[], because they are all the same numbers, but it's not the same.
2. Do not prepare the uInt8 array server side, because when sent over it's just an Array. Convert it back to uInt8 client-side.
3. de-flate sounds like de-crypt, but for decrypting/unzipping you actually need in-flate 😄

Example

As a friendly example for people who come across pako and - like me - aren't uptodate on the assumptions and limitations of Uint8Array (the docs and examples and API are useless in that regard - that's why I recommended something like this in the first place), let me underline that the uInt bytearray needs to be constructed from an array. Not a string, not a bytestring. Here's a little working example to get you started.

Consider your server sends you a bytearray for a gzipped string. Here's how to decompress:

// Get some gzipped binary data from the server in array form (string of numbers, representing characters, representing bytes)
var charData    = [31,139,8,0,0,0,0,0,0,3,5,193,219,13,0,16,16,4,192,86,214,151,102,52,33,110,35,66,108,226,60,218,55,147,164,238,24,173,19,143,241,18,85,27,58,203,57,46,29,25,198,34,163,193,247,106,179,134,15,50,167,173,148,48,0,0,0];

// Convert character data to binary data. The characters are byte-representations anyway so they are all 8 bit.
var binData     = new Uint8Array(charData);

// gunzip gzipped binary data
var data        = pako.inflate(binData);

// Convert gunzipped binary data to ascii string
var strData     = String.fromCharCode.apply(null, new Uint16Array(data));

console.log(strData);

Sending arrays from the server isn't optimal. Of course, Accept-Encoding may tackle this by gzipping the array, but gzipping an array representation of a gzipped string isn't the sweetest thing anyway. Better have the server send base64 encoded binary data.

// Get some base64 encoded binary data from the server.
var b64Data     = 'H4sIAAAAAAAAAwXB2w0AEBAEwFbWl2Y0IW4jQmziPNo3k6TuGK0Tj/ESVRs6yzkuHRnGIqPB92qzhg8yp62UMAAAAA==';

// Decode base64 (convert ascii to binary)
var strData     = atob(b64Data);

// Convert binary string to character-number array
var charData    = strData.split('').map(function(x){return x.charCodeAt(0);});

var binData     = new Uint8Array(charData);
var data        = pako.inflate(binData);
var strData     = String.fromCharCode.apply(null, new Uint16Array(data));

console.log(strData);

From here you can work with the API docs.

I hope this will bring some friendly and accessibility back to pako because it's obviously an awesome library. 😸

PS - Converting your data to an uint8 array in node using new Uint8Array(bson).buffer; before sending it over to the client, will freeze your Chromium browser tab when you execute pako.deflate(data);. Don't know what's going on, but it feels like something should be thrown in stead.

from pako.

@creationix thanks for the nice reply.

As I understand it, pako doesn't really want to be affiliated with the type of usage I employ it for, nor is the team particularly interested in a more basic/novice addition to its documentation.

I am inclined not to waste anyone else's time on this, including my own. 😄

from pako.

1. Giving advices is not contribution. It's a waste of time. Contrubution is making right pull requests.
2. Sending strings as splitted UTF-16 is less effective for english than UTF8
3. If you need to send big logs - just do proper gzip setup for json/text responses on your webserver, and send plain text. Browser will ungzip automatically. That does not require any coding.
4. Pako is about binary data processing, and not about string reencoding. There are tons of libs for encoding task.

You try to improve things without understanding pako goals and how it works. That's not as helpful as you think.

from pako.

1. Then why are you wasting people's time on Stack Overflow? :P
2. My bad. You can use String.fromCharCode.apply(null, new Uint8Array(data)); instead to reconstruct the string.
3. 4. Thanks for the advice. However, I got gzipped text. It's not mine. I don't control the server. Zlib is how I can gunzip that, and pako is a nice port.

I wasn't trying to improve pako itself, just the documentation. Looks like we got different views on what's helpful and what's not. Sorry for wasting your time.

Maybe this is a little bit helpful: If anyone else wastes your time like I did in the future, you can just point them to these examples:

Inflating gzipped string byteArray using pako:
http://jsfiddle.net/9yH7M/

Inflating base64 encoded gzipped string byteArray using pako:
http://jsfiddle.net/9yH7M/1/

from pako.

Just include into page one of files from dist. Then use window.pako.deflate/inflate. https://github.com/nodeca/pako/tree/master/test/browser

from pako.

Thanks. Can you briefly explain why I only need a file from dist/ even though the file seems to have a lot of references to ./lib/ and ./zlib/ even if they don't exist?

from pako.

Files in dist are autogenerated. See Makefile and read info about https://github.com/substack/node-browserify

from pako.

Thank you. 👍

from pako.

I cannot seem to recreate zlib.gunzip on my string buffer. I've been juggling with arrays and binaries and int8 and int16 and atob() and btoa(). All I get is a different string of jibberish or numbers.

I've got this binary or stringbuffer or array or base64 string or however I set my server to output my gzipped data, and I just can't figure out how to turn it into a readable text string. In node.js I can just use one command (gunzip) and I don't have to think about typecasting or whatever.

I highly recommend adding simple examples to your documentation if you appreciate a wider adoption of this zlib port, because it seems pretty cool (considering the list of features and speed). But after an hour, other (slower) libraries seem easier. And I'm not even that stupid.

Most databases store and retrieve binary data as either a buffer or a base64 encoded representation of binary data. For zlib's gunzip, we decode the base64 string to binary (e.g. using atob(base64String)) and we gan gunzip that back to text.

How do we do that in pako?

from pako.

@Redsandro I highly recommend you to read Example API in README

from pako.

Also, I highly recommend read Full API docs

from pako.

Keep in mind that the gzip format has more headers than just inflate and deflate. I could be wrong, but I don't see functions for adding or parsing gzip headers here. I just see deflate/inflate and raw-inflate/deflate. Compare with node.js which provides all three modes: http://nodejs.org/api/zlib.html

from pako.

@creationix all wrappers are supported. There is pako.gzip for gzipping. And pako.inflate autodetect deflate/gzip by header content. That's all exists in api docs.

from pako.

@creationix if you mean custom gzip headers content (not wrapper type), it's planned here #10 ... sometime...

PS. but node api also does not have functions to work with custom gzip headers content.

from pako.

As a Debian GNU/Linux user I refuse to agree with you on a statement that Debian is a hostile project. It's simply for those who don't want system to control it's users. That's why Ubuntu and Linux Mint are based on top of it. Because it's a low level if you want, providing sufficient minimum. Same goes for pako. It provides a sufficient minimum for developers. If you find pako interface way too low level, you can either:

• Provide a wraper library on top of pako that will provide a simple interface (although I really don't find pako any complex)
• Write a blog post for newbies and share your wisdom

from pako.

You don't get it. Pako is perfect. No problem! There's just a tiny gap between not knowing about the rules of binary juggling in javascript and being uptodate enough to get it working.

You can save 5000 manhours (1000 users wasting half a day) by just providing an easy example. Like I just did in my previous comment. After you made a witty response to my recommendation in stead of responding in kind. It's a proven concept to go from a working state to the API docs, rather than old-fashioned vice-versa. Like a lot of adoption-friendly "lowlevel" projects do; they often provide a minimalistic working example before any documentation is even provided.

I recommend putting (something like) my above shared wisdom (you're welcome - I know writing effective documentation can be hard for insiders) in the docs under client usage. Not for me, but for the benefit of others. After all, I already figured it out.

As for Debian, I rather not go offtopic, but yes, their developers are hostile in their communication, as opposed to the Mint developers. I am not talking about the project itself, but the community. I've seen developers literally being taunted away. I don't understand the need for hostile type communication.

Excuse my slight frustration, and apart from that, I'd like to say a genuine thank you to all the devs, pako is awesome. Thank you!

from pako.

Unfortunately, your example is invalid. Neither I nor @puzrin don't have a time right now to teach anybody about basics, as well as we don't have time to write valid examples on how to work with strings and pako. We are doing open source to solve our problems actually. So if that sounds hostile for you, you may boycott usage of pako. Otherwise, please please dive deeper into gzip/inflate before making any further suggestions.

from pako.

That doesn't sound hostile at all. 😄 But disregarding my efforts to help to make things clearer and just assuming I didn't read, or saying I should boycott pako in stead of saying thanks, kinda does.

If you're opensourcing for help, why disregard my contribution as "we don't have time for basics"? Isn't it beneficial to have a simple answer to those hundreds of questions out there how to gunzip gzipped strings client-side? After all, @Purzin himself advertises pako as the solution to some of those questions on Stack Overflow (that's how I got here), but neglected to provide proper documentation on how to actually do that client-side. I just did that for you.

I am now streaming gzipped server logs into my administrative panel using pako with my examples, so they are not invalid. Maybe gunzipping stringdata is not the intended use of pako, which would be weird since it's a full zlib port, but it works nicely nonetheless.

So one more time and then I am done trying to help improve the documentation and have a ready-made answer to the question:

<html>
<head>
  <title>Gunzipping (base64 encoded) binary gzipped string</title>
  <script type="text/javascript" src="pako.js"></script>
  <script type="text/javascript">
    // Get some base64 encoded binary data from the server. Imagine we got this:
    var b64Data     = 'H4sIAAAAAAAAAwXB2w0AEBAEwFbWl2Y0IW4jQmziPNo3k6TuGK0Tj/ESVRs6yzkuHRnGIqPB92qzhg8yp62UMAAAAA==';

    // Decode base64 (convert ascii to binary)
    var strData     = atob(b64Data);

    // Convert binary string to character-number array
    var charData    = strData.split('').map(function(x){return x.charCodeAt(0);});

    // Turn number array into byte-array
    var binData     = new Uint8Array(charData);

    // Pako magic
    var data        = pako.inflate(binData);

    // Convert gunzipped byteArray back to ascii string:
    var strData     = String.fromCharCode.apply(null, new Uint16Array(data));

    // Output to console
    console.log(strData);
  </script>
</head>
<body>
    Open up the developer console.
</body>
</html>

It works. 👍

(The fact that I gzipped the string poorly is irrelevant.)

from pako.

@Redsandro I think you're missing the sweet spot for pako. This is not meant to be a high-level library for the masses to use just as easily as they use jquery without even understanding JS or the native DOM. It's super awesome for people like me though.

I understand binary and string encoding in JavaScript very well. At first I was surprised that pako only accepts one format of binary data (Uint8Array) because most high-level libraries will accept several formats and try to auto-convert internally for you. But after thinking about it, I really like the way pako works.

I also know the deflate format fairly well. Writing a fast and correct deflate in pure JS (without resorting to C -> asm.js compilers) is a very hard task that is beyond even me. (And I can implement AES, MD5, and SHA1 from scratch all in a single day.)

Pako does the really hard parts for me and I don't want to distract the team from that very important work. They are doing something I can't do. (not well at least)

If you want to improve the docs, I doubt they would mind a pull request. I can review it for you since I'm quite comfortable working in the area that confused you. Between your outside view and my binary expertise we can make quite the nice pull request that is actually helpful to the pako developers who have different strengths than us.

from pako.

@creationix pako is ok untyped arrays too. Typed are just faster.

from pako.

Really? When I gave it a node Buffer (which acts like a normal array) it broke.

So it's supposed to accept any array-like and treat a byte array?

from pako.

Buffer !== Array. Pako expects Array or Uint8Array on input. And returns Array|Uint8Array on output, depending on support.

In theory, i can autodetect any data type on input (buffer, binary string, ucs2 [js] string), but can't detect required type on ouput. So, input autodetect will be only a half of solution. That's why #11 was created, to collect real use cases and change API in one step, if really required.

I'm not against API change, but against unstable API and against writing unused code "for the future".

from pako.

I'm not pushing for a change. I'm just a little surprised it supports Array, but not node's Buffer. Both support the .length property and square bracket access to byte values. Is there a technical reason for this or is it just type checking on input and Buffer isn't in the list?

from pako.

@creationix I just didn't checked buffers on input separately, because situation with output was unclear and i've frosen alternate types support until #11 collects more information from experienced developpers.

PS. just checked, pako consider node Buffer on input as Array, and works. But that's slower, than if you do new Uint8Array(buffer). Because will use per element copy to internal window, instead of block copy.

from pako.

Ok, that makes sense.

from pako.

how could you use pako to zip up multiple pdf files

from pako.

i think you need jszip instead - more high level lib

from pako.