Git Product home page Git Product logo

Comments (11)

puzrin avatar puzrin commented on August 20, 2024

https://github.com/nodeca/pako#example--api second example is for objects and binstrings.

from pako.

ociebieda avatar ociebieda commented on August 20, 2024

Thank you for your reply.
I already read this official doc, similar stackoverflow questions and googled related problems.

pako.deflate(data, {level:"9"})
Creates 58 kB data.

pako.deflate(data, {level:"9", to:"string"})
Creates almost twice larger data, 104 kB to store.

Due to such a huge difference in compressed data size, option "to string" is not a solution in my case.

I tried many functions to convert standard pako deflate output to JavaScript string, but all of them returns different data from original string.

from pako.

puzrin avatar puzrin commented on August 20, 2024

pako core works with byte arrays. uit8 encoder/decoder is just a wrapper for convenience. You can use any other preparation methods, optimal for you task. Just send typed (if possible) array to pako and receive new array back. Everything else is on your choice and out of pako goals.

from pako.

ociebieda avatar ociebieda commented on August 20, 2024

I tested many packers. I choosed Pako because it is incredibly fast with large data (counted in MB). Almost every packer takes it too long on larger data, or just freezes/crashes browser. Thats why I choose Pako.
Its awesome work. I will try to find the way to solve my problem. If original data was correct JavaScript binary string and compression and decompression was correct for making binary files, then there must be the way to convert it back to binary string. I will keep on research and thanks for such a great tool.

from pako.

puzrin avatar puzrin commented on August 20, 2024

Note, that term binary string means that every character has code 0..255. If you are sure, your data is really binary string, pako will process it as array of bytes. Also note:

  • deflate
    • input - array, binary string, JS string (utf16, autoconverted to utf8 first) - autodetected
    • output { to: 'string' } - binary string
  • inflate
    • input - array, binary string
    • output { to: 'string' } - expects that unpacked data is utf8 bytes stream and convert it to utf16 (js string)

That's done for the most used cases. But as i said, encoder is optional. If you use arrays on input and output, you can be sure, that pako works exactly as described in inflate/deflate specs, and result is binary equal to zlib.

from pako.

puzrin avatar puzrin commented on August 20, 2024

I can not guarantee, that built-in recoder is optimal for all cases. But is solves another task. Very often beginners ask "how should i send my object to server", they don't understand that JS strings are utf16, and server works with utf8. So, with built-in methods they can use json stringify/parse without big ass pain, and without writing special recoder on server side.

from pako.

ociebieda avatar ociebieda commented on August 20, 2024

Thank you for your reply.
After 3 days of fighting and testing anything, it seems that none of JavaScript expert is able to do it.
I created minimal code with problem as possible.
Question is how to convert back x3 to x2 or x1?

x1=getFile('filename');
// raw 16 bit binary string. not 0-255 but 0-65535 (not text)
x2=x1.split('').map(function(x){return x.charCodeAt(0);});
// changing characters to array with char codes
// first 4 elements: 63367 63457 95 22
x3=new Uint8Array(x2);
// converting char codes array to 8 bit array
// first 4 elements: 135 225 95 22
// then deflate with pako
// and then inflate with pako

from pako.

alippai avatar alippai commented on August 20, 2024

What is x2 for? You should use getFile() as binary (Uint8Array) instead of string.
So start with x3=getFile('filename') // this should be a raw Uint8Array
Check http://www.html5rocks.com/en/tutorials/webgl/typed_arrays/#toc-fileapis for more information

from pako.

ociebieda avatar ociebieda commented on August 20, 2024

Because I need to work with 16 bit strings (which are not just text, but raw binary data taken with XMLHttpRequest).
It is later used with few other JS apps which work with 16 bit strings only (as input), not arrays.
To save size of loading data (which for example can be compressed from 4 MB to 200 kB), I use pako (because it has the best performance - other packers freeze browser on large data).
So, I convert these strings to arrays to not lost special characters while compressing and decompressing with pako, and to get best possible compression (I tested it out, and "to string" produces much larger files, so 8 bit array is best).
Later, in other script, I decompress 8 bit array back with paco and I must send this as 16 bit string to few other JS apps (as input) that do not support arrays.
Thats all the story.

from pako.

creationix avatar creationix commented on August 20, 2024

@ociebieda so x2 is a JS array containing integers in the 16 bit range, but then you store it a Uint8Array. That will truncate all your higher numbers. Yes this will give you very small compressed results, but you're only storing half your data! It's impossible to inflate back to the original 16-bit ranges if you only ever give pako half the data in the first place.

What you should do, assuming your string really contains char codes in the 16-bit range, is store the bytes in an Uint16Array and then create a Uint8Array from the buffer of that, not the array.

// string contains raw 16-bit binary data encoded as 16-bit char codes, not unicode data.
function string16ToArray8(string16) {
  var length = string16.length;
  var array16 = new Uint16Array(length);
  for (var i = 0; i < length; ++i) {
    array16[i] = string16.charCodeAt(i);
  }
  // Reuse the memory buffer, but as an array of 8-bit values that's twice as long.
  return new Uint8Array(array16.buffer);
}

Then to do the reverse:

function array8ToString16(array8) {
  var array16 = new Uint16Array(array8.buffer);
  var string16 = "";
  var length = array16.length;
  for (var i = 0; i < length; ++i) {
    string16 += String.fromCharCode(array16[i]);
  }
  return string16;
}

from pako.

puzrin avatar puzrin commented on August 20, 2024

Closed - not a bug

from pako.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.