Pako fails inflate on concatenated gzip files about pako HOT 7 CLOSED

In case anyone else needs to do this, I'm using this snippet to inflate concatenated gzip files using pako:

function inflateConcatenatedGzip(buffer) {
  var position = 0;
  var blocks = [];
  do {
    var inflator = new pako.Inflate();
    inflator.push(buffer.slice(position));
    if (inflator.err) { throw inflator.msg; }
    blocks.push(inflator.result);
    position += inflator.strm.total_in;
  } while (inflator.strm.avail_in > 0);
  return blocks;
}

from pako.

You try to use high-level api for things, that it's not designed for. You need to use files from ./lib/zlib directly and write your own wrapper.

from pako.

In short, zlib interface is very complicated for regular users. Pako API is not designed to return multiple files. It's designed to be easy and small. Explanation is the same as for #34. I'm not sure that it's technically possible to fit all zlib usage patterns into such simple api.

But you can alwayas use zlib api directly, and make wrapper with another logic. Or, if you know how to fit additional features in pako api - create a pull request.

from pako.

modified code from @danvk, to inflate concatenated gzip arrayBuffer in modern browser.

{
  const loadPako = () => window.pako || fetch(`https://unpkg.com/[email protected]/dist/pako.min.js`)
    .then((response) => response.text())
    .then((text) => eval(text))
    .then(() => window.pako)

  const concatArrayBuffer = (arrayBufferList = []) => {
    const resultTypedArray = new Uint8Array(arrayBufferList.reduce((o, arrayBuffer) => o + arrayBuffer.byteLength, 0))
    let byteOffset = 0
    arrayBufferList.forEach((arrayBuffer) => {
      const { byteLength } = arrayBuffer
      resultTypedArray.set(new Uint8Array(arrayBuffer), byteOffset)
      byteOffset += byteLength
    })
    return resultTypedArray.buffer
  }

  const gunzipAll = (PAKO, arrayBuffer) => {
    const arrayBufferList = []
    let byteOffset = 0
    let byteLeft = 1
    while (byteLeft > 0) {
      const inflator = new PAKO.Inflate()
      inflator.push(new Uint8Array(arrayBuffer, byteOffset))
      if (inflator.err) throw inflator.msg
      arrayBufferList.push(inflator.result)
      byteOffset += inflator.strm.total_in
      byteLeft = inflator.strm.avail_in
    }
    return concatArrayBuffer(arrayBufferList)
  }
}

from pako.

Seems like a regular thing for the interface to this library needing a wrapper of some sort, and everyone needing the same wrappers.

Does that project exist somewhere?

I can't seem to find it. But I have a lot of code that is derived from things people posted in github issues. And am learning more about zlib internals than would be ideal.

Right now looking for something that has done this with a gz file so big the content needs to be streamed. I can't find that yet.

from pako.

The above code worked for me in browser but not Node.JS. It seems that strm.avail_in never decreases in Node. With the added sanity check that position < buffer.byteLength the code completes with corrupted results. A comparison of browser and Node.JS execution shows different successive position values for the same file. However on Node.JS you can use stream.pipe(zlib.createGunzip()) (but not zlib.inflateSync).

from pako.

@wholenews if you're using code from #35 (comment), then my guess is you used Node.js Buffer.buffer directly as ArrayBuffer, the problem would be some Node.js Buffer is internally shared, and need code like toArrayBuffer to convert them correctly.

from pako.