kesla / node-snappy-stream Goto Github PK

I need to save a JSON object to a Redis Database.

I am able to save, but I can't decrypt back from Redis.

It seems like I need to add a Stream Identifier to the compressed string to indicate it's a Snappy type.

`
/**
* All streams should start with the "Stream identifier", containing chunk
* type 0xff, a length field of 0x6, and 'sNaPpY' in ASCII.
*/
private static final byte[] STREAM_START = {
(byte) 0xff, 0x06, 0x00, 0x00, 0x73, 0x4e, 0x61, 0x50, 0x70, 0x59
};

`

This is the class the guys in Java are using..
https://programtalk.com/vs/?source=netty/codec/src/main/java/io/netty/handler/codec/compression/SnappyFrameEncoder.java

My question is, how can I generate a stream with those characteristics?

The rest of the payload it's a json object.

I.e.

const jsonObjectBody = '{ "myObjKey": "myObjValue" }';
const jsonObjectBodyStreamWithStreamIdentifier = ?????

Thank you very much!

I am facing issue while compressing a data stream for large amount of data. Buffer checksum size is out of range.

Node version: 10.16.0
Npm version: 6.9.0
OS: Ubuntu 18.04

Below is the error:
RangeError [ERR_OUT_OF_RANGE]: The value of "value" is out of range. It must be >= 0 and <= 4294967295. Received 4675058066 at checkInt (internal/buffer.js:35:11) at writeUInt32LE (internal/buffer.js:509:3) at Buffer.writeUInt32LE (internal/buffer.js:522:10) at module.exports (/home/dhvani/IoT-Sense-m_v3.6.3/gdmtrunk/node_modules/snappy-stream/lib/checksum.js:9:14) at CompressStream._compressed (/home/dhvani/IoT-Sense-m_v3.6.3/gdmtrunk/node_modules/snappy-stream/lib/compress-stream.js:33:9) at /home/dhvani/IoT-Sense-m_v3.6.3/gdmtrunk/node_modules/snappy-stream/lib/compress-stream.js:60:12

I am not able to install snappy-steam due to below error. I believe if we only need to update this package dependencies to use latest snappy version (if that doesn't break the functionality).
`

[email protected] install /Users/faraz.zahabian/dev/playground/node_modules/snappy-stream/node_modules/fast-crc32c/node_modules/sse4_crc32
node-gyp rebuild
CXX(target) Release/obj.target/crc32c_sse42/src/crc32c_sse42.o
LIBTOOL-STATIC Release/crc32c_sse42.a
CXX(target) Release/obj.target/sse4_crc32/src/crc32c.o
SOLINK_MODULE(target) Release/sse4_crc32.node
npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN cannot run in wd [email protected] node-gyp rebuild (wd=/Users/faraz.zahabian/dev/playground/node_modules/snappy-stream/node_modules/snappy)``
`

Hi..can you please update snappystream to latest version? I get the security warning in one of our Nodejs App audit process. The snappystream module has already updated dependency for minimatch to latest.

Thanks

Running the following code:

$cat SnappyStreamExample.js
var snappyStream = require('stream')
    , compressStream = snappyStream.createCompressStream()
    , uncompressStream = snappyStream.createUncompressStream({
        asBuffer: false // optional option, asBuffer = false means that the stream emits strings, default: true
    })

    compressStream.on('data', function (chunk) {
        console.log('Som data from the compressed stream', chunk)
        uncompressStream.write(chunk)
    })

    uncompressStream.on('data', function (chunk) {
        console.log('The data that was originally written')
        console.log(chunk)
    })

    compressStream.write('hello')
    compressStream.write('world')
    compressStream.end()

I got the following error on line 2:

$node SnappyStreamExample.js
    , compressStream = snappyStream.createCompressStream()
                                    ^
TypeError: undefined is not a function

Perhaps I'm doing something wrong, or there's a problem in the README code?

Thanks

How can I check if data is snappy compressed in the Stream format?

https://github.com/kesla/node-snappy provides isValidSnappyCompressed(input, callback).

Perhaps there's something similar for this library?

Thanks

A Lodestar user has reported issues building sse4_crc32 locally. Maybe a bump to the latest version of fast-crc32c 2.0.0 could fix the issue?

See ChainSafe/lodestar#2475 for more context

All the tests only consider single chunks. Uncompressing, however, fails when multiple chunks are piped through. This is because it expects the identifier to precede EVERY chunk, even though the specification only requires the very first chunk to be an identifier.

I'm adding a test case that makes this evident and will try to make a PR to fix it