Git Product home page Git Product logo

Comments (5)

ZJONSSON avatar ZJONSSON commented on August 20, 2024

Can you see from the stack which file/line is throwing the cb is not a function error?

I think the lack of end might be a problem with using on('data') which is an event emitter and might not actually pull from the pullStream. So if you wanted to collect the data without an emitter you could do something like:

  .pipe(filterArray)
  .pipe(new Transform({
    objectMode: true,
    transform: (d,e,cb) => {
      text += d;
      cb();
     }
  })
  .on('finish', () => resolve(text));

Here are some other hints:

  • parseOne(regex) parses the first file that matches a regex.
  • .promise() returns a promise on all the data as an array of the packets unzipped.

So you could do this instead of using the SelectFile transform:

var streamPromiseArr = fileNames.map(async fileName => {
  var stream = await sftp.get(fileName, true, 'utf8', { highWaterMark: 256 * 1024 });
  var data =  await stream.pipe(unzipper.ParseOne('DailyPunch')).promise();
  // data = filterarray(data) ?
  return data.join('');
});

from node-unzipper.

JonnyBoy333 avatar JonnyBoy333 commented on August 20, 2024

Thanks for the feedback, that gives me some food for thought. In my case, there are actually multiple files that match 'DailyPunch' inside the zip folders, so I'm guessing that eliminates the ability to use unzipper.ParseOne()? Feature request for a ParseMultiple(regex)?

I'm still experimenting with the event triggers, I did try using 'end' instead of 'finish' but they both have the same result. There is something strange happening that I can't pinpoint when downloading large zips that get broken into multiple chunks. I'll keep investigating.

This is the stack trace.

TypeError: cb is not a function
    at afterWrite (_stream_writable.js:454:3)
    at onwrite (_stream_writable.js:445:7)
    at /Users/jonlamb/projects/node-test/node_modules/unzipper/lib/PullStream.js:59:60
    at afterWrite (_stream_writable.js:454:3)
    at _combinedTickCallback (internal/process/next_tick.js:144:20)
    at process._tickCallback (internal/process/next_tick.js:180:9)

from node-unzipper.

ZJONSSON avatar ZJONSSON commented on August 20, 2024

I have never experienced this before so I'm not sure exactly how this happens. But it seems that the _write method in Pullstream ends up being called without a callback (presumably by the pipe from the sftp).

I just published a version where we ignore the missing callback. Can you please check it out and see if you have any issues? You could also try to put a PassThrough stream between the sftp client and the unzipper and see if node internally does some magic to fix the piping. Also you need to ensure that the sftp stream actually ends when the file is done.

Unrelated: You can always do .buffer().then(...) to get the buffered output instead of your SelectedFile custom transform.

from node-unzipper.

ZJONSSON avatar ZJONSSON commented on August 20, 2024

@JonnyBoy333 did you check out the new version ?
npm install [email protected]

from node-unzipper.

ZJONSSON avatar ZJONSSON commented on August 20, 2024

Hey @JonnyBoy333 this has probably been fixed here
Latest version is: + [email protected]

Can you please verify and let me know? If this issue persist please reopen the ticket

from node-unzipper.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.