mapbox / s3scan Goto Github PK

Sometimes it appears that the aws-sdk can return no error object, and also provide a data object where .Contents is undefined. In that case we would throw an unhandled error in the callback when we try and .slice the anticipated array.

s3.listObjects(params, function(err, data) {
  assert.notOk(err, 'there is no error');
  assert.notOk(data.Contents, 'there is also no information');
});

cc @yhahn

There are have been 2 commits to master since the last release -- I'd like to be able get them when I require s3scan: v0.2.0...master

cc @davidtheclark

The get stream is broken on [email protected].

/Users/r/gh/mapbox/s3scan/node_modules/aws-sdk/lib/request.js:30
            throw err;
                  ^
Error
    at check (/Users/r/gh/mapbox/s3scan/node_modules/queue-async/build/queue.js:56:36)
    at Object.q.defer (/Users/r/gh/mapbox/s3scan/node_modules/queue-async/build/queue.js:67:9)
    at Transform.getStream._transform (/Users/r/gh/mapbox/s3scan/lib/get.js:32:21)

It would be a nice thing to do for your bucket's well-being.

It would be cool to have a cancel method. Right now I'm using this to pipe an s3scan.Scan to another transform stream, and then when I've gotten as much as I need I end that stream, but trying to end the scan stream results in a write after end error because of the queue. Something like a purge or cancel option to delete all queued items and cancel the stream would be useful in cases like this.