kmalakoff / readdirp-walk Goto Github PK

Not sure if this should be 1, Infinity, or based on performance tests.

Dependent on #1

Does this module scale up to large directories - (10 million files) - in the same way https://github.com/kemitchell/native-readdir.js. Or does it need to list out all the files first before it begins?

I have been using this package, it's been an awesome experience, as it uses a lot less memory than readdirp! Thanks for crafting it!

I have stumbled upon another difference between the two packages. readdirp-walk stops on any error -- in my case, the walker is trying to enter OS X's .Trashes directory, what gives me an EACCES. When using readdirp this error doesn't happen. Here's the log of .on('end', function() { console.log(arguments) }).

{ '0': 
   { [Error: EACCES: permission denied, scandir '/Volumes/Bad Disk/.Trashes']
     errno: -13,
     code: 'EACCES',
     syscall: 'scandir',
     path: '/Volumes/Bad Disk/.Trashes' } }

Take a look at it("TODO", function() {}) and fill them in.

Also, look for other cases to test

I am currently using readdirp to scan a very large file system and process each of the files found. I have an issue that eventually it just blows up my system as it has too many threads and buffers open.

I can across your modified library readdirp-walk and was hoping this might help me with this issue if I set the concurrency down to 1.
The problem is I can't work out how to install your library. I tried doing a Git Clone and then a node install --production but I still cant work out what I need to do to get your library to replace the existing readdirp library. Can you perhaps put some detailed instructions into the readme file?

many thanks
peter.

There should be a way to bound the maximum number of parallel inflight requests.

function eachArrays(limit) {
  this.limit = limit;
  this.inFlight = 0;
  this.parallelArrays = [];
}

eachArrays.prototype.each(array, fn, callback) { this.arrays.push({array, fn, callback}); }
eachArrays.prototype.end(callback) { /* all done */ }