TimeoutError / EHOSTUNREACH error when uploading multiple files simultaneously about ocwa HOT 2 CLOSED

The Validation API is running OOM when you upload this many files due to #64 having to load the entire file(s) into memory for each rule. So when you upload 5 files it loads 25 files into memory simultaneously.

from ocwa.

For MVP we have decided that upping the memory is sufficient. In a Future release, we will support a configuration parameter "workingLimit". All validation jobs that are in pending state are loaded into a queue. Anytime the workingLimit has not been hit the api can pull the next item in the queue if and only if it would not EXCEED the working limit as each job is 1 rule 1 file it's an easy check of the file CTL.

Note that if the CTL of a file is itself larger than the workingLimit it should fail it immediately (when it checks it for CTL) with message: "File too large to run validation on"

from ocwa.