Add performance tests for scenarios we care about about machinelearning HOT 5 CLOSED

1. We need to have a set of tests that contributors can run to verify that their PRs have desired impact on perf, i.e. they would run the tests on master, then on the branch with changes, and compare the results.
2. This project is not to do comparative benchmarking between different ML frameworks.

from machinelearning.

Sounds good, then we should probably rename the project/file to PerfRegressionTest.
Benchmarks are usually competitive.
Hopefully, in this case the benchmarks can be normalized against hardware, so that if the run is on a different hw platform it will not fail. Alternatively it should track the perf and the hw platform it runs against.

from machinelearning.

Hopefully, in this case the benchmarks can be normalized against hardware, so that if the run is on a different hw platform it will not fail. Alternatively it should track the perf and the hw platform it runs against.

The way we use such benchmarks is the following:

1. Run the benchmark on master branch.
2. Switch to PR branch
3. Run the tests again.
4. Post the diff in the PR description.

This "isolates" the hardware.
Having said that, the reports from BDN record information about the hardware, runtime, etc.

from machinelearning.

PR #62 merged this.

from machinelearning.

The PR just merged an infrastructure project and a "demo" test. I think it would be good to add more real tests, e.g. flight delay prediction. I would be more than happy to add it, if we think this is the right representative test and we add the data set to the repo.

from machinelearning.