Problems with ConnectionsPerformanceTest about htm.core HOT 7 CLOSED

A related problem is the time it takes for this test:
[ RUN ] HelloSPTPTest.performance
starting test. DIM_INPUT=10000, DIM=2048, CELLS=10
EPOCHS = 10000
starting: 10000 iterations.Epoch = 9999
SP= 137,562,639,926,1133,1295,1335,1340,1346,1595
TP= 137,562,639,926,1133,1295,1335,1340,1346,1595
Total elapsed time = 20 seconds
[ OK ] HelloSPTPTest.performance (20699 ms)
[----------] 1 test from HelloSPTPTest (20699 ms total)

from htm.core.

Here are some more examples:
in Debug mode, running on my Ubuntu machine here. Very long times.
What we need is some way to shorten the time to run the tests and compensate for the speed of the target machine. Perhaps run some evaluation of processor speed to compare against. And adjust the amount of resources needed such that the test runs in a reasonable amount of time regardless of the machine on which it is ran. Or....just not run this as part of unit_test.

[----------] 4 tests from ConnectionsPerformanceTest
[ RUN ] ConnectionsPerformanceTest.testTM
0.004817 in temporal memory: initialize
21.867 in temporal memory: initialize + learn
26.3301 in temporal memory: initialize + learn + test
/home/dave/cpp/src/test/unit/algorithms/ConnectionsPerformanceTest.cpp:269: Failure
Expected: (tim) <= (3.5f), actual: 26.3301 vs 3.5
[ FAILED ] ConnectionsPerformanceTest.testTM (26362 ms)
[ RUN ] ConnectionsPerformanceTest.testTMLarge
0.039923 in temporal memory (large): initialize
178.824 in temporal memory (large): initialize + learn
213.026 in temporal memory (large): initialize + learn + test
/home/dave/cpp/src/test/unit/algorithms/ConnectionsPerformanceTest.cpp:277: Failure
Expected: (tim) <= (28.0f), actual: 213.026 vs 28
[ FAILED ] ConnectionsPerformanceTest.testTMLarge (213184 ms)
[ RUN ] ConnectionsPerformanceTest.testSP
1.22859 in spatial pooler: initialize
746.841 in spatial pooler: initialize + learn
750.429 in spatial pooler: initialize + learn + test
/home/dave/cpp/src/test/unit/algorithms/ConnectionsPerformanceTest.cpp:285: Failure
Expected: (tim) <= (10.0f), actual: 750.429 vs 10
[ FAILED ] ConnectionsPerformanceTest.testSP (750514 ms)
[ RUN ] ConnectionsPerformanceTest.testTP
11.4991 in temporal pooler: initialize

from htm.core.

The performance tests are taken from an executable that ran the tests optionally, but it didn't use to run as regular tests.
I thought to intorude it (with inflated enough) threshold to check we keep our performance. But as we see here, the performance probably varies too much from machine to machine (and load to load).
So I'll just diable the ASSERTs.

/home/dave/cpp/src/test/unit/algorithms/ConnectionsPerformanceTest.cpp:285: Failure
Expected: (tim) <= (10.0f), actual: 750.429 vs 10

Still interesting to see this big slowdown?! (I'm running on quite old notebook)

from htm.core.

@dkeeney does it fail in the CI, or your machine? Same with Release mode?

from htm.core.

from htm.core.

Perhaps some sort of simple benchmark to measure speed of machine (maybe 1 second on slow machine) then measure the speed of the algorithm in units of the benchmark speed.

This is a really nice idea!

If running in Debug mode, skip the test entirely because debug mode is always going to be slower.

I think our CI is running in Debug for reasons to make more checks, so this should run even in debug (the parametric slowdown above should handle it)

from htm.core.

from htm.core.