Benchmark speed reference values and division with CPU_MHZ about embench-iot HOT 3 OPEN

@TilakGirijeswar

Thanks for reporting this. Clearly a deficiency in the documentation. The benchmark_speed.py script reports the relative score without dividing by CPU_MHZ. You should do this yourself to get a per MHz score. Both are useful. One is a measure of the raw performance of a processor, the other the efficiency of the implementation. You are correct that the baseline processor (Arm Cortex M4) was run at 16MHz.

I've assigned to @PaoloS02 to investigate and make appropriate changes.

from embench-iot.

@jeremybennett ,

Thanks for looking into this and making a note on improving the document.

Meanwhile, My concern was also on the process.

I agree that raw performance is helpful, but this method will impact relative score.
When I am running an ARM core at 48MHz (CPU_MHZ=48) more iterations are executed leading to more time consumption. There is no common base for comparison when using relative speed data as reference which uses 16MHz and obtained results are at 48MHz

For example,
ATSAML21 (Cortex M0+) running at 48 MHz takes 10027ms for aha-mont64 program.
On calculating relative speed:

Reference time by STM32 on aha-mont64 @ 16MHz / Obtained time by SAM on aha-mont64 @ 48MHz
4004/10027
0.399

You can see that this score is too low.

But if you normalize frequency(by including divide by CPU_MHZ in script), change in calculation will be:

(Reference time by STM32 on aha-mont64/16) / (Obtained time by SAM on aha-mont64/48)
(4004/16)/(10027/48)
250.25‬/208.89
1.197

This comparison seems to be reasonable.

from embench-iot.

Hi @TilakGirijeswar

The benchmarks are scaled according to the speed of the machine. The intention is that each should run for about 4 seconds - a period long enough to be measured accurately, while meaning all the tests can be run in a few minutes.

This is necessary, because there is at least 3 orders of magnitude in variation in speed between the smallest and largest microcontrollers (think entry level Atmel AVR ATTiny and top end Cortex M4/M7).

So we end up with a lot of multiplying and dividing by CPU_MHZ. Your ARM core should execute 3 times more of each benchmark compared to the baseline, but that is then factored out, so your Embench score should be 3x higher. When we divide by MHz, you should be almost identical.

I think this serves to underline why we need to improve the documentation!

from embench-iot.