ion-python-benchmark-cli's threshold value need to be more accurate. about ion-python HOT 2 OPEN

Some thoughts on this.

In the short term, we should increase the warm-up and iteration numbers to make sure the results become stable.

in the long term,

1. We should compare the new commit not only to the most recent one but also to the average value from the past months to prevent unnoticed false negatives from occurring.

2. We also need to reduce the noise caused by the machine. Currently, we only run performance regression detection on Ubuntu but we plan to add other OSs in the future. To ensure more consistent benchmark results on Linux, we can control factors such as CPU affinity - https://easyperf.net/blog/2019/08/02/Perf-measurement-environment-on-Linux

3. We should create more specific files that target specific data models. For example, if a new commit introduces changes that impact the speed of string serialization, it can be hard to find. By having files that include numerous test cases for Ion string values, we can magnify this regression and easily detect it.

4. We can also mimic JMH (Java Microbenchmark Harness) and implement another warm-up mode based on"clock time" instead of "execution numbers" which ensures that the benchmark-cli warm up for a specific duration. For example, instead of executing the specific benchmark code 20 times, we warm up the benchmark-cli for 20 minutes.

from ion-python.

Few TODOs:

• We can measure the percentile of the benchmark results - https://learn.microsoft.com/en-us/previous-versions/msp-n-p/bb924370(v=pandp.10)
• We can try running each command for both previous and current commits together and see if it has less variance (currently we are running all commands for each commit each time).

from ion-python.