Report a quality score for each measurement about caliper HOT 15 CLOSED

+1

Also, I don't know what's the science behind the cutoffs chosen, but I just 
changed 
them from 0.9-1.1, to 0.5-2.0 -- maybe I went too lax on it, I dunno.

It's a sanity check. I think the best fix is to write a new measurer. Perhaps 
we abandon 
the "always run for 5 seconds" default, and instead watch the numbers and wait 
for 
them to converge. If they don't converge for a long time, then fail.

Here are my latest thoughts:

We don't need to think of this as a sanity check that fails your whole run; 
instead, 
we could calculate and report a "quality score" for each measurement, and let 
the 
user decide.

As we run post-warmup trials, instead of keeping the rep count fixed 
(post-warmup), 
we can let it vary somewhat within a range. The "score" I'm referring to 
evaluates 
how linear these (time, reps) pairs appear to be. (Optionally, if they're 
linear but 
with a positive y-intercept, we could consider that value to be the per-trial 
overhead and correct for it in the values reported?)

Now, an intelligent measurer can be configured with not only a maximum run 
length in 
seconds, but also an "early termination" score that, if achieved, will end the 
run 
early to save time.

And, in a multiple-measurements-per-scenario situation, we no longer have to 
just 
blindly average (or median) the individual measurements; we can look at their 
quality 
scores and discard some of them, or weight them lower.  Of course, I think that 
most 
often if one measurement has low quality, the rest probably do too, because 
it's 
likely an inherent problem in the way the benchmark is constructed.

Incidentally if it's a benchmark that allocates any memory, then it's unlikely 
that every timing interval will get 
interrupted by GC the same number of times with the same workload to do, so 
variation is inevitable.

Still think that varying the rep count and evaluating the linearity of results 
would 
be a great thing to do.

With r132 I've added a basic sanity check to give a friendly exception when the 
benchmark body is optimized away.

The only thing we're missing is checks for enh's reported error case: when the 
benchmark is close to, but not linear. For example, if the number of reps 
impacts the runtime. I'm unsure what the fix here will be.

It would be nice to simply report a 'quality score' for each measurement 
somewhere.  Then just let users decide what to do.

So, if the results are absolutely linear, the quality score is 1.  It's 
possible all we need here is the statistical correlation coefficient between 
the rep counts and elapsed time measurements.

Earlier I said "It's possible all we need here is the statistical correlation 
coefficient between the rep counts and elapsed time measurements."  This is 
probably not true.  Probably, we can vary the rep count just to check for 
concavity for sanity's sake, but the statistic the user will want to see really 
is the standard deviation.

In caliper 1.0 we gather and report all the relevant data (subject to the 
user's preferences in caliperrc).  So even if we don't show a stddev on the 
screen, anyone who cares can always compute it later.  Tagging post-1.0.

I no longer think we want/need to do this.