Support params in :options map about jmh-clojure HOT 3 CLOSED

I see how this could be useful.

The problem with using parameter sequences for this is that they are a JMH construct used for "walking the configuration space for a benchmark". Unfortunately, this "configuration space" does not include static annotations like @OperationsPerInvocation.

If I understand your use case correctly, for something like the below:

(def bench-env
  {:benchmarks [{:fn 'my.bench/do-work-n-times
                 :args [:param/times-to-run]
                 :options {:ops-per-invocation :param/times-to-run}}]
   :params {:times-to-run 42}})

it is clear how to compile the benchmark. We just replace the :param/times-to-run in the :options with 42 and use that value for the @OperationsPerInvocation annotation value. However, the value 42 is actually a shortcut for specifying the single-element parameter sequence [42].

For example, which value would I compile into our benchmark class if we ran like so?:

(require '[jmh.core :as jmh])
(jmh/run bench-env {:params {:times-to-run [1 10 100]}})

As an alternative, you could do something like the following:

(def run-count 42)

(def bench-env
  {:benchmarks [{:fn 'my.bench/do-work-n-times
                 :args [:param/times-to-run]
                 :options {:ops-per-invocation run-count}}]
   :params {:times-to-run run-count}})

You could also specify the option globally via the task when running with lein-jmh, optionally filtering the benchmarks to run with :select. Something like:

$ for COUNT in 1 10 100; do
    lein jmh "{:ops-per-invocation $COUNT, :params {:times-to-run $COUNT}, :select :list}"
  done

from jmh-clojure.

Sorry I didn't get back to you sooner.

The last example you provided for the shell script suggests compiling the same test multiple times, and providing a single value for the parameter to each? That's what I would expect just based on the data that's involved. I can understand that's a bit sucky from a code perspective, so maybe restricting it to params with only a single value could work? Might surprise people, not really sure though.

If you can give me a pointer on where in the codebase to look, I'm happy to try and put together a PR for this if you want?

from jmh-clojure.

No problem. Thanks for your follow up.

Yes, each lein invocation in the second example compiles the benchmarks. However, each time benchmarks are run, whether via Leiningen, or in a REPL, they must be compiled fresh - even if just a single annotation value has changed. JMH subprocesses require actual .class files on disk.

As an short aside, if being able to link :ops-per-invocation with a loops is the only reason you need this feature, I would recommend checking out this file from the JMH samples, if you haven't already seen it. Just in case you are coming from another benchmarking framework that encourages loops and are unfamiliar with JMH methodology. Please disregard this paragraph if this is not the case.

Anyway, if parameters are also available for use in :options maps, can they be used anywhere, or only at specific locations? What about as elements in nested data? For example, is this allowed?:

{:fork {:jvm {:prepend-args
              ["-cp" :param/classpath-entries "-Dsome.prop=true"]}}}

I guess option parameters would also need to be expanded recursively. Meaning :count below would be 4 and not the invalid keyword value.

{:options {:jmh/default {:fork :param/fork-options}}
 :params {:fork-options {:count :param/fork-count, :warmups 0}
          :fork-count 4}}

Additionally, we would have to throw an exception if a user gave more than one value for a parameter sequence that was used as an option. As you mentioned, this would likely surprise users who don't understand some of the inner-workings of JMH benchmarks. There is also a point of confusion with option values that are themselves sequences. For example:

{:options {:jmh/default {:timeout :param/my-timeout}}
 :params {:my-timeout [500 :ms]}}

Above, the :timeout parameter value would be correct if it was used as a normal option value, but here it is invalid. It needs to be wrapped in an additional single-element sequence. Granted, we could make special conditions for certain options to address this. Although, I don't particularly want to write code like "if :option-a ... else if :option-b ... else", etc.

As it stands now, I think that it might be better to stick with what we have, or implement some other mechanism rather than overloading parameters to handle this functionality as well.

However, maybe I'm not understanding the benefits this feature provides over the alternatives presented. As far as I can tell, the only downside to them is having to specify a value in two places. Am I missing something? Would it be possible for you to explain how they don't suit your use case, or are otherwise infeasible?

Regardless of any short-term decisions, we can always leave this issue open for the time being, so more users may voice their opinions.

from jmh-clojure.