When there is variable setup time about benchmark-ips HOT 12 CLOSED

x.benchmark(timer: true) do |timer|
	client = connect_to_server
	
	timer.capture do |repeats|
		index = 0
		while index++ < repeats
			client.get(...)
		end
	end
end

is basically

client = connect_to_server
x.benchmark(timer: true) do |timer|
	timer.capture do |repeats|
		index = 0
		while index++ < repeats
			client.get(...)
		end
	end
end

and while that is not capturing the setup (connect_to_server) it is basically what I tend to do.
And yes, it does not capture the amount of time for warmup.

But this is called "iterations per second" - so it does loose the nuance of warming up and getting up to speed.

from benchmark-ips.

@ioquatix I tend to setup the connection outside my benchmarks.
So I can get as little setup inside my loop.

Is it possible for you to do something like that?

from benchmark-ips.

@ioquatix yea. I've seen some nice benchmarks that show the before warmup numbers and the after warmup numbers. So you can get an idea if a cache is used, or connection pooling. Very nice stuff.

This does provide the warmup phase, but isn't setup to record/graph the warmup.

from benchmark-ips.

@ioquatix Maybe you misunderstood. The timer block just provides a reference to the capture method. Moving a method into that block or keeping it out is fine / does the same thing. The code in that block are there to setup the benchmark. Any code you put in there will be run before the benchmarks actually begin.

from benchmark-ips.

I get it. It’s just a shared name space for all benchmarks so if I have two very similar things but different implementations I have to call them e.g. client_implementation_x and client_implementation_y so they don’t clash and it just gets a bit messy.

from benchmark-ips.

Another interesting idea would be to run the benchmark with times=0 to compute the overhead and subtract that from subsequent test runs.

from benchmark-ips.

@kbrock thanks for your reply.

This was my ultimate conclusion - trying to do setup work throws the system off too much. I think we can close this issue.

That being said, for further discussion:

Setup is still part of the overall cost.

It also makes the comparisons less isolated because setup is done outside of the benchmark. In some ways, it makes sense if you just want to compare the operational overhead, but if you want to compare the full cost of two different implementations, it's tricky.

I did try using plain benchmark with fixed repetitions, and I think maybe for my use case, that makes more sense, w.r.t. testing setup as well as overhead. But it lacks the instructions/second which is pretty useful, and the formatting is less pretty, warmup phase, etc

from benchmark-ips.

One way you could do it is inject into the benchmark some "timer" object, e.g.

x.benchmark do |repeats, measure|
    client = connect_to_server
    measure.setup! # measure the time it took to setup
    index = 0
    while index < repeats
        client.get(...)
        measure.warmup
    end
end

There are other ways to do this, maybe with a better interface. But that's roughly how it could work.

from benchmark-ips.

Another proposal/idea:

x.benchmark(timer: true) do |timer|
	client = connect_to_server
	
	timer.capture do |repeats|
		index = 0
		while index < repeats
			client.get(...)
		end
	end
end

You could capture multiple blocks, e.g.

x.benchmark(timer: true) do |timer|
	client = connect_to_server
	
	timer.capture("get") do |repeats|
		index = 0
		while index < repeats
			client.get(...)
		end
	end
	
	timer.capture("post") do |repeats|
		index = 0
		while index < repeats
			client.post(...)
		end
	end
end

but maybe that's getting too complicated.

from benchmark-ips.

I agree with you and I realise my proposals are adding more complexity without a huge gain in functionality. The main point is to keep all the benchmark code within one block. To avoid leaking state. It gets a bit ugly when you have a ton of setup code, especially if different benchmarks shares similar concepts. Because it's not clear where one benchmark ends and the next one starts. It all gets mixed up and variable names are used to untangle it at the top level scope.

from benchmark-ips.

@eregon this is why the warmup cycles needs to be more than one.

from benchmark-ips.

I think it's good enough to take the setup outside of the report block here.
And so I'd suggest to close this issue, moving the setup outside the report is the recommended approach.

If you want to isolate local variables, you can always use an extra lambda as a scope:

Benchmark.ips do |x|
  -> {
    client = connect_to_server
    x.report "mybench" do
      client.get(...)
    end
  }.call
end

from benchmark-ips.