evanphx / benchmark-ips Goto Github PK

It appears a few files are missing from the gem file for the 2.8.0 release. A couple of files reference StatsMetric, which is defined in lib/benchmark/ips/stats/stats_metric.rb, but that file is not included in the released gem file.

Hi, could you please update the history.txt file with information about the 2.8.x releases? From the commits alone it's hard to see what could be relevant to know for update and what not. Thanks!

When I try to benchmark some code, my program seems to be stuck at calculation phase after successful warm-up

I tried next piece of code (with and without manually set time and warmup):

require "benchmark/ips"

a = (1..10).to_a
b = (1..10).to_a

Benchmark.ips do |x|
  x.config(:time => 2, :warmup => 2)
  x.report("+ code performance") { a + b }
  x.report("concat code performance") {  a.concat(b) }
  x.compare!
end

ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-darwin17] and benchmark 2.7.2 were used

I believe the commit for v2.8.2 is a44e3dc. Thanks!

It would be nice, to have a way to capture the output to a String in order to print it or send it to an external service, like Bugsnag. Also in a PASS like heroku it's hard to get the reports because the logs are combined from logger, stdout and stderr.

By output I mean the full report:

Calculating -------------------------------------
            addition    71.254k i/100ms
           addition2    68.658k i/100ms
           addition3    83.079k i/100ms
addition-test-long-label
                        70.129k i/100ms
-------------------------------------------------
            addition     4.955M (± 8.7%) i/s -     24.155M
           addition2    24.011M (± 9.5%) i/s -    114.246M
           addition3    23.958M (±10.1%) i/s -    115.064M
addition-test-long-label
                         5.014M (± 9.1%) i/s -     24.545M

Comparison:
           addition2: 24011974.8 i/s
           addition3: 23958619.8 i/s - 1.00x slower
addition-test-long-label:  5014756.0 i/s - 4.79x slower
            addition:  4955278.9 i/s - 4.85x slower

This isn't possible today:

Benchmark.ips do |x|
  x.suite = suite
  x.warmup = 3
  x.time = 10
  x.iterations = 5
end

but this is

Benchmark.ips do |x|
  x.config(
    suite: suite,
    warmup: 3,
    time: 10,
    iterations: 5,
  )

Is not being able to set x.suite = a miss? If so, I'm happy to PR that change.

Do you think it's possible to add a offset and/or factor to control the times parameter?

It would be nice if we could tell it to run at least 10 iterations minimum, for example.

Files required in lib/benchmark/ips.rb are not included in the released gem:

$ irb
irb(main):001:0> require "benchmark/ips"
LoadError: cannot load such file -- benchmark/ips/job/entry
    from /Users/jip/.rubies/ruby-2.3.0/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /Users/jip/.rubies/ruby-2.3.0/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /Users/jip/.gem/ruby/2.3.0/gems/benchmark-ips-2.4.0/lib/benchmark/ips.rb:5:in `<top (required)>'
    from /Users/jip/.rubies/ruby-2.3.0/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in `require'
    from /Users/jip/.rubies/ruby-2.3.0/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:127:in `rescue in require'
    from /Users/jip/.rubies/ruby-2.3.0/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require'
    from (irb):1
    from /Users/jip/.rubies/ruby-2.3.0/bin/irb:11:in `<main>'
irb(main):002:0> 

$ tree (dirname (gem which benchmark/ips))
/Users/jip/.gem/ruby/2.3.0/gems/benchmark-ips-2.4.0/lib/benchmark
├── compare.rb
├── ips
│   ├── job.rb
│   └── report.rb
├── ips.rb
└── timing.rb

1 directory, 5 files

Unfortunately, I'm not familiar enough with hoe to make a pull request.

this gem defines Benchmark::IPS::Report::Entry.new with an initialization arity of 5 parameters
https://github.com/evanphx/benchmark-ips/blob/master/lib/benchmark/ips/report.rb

but davy/benchmark-bigo appears to call this code with an arity of 6 parameters
https://github.com/davy/benchmark-bigo/blob/master/lib/benchmark/bigo/report.rb

there's also a mismatch with the add_entry method call itself (in reverse actually, this gem calls add_entry which is defined with a different arity in davy/benchmark-bigo), which I filed as an issue over at benchmark-bigo (see davy/benchmark-bigo#14)

The current released version is 2.7.2 from almost four years ago. However new features (#save!, for example, 18 months ago) have come in since then.

Would you be able to push a new version to RubyGems?

I faced the following error for the version 2.8.0 that was released today.
I am using Ruby 2.7.1 on Fedora 31.

$ which ruby
/usr/local/ruby-2.7.1/bin/ruby

$ which gem
/usr/local/ruby-2.7.1/bin/gem

$ gem -v
3.1.2

$ gem install --user-install benchmark-ips
Fetching benchmark-ips-2.8.0.gem
Successfully installed benchmark-ips-2.8.0
Parsing documentation for benchmark-ips-2.8.0

RDoc::Parser::Ruby failure around line 176 of
lib/benchmark/ips.rb

end

Before reporting this, could you check that the file you're documenting
has proper syntax:

  /home/root/local/ruby-2.7.1/bin/ruby -c lib/benchmark/ips.rb

RDoc is not a full Ruby parser and will fail when fed invalid ruby programs.

The internal error was:

	(NoMethodError) undefined method `[]' for nil:NilClass

ERROR:  While executing gem ... (NoMethodError)
    undefined method `[]' for nil:NilClass

$ gem install --user-install benchmark-ips -v 2.8.0
Successfully installed benchmark-ips-2.8.0
Parsing documentation for benchmark-ips-2.8.0

RDoc::Parser::Ruby failure around line 176 of
lib/benchmark/ips.rb

end

Before reporting this, could you check that the file you're documenting
has proper syntax:

  /home/root/local/ruby-2.7.1/bin/ruby -c lib/benchmark/ips.rb

RDoc is not a full Ruby parser and will fail when fed invalid ruby programs.

The internal error was:

	(NoMethodError) undefined method `[]' for nil:NilClass

ERROR:  While executing gem ... (NoMethodError)
    undefined method `[]' for nil:NilClass

It is okay for the 2.7.2.

$ gem install --user-install benchmark-ips -v 2.7.2
Successfully installed benchmark-ips-2.7.2
Parsing documentation for benchmark-ips-2.7.2
Done installing documentation for benchmark-ips after 0 seconds
1 gem installed

As implementations of Ruby get more powerful, benchmarks written using benchmark-ips, and other micro-benchmarks in general are at risk of being silently optimised away. Benchmarks are already confusing for non-specialists, and at the moment the only way of working that out is to look at generated machine code.

Take this example benchmark from the documentation:

  x.report("addition2") do |times|
    i = 0
    while i < times
      1 + 2
      i += 1
    end
  end

The first problem is that the operation being benchmarked here is runtime constant! With inline caches, dynamic inlining and constant folding, we can reduce 1 + 2 to 3, and with dynamic deoptimization we can do it without any guards. I think at least Topaz and Truffle can achieve that today, and JRuby probably will be able to achieve it with the new IR - I'm not sure. I'm also not sure about Rubinius. Maybe a future MRI JIT will also be able to do it.

The second problem is that the whole loop itself is also vulnerable to being optimised away. It performs no side effects and produces no value (except nil). You could say it observes side effects, but with dynamic deoptimisation, or with hoisting guard out of the loop, all the side effect observations of the loop can be modelled as happening instantaneously, once for the entire loop. I don't believe any implementation of Ruby can currently remove this loop (it is not easy to do in practice) but we're certainly working towards it very quickly in Truffle.

What can we do about this?

The root of the problem is that the literal values 1 and 2 are constants, and the compiler can see this. What about introducing a special function that the compiler will pretend that it cannot see through. Assuming we could get all implementations on board, we could perhaps call this Kernel#optimisation_barrier. Then the code would look like this:

  x.report("addition2") do |times|
    i = 0
    while i < times
      optimisation_barrier(1) + optimisation_barrier(2)
      i += 1
    end
  end

This solves the first problem. I'm not sure if it also solves the second (currently hypothetical) problem, as the loop body is no longer constant but does that matter for removing the loop? We could remove the computation if we are sure it has no side effects - and there's no possibility of overflow here so I don't think there are any. I can implement this optimisation_barrier in Truffle today. For MRI and other implementations it could perhaps be a no-op. If we can't get all implementations on board, benchmark-ips could define it as a no-op if the implementation doesn't provide one. We could also pull that out into a separate gem.

Downsides are that the person writing benchmarks has to figure out where to add these barriers, and although it's a no-op in MRI, MRI is not able to inline through it and so it may add significant overhead.

What do other Ruby implementors, @headius and @brixen, think about this? Should we standardise on a barrier like this across all implementations?

I noticed that doing something like this no longer works on the latest version of this gem.

report = Benchmark.ips(time, warmup, true) do |bm|
  bm.report(label, &block)
end

entry = report.entries.first

puts entry.ips
puts entry.stddev_percentage

It looks like the accessor/method have been removed in #69 but the changes were not reflected in a512373.

I'll like to find out if Benchmark::IPS::Report is meant for the Gem's internal use or if they are exposed as public APIs. I'll be more than happy to submit a PR to deprecate the methods first instead if the latter is true.

as the title says. It's just noise at that point.

I'm getting this consistently across multiple Ruby implementations benchmarking Celluloid's UUID implementation. I hope that means it's roflscale.

Repro:

clone git://github.com/celluloid/celluloid.git
bundle
ruby benchmarks/uuid.rb

I get:

Calculating -------------------------------------
                uuid     22819 i/100ms
-------------------------------------------------
                uuidI, [2012-03-24T16:45:12.456314 #66670]  INFO -- : Shutdown completed cleanly
/Users/tony/.rvm/gems/ruby-1.9.3-p125/gems/benchmark-ips-1.1.0/lib/benchmark/ips.rb:239:in `round': NaN (FloatDomainError)
    from /Users/tony/.rvm/gems/ruby-1.9.3-p125/gems/benchmark-ips-1.1.0/lib/benchmark/ips.rb:239:in `block in ips'
    from /Users/tony/.rvm/gems/ruby-1.9.3-p125/gems/benchmark-ips-1.1.0/lib/benchmark/ips.rb:201:in `each'
    from /Users/tony/.rvm/gems/ruby-1.9.3-p125/gems/benchmark-ips-1.1.0/lib/benchmark/ips.rb:201:in `ips'
    from benchmarks/uuid.rb:8:in `<main>'

Or on rbx:

An exception occurred running benchmarks/uuid.rb
    Numerical argument out of domain - sqrt (Errno::EDOM)

Or on JRuby:

Math::DomainError: Numerical argument is out of domain - "sqrt"
    sqrt at org/jruby/RubyMath.java:494

Hello there, and as always, thanks for the amazing job on benchmark-ips!

I noticed this paper cut on one of the tests we do on Datadog's ddtrace gem: if the warm up time is "small enough", we can trigger a division by 0.0.

Here's an extreme example:

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'

  gem 'benchmark-ips'
end

Benchmark.ips do |x|
  x.config(:warmup => 0.0000000001, :time => 1)
  x.report("addition") { 1 + 2 }
  x.compare!
end

And the result:

$ ruby bench-ips-testcase.rb
Warming up --------------------------------------
            additionTraceback (most recent call last):
	9: from bench-ips-testcase.rb:9:in `<main>'
	8: from /Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips.rb:53:in `ips'
	7: from /Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips/job.rb:247:in `run'
	6: from /Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips/job.rb:247:in `times'
	5: from /Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips/job.rb:248:in `block in run'
	4: from /Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips/job.rb:263:in `run_warmup'
	3: from /Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips/job.rb:263:in `each'
	2: from /Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips/job.rb:294:in `block in run_warmup'
	1: from /Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips/job.rb:189:in `cycles_per_100ms'
/Users/ivo.anjo/.rvm/gems/ruby-2.7.5/gems/benchmark-ips-2.9.2/lib/benchmark/ips/job.rb:189:in `to_i': Infinity (FloatDomainError)

I ran into this on a test where we didn't need a lot of warmup, but did not disable it either -- only set it low-ish enough that it ended up working most of the time, but of course, caused a flaky test in CI.

The correct solution in our case is to disable the warmup (which I'll do) but I thought I'd report it anyway :)

I just wanted to use a % sign in the labels of my benchmarks, then I saw this:

Calculating -------------------------------------
/Users/paddor/.gem/ruby/2.1.3/gems/benchmark-ips-2.0.0/lib/benchmark/ips/job.rb:154:in `printf': malformed format string - %) (ArgumentError)
    from /Users/paddor/.gem/ruby/2.1.3/gems/benchmark-ips-2.0.0/lib/benchmark/ips/job.rb:154:in `block in run_warmup'
    from /Users/paddor/.gem/ruby/2.1.3/gems/benchmark-ips-2.0.0/lib/benchmark/ips/job.rb:150:in `each'
    from /Users/paddor/.gem/ruby/2.1.3/gems/benchmark-ips-2.0.0/lib/benchmark/ips/job.rb:150:in `run_warmup'
    from /Users/paddor/.gem/ruby/2.1.3/gems/benchmark-ips-2.0.0/lib/benchmark/ips.rb:37:in `ips'
    from compression_benchmark.rb:15:in `<main>'

Using %% instead is a workaround in the calculation round, but in the second round (the results round), I'll see %% in the output.

I know it's not very common, but I was benchmarking some different compression algorithms for a project and also wanted to include the compression ratio in the benchmark report. :)

I have to look up Benchmark.ips EVERY time and the (ri) doco is still a PITA.

Pls consider a simple top level driver method like so:

def quick_bench_ips(*methods, warm:nil, time:nil, show:nil)
  Benchmark.ips do |x|
    x.warmup = warm if warm
    x.time   = time if time
    x.compare!

    if show then
      max = methods.map(&:size).max
      methods.each do |name|
        val = send name
        puts "%*s: %p" % [max, name, val]
      end
      return
    end

    methods.each do |name|
      x.report(name) do |x|
        x.times { send name }
      end
    end
  end
end



def report_tally1
  # ...
end

def report_tally2
  # ...
end

def report_tally3
  # ...
end

quick_bench_ips(:report_tally1,
                :report_tally2,
                :report_tally3)

I need to have an object or public variable to get the report information. Right now you keep it as string but could be amazing to have a better way to see this.

The link to benchmark.fyi in the README.md in the section Online sharing is pointing to what looks like a default page for ArcGIS, which I guess is not where that should be pointing. A quick google shows that https://ips.fastruby.io/ is probably where that link should be pointing.

The travis page of benchmark-ips is here: https://travis-ci.org/evanphx/benchmark-ips

What do we need to do to activate it on Travis CI 😄 ? Thanks!!!

Hi Evan,

the latest release of benchmark-ips has been mid January.
Since then some nice changes have been made: 75d789b...master

Do you plan to release a new version soonish?

Kind regards,
Peter

I need to benchmark and compare 2 blocks of code with and without the inclusion of a module.
If I do something like:

def my_method
  Benchmark.ips do |x|

    x.report('without') do
      something
    end

    code_to_include_module

    x.report('with') do
      something
    end

    x.compare!

  end
end

The inclusion happens immediately, so both blocks run with the inclusion. Using hold! doesn't change it. How could I do it?

The ri info available on Benchmark::IPS, Benchmark.ips, and Benchmark::IPS.ips is either nonexistant or devoid of information to the point where it isn't useful at all. Please for the love of god make the doco on this project useful. I don't want to have to google this stuff just because I only use it once a month.

I'm interested in comparing two (or more) implementations of the same thing.

There is some "one off" setup cost - establishing the connection, and then the repeated times cost, i.e. what I'm interested in.

For one benchmark, the setup cost might be N and for the other, 2N. The iteration cost is largely the same. However, what ends up happening is that for the benchmark with N setup overhead, the times repeats is much larger than the benchmark with 2N overhead. This causes the effect of the setup to be even more pronounced, because benchmark-ips will set times to, say, 80, for the case of 2N setup cost, and 500 for the case of N setup cost, so you end up with:

• (2N + 80) * number of repeats until 5 seconds is elapsed, vs
• (N + 500) * number of repeats until 5 seconds is elapsed.

Ultimately, it makes the 2nd case look much better even though the different is mostly in the setup overhead.

Is there some way to take this bias into account? My initial thoughts were to use the upper bound for times (or perhaps the average) so that each benchmark would be largely running with the same number, proportionally, of setup overhead to number of times.

Hello,

First of all, thank you for such a useful gem!

I just tried to use SHARE=1 to upload my results to https://benchmark.fyi but I got this error message:

$ SHARE=1 bundle exec ruby benchmarks/unless_nil_vs_if_object.rb
Warming up --------------------------------------
          unless nil    63.507k i/100ms
           if object   271.532k i/100ms
Calculating -------------------------------------
          unless nil    799.058k (± 3.0%) i/s -      4.001M in   5.011766s
           if object     12.172M (± 5.2%) i/s -     60.823M in   5.011318s

Comparison:
           if object: 12172119.9 i/s
          unless nil:   799058.5 i/s - 15.23x  slower

Error sharing report

After debugging benchmark-ips, I noticed that I'm getting this response:

[12] pry(#<Benchmark::IPS::Share>)> res
=> #<Net::HTTPBadRequest 400 Bad Request readbody=true>

It's weird, because my code is pretty standard:

require 'benchmark/ips'

Benchmark.ips do |x|
  OBJECT = "nil".freeze

  x.report("unless nil") do
    20.times do
      true unless OBJECT.nil?
    end
  end

  x.report("if object") do
    true if OBJECT
  end

  x.compare!
end

This is the body in the request:

[8] pry(#<Benchmark::IPS::Share>)> req.body
=> "{\"entries\":[{\"name\":\"unless nil\",\"central_tendency\":797392.5723155453,\"ips\":797392.5723155453,\"error\":34976,\"stddev\":34976,\"microseconds\":5076026.0,\"iterations\":4038208,\"cycles\":63097},{\"name\":\"if object\",\"central_tendency\":12144478.462478887,\"ips\":12144478.462478887,\"error\":644754,\"stddev\":644754,\"microseconds\":5012625.0,\"iterations\":60690718,\"cycles\":268543}],\"options\":{\"compare\":true}}"

After reviewing the code for benchmark.fyi, it seems that the problem is with some of the parameters: https://github.com/evanphx/benchmark.fyi/blob/master/app/controllers/reports_controller.rb#L8-L54

If any of the keys for the entry is not recognized, it will error out with a 400 response code: https://github.com/evanphx/benchmark.fyi/blob/master/app/controllers/reports_controller.rb#L6

So, it seems to be an issue with benchmark.fyi, not benchmark-ips:

The API accepts these keys per entry:

DATA_KEY = %W!name ips stddev microseconds iterations cycles!

But benchmark-ips is sending these keys:

["name", "central_tendency", "ips", "error", "stddev", "microseconds", "iterations", "cycles"]

What do you think is the best solution to solve this in benchmark.fyi?

Due the missing gemspec, the gem can't be used with Bundler as a git repo:

gem 'benchmark-ips', github: 'evanphx/benchmark-ips', commit: 'abc'

@evanphx do you mind if I send a PR fixing it?

Hello!

I'm trying to benchmark a few different approaches to a problem but the various approaches make process-global changes which can pollute and affect the runtimes of the other reports. What do you think about:

1. Running each report in its own child fork?
2. Adding support for a post-fork, pre-report callback to trigger the global state change?

Specifically the flavor! call below makes global changes to ::JSON, which can affect everything.

Benchmark.ips do |x|
  x.report("Sidekiq::Job v7") do |times|
    i = 0
    while i < times
      MyJob.perform_async("foo", 123, {"mike" => true}, Time.now.to_f)
      i += 1
    end
  end
  x.report("ActiveJob v#{ActiveJob.version}") do |times|
    i = 0
    while i < times
      AJob.perform_later(:foo, 123, {mike: 0..5}, Time.now)
      i += 1
    end
  end
  x.report("Sidekiq::Job v8") do |times|
    Sidekiq::JSON.flavor!(:v8)
    i = 0
    while i < times
      MyJob.perform_async(:foo, 123, {"mike" => 0..5}, Time.now)
      i += 1
    end
  end
end

While working on #113 I noticed the test suite here is really slow:

Has anyone looked into this?

ruby -Ilib:test:. -w -e 'require "minitest/autorun"; require "test/test_benchmark_ips.rb"' -- --profile
Run options: --profile --seed 51810

# Running:

...............

Finished in 59.705193s, 0.2512 runs/s, 0.7035 assertions/s.

15 runs, 42 assertions, 0 failures, 0 errors, 0 skips

================================================================================
Your 10 Slowest Tests
================================================================================

17.0485ms - TestBenchmarkIPS#test_quiet
 7.0211ms - TestBenchmarkIPS#test_json_output
 7.0170ms - TestBenchmarkIPS#test_ips_report_using_symbol
 7.0169ms - TestBenchmarkIPS#test_ips_defaults
 7.0166ms - TestBenchmarkIPS#test_ips_default_data
 3.9220ms - TestBenchmarkIPS#test_ips
 3.0117ms - TestBenchmarkIPS#test_output
 2.0114ms - TestBenchmarkIPS#test_kwargs
 2.0114ms - TestBenchmarkIPS#test_ips_alternate_config
 2.0113ms - TestBenchmarkIPS#test_ips_old_config

When I profile it, it seems like it's waiting on a thread but I can't seem to figure out where that's coming from

ruby -r profile test/test_benchmark_ips.rb -n test_quiet
Run options: -n test_quiet --seed 55234

# Running:

.

Finished in 10.115211s, 0.0989 runs/s, 0.0000 assertions/s.

1 runs, 0 assertions, 0 failures, 0 errors, 0 skips
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
197.30    16.22     16.22        2  8107.71  8107.71  Thread::Queue#pop
 34.78    19.07      2.86       77    37.12    73.40  #<Benchmark::IPS::Job::Entry:0x0000556a0dda3140>.call_times
 26.24    21.23      2.16   386597     0.01     0.03  TestBenchmarkIPS#test_quiet
 14.94    22.46      1.23       37    33.19    65.53  #<Benchmark::IPS::Job::Entry:0x0000556a0dc2cd70>.call_times
  7.61    23.08      0.63   386708     0.00     0.00  Integer#<

I opened up ruby/profile#3 to see if there's a way to track that down.

Right now I'm using awk, sed, etc. to parse i/ps and std. dev from the output. It would be nice to have some JSON output function.

Something like this would be nice:

[
  {
    "name": "fn. 1",
    "i/ps": 12.34,
    "perc. std. dev.": 56,
    "etc.": "etc."
  }
]

Maybe, I'll try to throw something together later.

Showerthought.

The reason why benchmark/ips is so great is it helps us to set and decide on what would otherwise be pretty arbitrary parameters for benchmarks - particularly of course number of iterations.

I think we could go further.

• A benchmark is "warmed up" when iterations of the benchmark don't get significantly faster.
• Rather than just running for 5 seconds, a benchmark iteration can run until significance is achieved or a timeout is reached.
• Once a result is achieved, it can be immediately replicated by iterating the benchmark again. If the result is replicated, stop. If it isn't replicated, try again, and if it doesn't replicate after X attempts, error out.

Just some ideas. I think I just want to get rid of the three remaining "config points" in benchmark/ips: warmup time, iteration runtime, and number of iterations.

TL;DR: It would be cool to set an array of inputs as part of benchmark job config, and have every input yielded to each report block.

Motivating Example

Let's say you're benchmarking different regular expressions. You'll preferably try each variant on not just one, but a set of input strings:

STRINGS = ["fool", "founded", "funded", "foolish"]
REGEX1 = /fo+/
REGEX2 = /foo*/

Benchmark.ips do |job|
  job.report("regex1"){ STRINGS.each{|s| s =~ REGEX1 } }
  job.report("regex2"){ STRINGS.each{|s| s =~ REGEX2 } }
  job.compare!
end

This works, but it's a tad messy. You could DRY it up by defining regex1 and regex2 methods, but that adds some marginal amount of performance overhead. On the topic of overhead, though, those STRINGS.each calls may be a problem. You could eliminate the each loop using careful string concatenation:

STRINGS = ["fool", "founded", "funded", "foolish"]
REGEX1 = /fo+/
REGEX2 = /foo*/

R1 = STRINGS.each_index.map{|i| "STRINGS[#{i}] =~ REGEX1" }.join(";")
R2 = STRINGS.each_index.map{|i| "STRINGS[#{i}] =~ REGEX2" }.join(";")

Benchmark.ips do |job|
  job.report("regex1", R1)
  job.report("regex2", R2)
  job.compare!
end

But that's even uglier, and more error-prone, and if your expressions boil down to e.g. == comparison, the interpreter may throw warnings at you like "possibly useless use of == in void context."

Proposed Solution

It would be nice if you could do something like the following:

REGEX1 = /fo+/
REGEX2 = /foo*/

Benchmark.ips do |job|
  job.inputs = ["fool", "founded", "funded", "foolish"]
  job.report("regex1"){|s| s =~ REGEX1 }
  job.report("regex2"){|s| s =~ REGEX2 }
  job.compare!
end

Preserving the semantics of the first example, while not counting the loop overhead in the run time.

Other Benefits

Tasks which destructively modify their inputs can be difficult to accurately benchmark, because cloning their inputs between iterations can skew run times. (See fastruby/fast-ruby#101 for an example of this.) Given a reference to the inputs, the job could perform the necessary clone, also excluding it from the run times. For efficiency, this behavior could be disabled by default, and enabled via e.g. job.destructive? = true.

8687 % g -w 20
lib/benchmark/ips/report.rb
95:              left.ljust(20) + (" - %s in %10.6fs" % [iters, runtime])
97:              left.ljust(20) + (" - %s" % iters)
103:              left.ljust(20) + (" - %10d in %10.6fs" % [@iterations, runtime])
105:              left.ljust(20) + (" - %10d" % @iterations)
110:        # Return header with padding if +@label+ is < length of 20.
113:          @label.to_s.rjust(20)

lib/benchmark/ips/job/stream_report.rb
51:        # Add padding to label's right if label's length < 20,
52:        # Otherwise add a new line and 20 whitespaces.
56:          if label.size > 20
57:            "#{label}\n#{' ' * 20}"
59:            label.rjust(20)

for the love of all that is holy... please pull this up to a config value

ruby version: ruby 2.5.7p206 (2019-10-01 revision 67816) [x86_64-linux]

gem version benchmark-ips (2.8.0)

trace:

   10: from profiling.rb:2:in `<main>'
    9: from profiling.rb:2:in `require'
    8: from /home/dmitry/work/some_project/lib/env.rb:3:in `<top (required)>'
    7: from /home/dmitry/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/bundler-2.1.4/lib/bundler.rb:174:in `require'
    6: from /home/dmitry/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `require'
    5: from /home/dmitry/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `each'
    4: from /home/dmitry/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:65:in `block in require'
    3: from /home/dmitry/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:88:in `rescue in block in require'
    2: from /home/dmitry/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:88:in `require'
    1: from /home/dmitry/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/benchmark-ips-2.8.0/lib/benchmark/ips.rb:4:in `<top (required)>'
/home/dmitry/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/benchmark-ips-2.8.0/lib/benchmark/ips.rb:4:in `require': cannot load such file -- benchmark/ips/stats/stats_metric (LoadError)

When I have reinstalled with gem source to github repo everything worked fine.

Failure/Error: require 'benchmark/ips'

LoadError:
  cannot load such file -- benchmark/ips/noop_suite
# /Users/samuel/.gem/ruby/3.0.0/gems/benchmark-ips-2.9.0/lib/benchmark/ips.rb:8:in `require'
# /Users/samuel/.gem/ruby/3.0.0/gems/benchmark-ips-2.9.0/lib/benchmark/ips.rb:8:in `<top (required)>'
# ./spec/db/benchmark_spec.rb:23:in `require'
# ./spec/db/benchmark_spec.rb:23:in `<top (required)>'

I'm not sure what's going on, but the latest release seems broken.

I'd suggest you change to use require_relative everywhere too:

gem "benchmark-ips", "~> 2.8.0"

fixes the issue for me.

Hello there! Thanks a lot for benchmark-ips :)

Today I was using it for a few tests and noticed that whenever the percentage of standard deviation shows up in the comparison, it's always zero. E.g. with the example on the README.md I get:

Warming up --------------------------------------
            addition     2.390M i/100ms
           addition2     5.597M i/100ms
           addition3     5.260M i/100ms
addition-test-long-label
                         2.377M i/100ms
Calculating -------------------------------------
            addition     23.737M (± 3.0%) i/s -    119.519M in   5.040228s
           addition2     54.560M (± 3.0%) i/s -    274.271M in   5.031700s
           addition3     52.988M (± 1.8%) i/s -    268.258M in   5.064310s
addition-test-long-label
                         24.121M (± 1.3%) i/s -    121.248M in   5.027489s

Comparison:
           addition2: 54560399.9 i/s
           addition3: 52988283.5 i/s - same-ish: difference falls within error
addition-test-long-label: 24121165.9 i/s - 2.26x  (± 0.00) slower <-- zero?
            addition: 23736650.2 i/s - 2.30x  (± 0.00) slower <--- zero?

Every benchmark I also ran today always showed the (± 0.00) in the comparison. Is this expected?

I'm using benchmark-ips version 2.8.4, using different Ruby versions, on macOS.