r10k provides comparative benchmarks for many ruby web application frameworks with 10, 100, 1000, and 10,000 routes. Here are the currently supported frameworks (in order of number of gem downloads):
- rails
-
Ruby on Rails (rubyonrails.org/)
- sinatra
-
Sinatra (www.sinatrarb.com/)
- roda
-
Roda (roda.jeremyevans.net)
- cuba
-
Cuba (cuba.is)
- hanami
-
Hanami (hanamirb.org/)
- syro
-
Syro (soveran.github.io/syro/)
These frameworks are designed to be run with the default settings. There are some variants that can be run with non-default settings:
- roda-hash-routes
-
Roda, using the hash_routes plugins
- roda-osm
-
Roda, using optimized_string_matchers and optimized_segment_matchers plugins
- hanami-api
-
Hanami, using hanami-api gem
The frameworks that are tested by default are those with more than 250,000 gem downloads, except for Sinatra:
rails roda cuba hanami
Sinatra is excluded by default because testing sinatra takes about 6x longer than testing Rails, and testing Rails takes 6-25x longer than testing Roda, Cuba, or Hanami.
After cloning the repository, install the related gems:
gem install gruff # if you want to create the graphs gem install rails roda cuba hanami # frameworks to test
A Gemfile is not used, because all web frameworks are tested independently, and with a Gemfile, bundler assumes all gems must be able to work together.
To use:
rake bench graphs
Note that rake bench can take significant time, especially on slower hardware. If you only want to benchmark specific apps, set the R10K_APPS environment variable:
rake bench graphs R10K_APPS="roda cuba"
The builders directory contains the files that build the applications. If you want to examine the application files used for benchmarking, you can use the build task and then look in the apps directory. The bench task produces CSV files which are stored in the data directory. The graphs task takes those CSV files and turns them into nice looking graphs (using gruff), storing the graphs in the graphs directory.
If you want to add support for a new framework, or check that support for existing frameworks runs correctly, use the check task:
rake check
This ensures the following:
-
Incomplete routes return 404
-
POST requests return 404 or 405
-
All valid routes return expected bodies
-
All valid routes return 200 status
-
All valid routes include text/html in Content-Type header
-
All valid routes return correct Content-Length
Because most Ruby web frameworks tested do not treat trailing slashes in routes correctly, considering a request without a trailing slash the same as a request with a trailing slash, the check task will only warn for incorrect handling of trailing slashes, instead of raising an exception. Of the web frameworks tested, only Roda and Sinatra treat routes with trailing slashes correctly.
To save time, checks are done with only 2 routes per level.
r10k historically used static paths in benchmarks. This was determined to be a poor benchmark, since the fastest implementations basically just used a Ruby hash for route lookup. The dynamic route benchmarks that r10k now uses are more similar to production routes. However, there may still be be value it benchmarking static route performance, so the static route benchmarks are still available:
rake static_bench static_graphs
The data for the static route benchmarks can be found in the static/data directory, and the graphs in the static/graphs directory.
Additional web frameworks are supported for the static benchmarks:
- nyny
-
NYNY (github.com/alisnic/nyny)
- rack-app
-
Rack::App (github.com/rack-app/rack-app)
- synfeld
-
Synfeld (github.com/swerling/synfeld)
- watts
-
Watts (github.com/pete/watts)
Additional variants are supported for the static benchmarks:
- static-route
-
A simple Rack implementation
- rails-minimal-middleware
-
Rails, with most middleware removed
- roda-multi-route
-
Roda, using the multi_route plugin
- roda-run
-
Roda, using separate Roda applications per branch
- roda-static-routing
-
Roda, using the static_routing plugin
- sinatra-minimal-middleware
-
Sinatra, using the minimum middleware
Environment variables:
- DIM
-
Dimensions for the graphs (default: 1280x720)
- LEVELS
-
The number of levels for the routes (default: 4)
- LEGEND_FONT_SIZE
-
Override base font size for legend text in graphs
- R10K_APPS
-
Which applications/configurations to benchmark
- R10K_ITERATIONS
-
The number of iterations to run (default: 2)
- R10K_NUM_THREADS
-
The number of threads to run (default: 1)
- R10K_WARMUP_ITERATIONS
-
The number of warmup iterations to run (default: 0)
- ROUTES_PER_LEVEL
-
The number of routes in each level (default: 10)
Total number of routes in largest application is ROUTES_PER_LEVEL**LEVELS.
MIT
Jeremy Evans <[email protected]>