Hi, We are benchmarking some <code class="notransl

Sure. <g-emoji class="g-emoji" alias="warning" fallback-src="https://github.github

I have now (env : 4 CPU x 8 Go) wrk

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

I just noticed the Dockerfile <code class="notranslat

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Comments (20)

ioquatix commented on May 21, 2024 6

There are two aspects of this benchmark which I think needs to be clarified:

Falcon is still < 1.0 and I haven't invested a huge amount of effort into raw performance.
Falcon is a server built around concurrency. That means I was more interested in this kind of graph.

I think setting up benchmarks like these is awesome and a great way to push development (and understanding). Puma is obviously a lot more mature than Falcon.

One final thing to keep in mind, is that Puma is typically run behind Nginx. Falcon is designed to be run directly connected to the client. Internally Falcon will probably end up with a proxy (need to receive the request and forward to a different process). That being said, the way this affects performance in production is ultimately something worth discussing and testing.

from falcon.

mhenrixon commented on May 21, 2024 6

HAH! Puma just released v4 which uses nio4r :) You are onto something @ioquatix and I am super excited about where you are going to take falcon now :)

from falcon.

ioquatix commented on May 21, 2024 4

Nice work everyone I will take a look when I have time.

from falcon.

waghanza commented on May 21, 2024 2

Sure.
⚠️ You need to install crystal first => https://crystal-lang.org/docs/installation/

Clone https://github.com/the-benchmarker/web-frameworks
Run shards install && shards build --release && bin/neph ruby && bin/benchmarker ruby

ℹ️ This is using puma

And you need to

Clone my fork on the branch use_falcon from https://github.com/waghanza/http-benchmark
Run shards install && shards build --release && bin/neph ruby && bin/benchmarker ruby

ℹ️ All stuff are running on containers (wip to change)

from falcon.

waghanza commented on May 21, 2024 1

I have now (env : 4 CPU x 8 Go)

wrk http://0.0.0.0:3000

falcon
===============================
Running 10s test @ http://0.0.0.0:3000
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.85ms  727.69us  20.33ms   94.77%
    Req/Sec     6.10k     2.90k    9.43k    47.00%
  121338 requests in 10.00s, 26.61MB read
Requests/sec:  12132.35
Transfer/sec:      2.66MB

puma
===============================
Running 10s test @ http://0.0.0.0:3000
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   646.64us  807.83us  21.56ms   94.33%
    Req/Sec     9.47k   570.28    10.20k    76.00%
  188404 requests in 10.00s, 30.90MB read
Requests/sec:  18837.14
Transfer/sec:      3.09MB

our benchmark (wrk on 3 rules the-benchmarker/web-frameworks#221 (comment))

	Framework (Middleware)	Average	50th percentile	90th percentile	99th percentile	99.9th percentile	Standard deviation
puma	roda (3.13)	1.74 ms	0.19 ms	4.21 ms	27.84 ms	88.72 ms	5173.33
puma	rack-routing (0.0)	2.12 ms	0.35 ms	5.13 ms	30.27 ms	103.19 ms	5662.33
puma	flame (4.18)	3.62 ms	0.54 ms	10.13 ms	39.80 ms	98.39 ms	7542.67
puma	sinatra (2.0)	4.12 ms	1.16 ms	10.07 ms	49.56 ms	120.51 ms	8978.00
puma	rails (5.2)	18.39 ms	2.55 ms	63.96 ms	118.43 ms	256.87 ms	28863.00
puma	hanami (1.2)	5.87 ms	4.42 ms	10.59 ms	29.85 ms	174.60 ms	6269.00
falcon	roda (3.13)	58.55 ms	52.60 ms	86.29 ms	136.56 ms	783.38 ms	29647.00
falcon	rack-routing (0.0)	65.72 ms	63.42 ms	83.80 ms	133.13 ms	642.12 ms	26455.33
falcon	sinatra (2.0)	102.68 ms	82.64 ms	151.66 ms	470.13 ms	1613.69 ms	93714.67
falcon	flame (4.18)	96.56 ms	86.32 ms	145.91 ms	277.76 ms	942.45 ms	58451.00
falcon	hanami (1.2)	146.52 ms	134.39 ms	231.37 ms	314.18 ms	757.97 ms	65566.33
falcon	rails (5.2)	328.06 ms	320.40 ms	386.02 ms	1097.99 ms	2080.99 ms	163640.33

For me falcon perform better than puma, but using docker is misleading, I will follow our update / change for docker to instances

from falcon.

waghanza commented on May 21, 2024 1

@ioquatix Sure, puma is more mature. That's why I will no merge to master 😝

In our benchmark, we use raw puma (no nginx) -> keep track of falcon on our production upgrade (we will run on http://digitalocean.com 8 CPU x 32 Go)

from falcon.

aemadrid commented on May 21, 2024

Not looking good. Could you give us some way to replicate the tests?

from falcon.

picatz commented on May 21, 2024

@waghanza I just did a bit of local testing, specifically on the sinatra one. I think I may know why you're getting those results.

I hope I am not mistaken here; but I don't believe the setup laid out in the config.ru is correct, and in my testing with rackup it defaulted to webbrick. @ioquatix may know how to do this without changing much.

However, to "fix" this, there is the falcon command (which is installed with the gem) which I used that dramatically changed the performance of the server.

Moreover, it's important to remember that HTTP2 is on by default, and this requires SSL which isn't enforced by the other frameworks; but this can be turned off with a proper --bind flag:

$ falcon serve -c config.ru --bind http://localhost:9292

Note: if I use rackup for the puma example in the repo without changing anything, it uses puma under the hood. So, I imagine this is a point of contention for sure, since this isn't the case for falcon.

In my stupidly simple apache bench test:

$ ab -n 5000 -c 100 http://localhost:9292/

Puma Perf

Started with rackup

Server Software:        
Server Hostname:        localhost
Server Port:            9292

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      100
Time taken for tests:   4.209 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      860000 bytes
HTML transferred:       0 bytes
Requests per second:    1187.87 [#/sec] (mean)
Time per request:       84.185 [ms] (mean)
Time per request:       0.842 [ms] (mean, across all concurrent requests)
Transfer rate:          199.52 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       5
Processing:     9   83   8.3     82     126
Waiting:        7   81   8.3     81     125
Total:         12   83   8.1     82     126

Percentage of the requests served within a certain time (ms)
  50%     82
  66%     84
  75%     84
  80%     85
  90%     88
  95%     95
  98%    109
  99%    117
 100%    126 (longest request)

Falcon Performance

Started with falcon serve -c config.ru --bind http://localhost:9292

Server Software:        falcon/0.18.14
Server Hostname:        localhost
Server Port:            9292

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      100
Time taken for tests:   1.420 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      1245000 bytes
HTML transferred:       0 bytes
Requests per second:    3521.11 [#/sec] (mean)
Time per request:       28.400 [ms] (mean)
Time per request:       0.284 [ms] (mean, across all concurrent requests)
Transfer rate:          856.21 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   1.6      2       7
Processing:     2   25  22.1     21     184
Waiting:        1   12  15.1      8     182
Total:          2   28  22.4     23     185

Percentage of the requests served within a certain time (ms)
  50%     23
  66%     28
  75%     32
  80%     35
  90%     44
  95%     55
  98%    138
  99%    147
 100%    185 (longest request)

With HTTPs, things are different (since there is literally more being done for each request):

Server Software:        falcon/0.18.14
Server Hostname:        localhost
Server Port:            9292
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-CHACHA20-POLY1305,2048,256
TLS Server Name:        localhost

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      100
Time taken for tests:   38.872 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      1245000 bytes
HTML transferred:       0 bytes
Requests per second:    128.63 [#/sec] (mean)
Time per request:       777.437 [ms] (mean)
Time per request:       7.774 [ms] (mean, across all concurrent requests)
Transfer rate:          31.28 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       12  527 108.3    543     795
Processing:    18  242 103.3    218     714
Waiting:       10  104 126.0     46     692
Total:         56  769  59.4    761    1190

Percentage of the requests served within a certain time (ms)
  50%    761
  66%    766
  75%    770
  80%    772
  90%    788
  95%    807
  98%    974
  99%   1136
 100%   1190 (longest request)

from falcon.

picatz commented on May 21, 2024

I just noticed the Dockerfile CMD to see that both puma and falcon commands are used in the use_falcon branch.

(This is correct behavior, as far as I know.)

from falcon.

waghanza commented on May 21, 2024

@picatz on which framework ?

from falcon.

picatz commented on May 21, 2024

@waghanza sinatra

from falcon.

waghanza commented on May 21, 2024

@picatz I don't have that in https://github.com/waghanza/http-benchmark/blob/use_falcon/ruby/sinatra/Dockerfile

from falcon.

picatz commented on May 21, 2024

@waghanza Right, I was wrong to use the rackup to test. I apologize for the confusion.

When I use puma and falcon ~~as they are in the Dockerfiles~~:

Puma

$ puma -p 3000 -e production

Server Software:        
Server Hostname:        0.0.0.0
Server Port:            3000

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      100
Time taken for tests:   1.668 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      860000 bytes
HTML transferred:       0 bytes
Requests per second:    2997.75 [#/sec] (mean)
Time per request:       33.358 [ms] (mean)
Time per request:       0.334 [ms] (mean, across all concurrent requests)
Transfer rate:          503.53 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       5
Processing:     2   33   2.6     33      41
Waiting:        2   32   2.6     32      40
Total:          7   33   2.3     33      42

Percentage of the requests served within a certain time (ms)
  50%     33
  66%     33
  75%     34
  80%     34
  90%     35
  95%     36
  98%     38
  99%     39
 100%     42 (longest request)

Falcon

$ falcon serve -c config.ru --bind tcp://0.0.0.0:3000

Server Software:        falcon/0.18.14
Server Hostname:        0.0.0.0
Server Port:            3000

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      100
Time taken for tests:   1.384 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      1245000 bytes
HTML transferred:       0 bytes
Requests per second:    3613.98 [#/sec] (mean)
Time per request:       27.670 [ms] (mean)
Time per request:       0.277 [ms] (mean, across all concurrent requests)
Transfer rate:          878.79 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.2      1       6
Processing:     1   25  23.4     20     186
Waiting:        1   13  17.6      7     184
Total:          2   27  23.7     21     190

Percentage of the requests served within a certain time (ms)
  50%     21
  66%     26
  75%     31
  80%     34
  90%     46
  95%     62
  98%    137
  99%    145
 100%    190 (longest request)

from falcon.

aemadrid commented on May 21, 2024

@waghanza I got an error in benchmarker because kore.io did not have a language version. I had to change benhmarker around line 74 to:

      if row.has_key?("language") && row.has_key?("version")
        target = Target.new(lang.as_s,
                            framework.as_s,
                            "http://#{link.to_s}", row["version"].to_s,
                            row["language"].to_s)
        targets.push(target)
      end

Here is the error I was getting:

Unhandled exception: Missing hash key: "language" (KeyError)
  from /usr/local/Cellar/crystal/0.26.1_1/src/gc/boehm.cr:71:5 in 'frameworks'
  from tools/src/benchmarker.cr:180:1 in '__crystal_main'
  from /usr/local/Cellar/crystal/0.26.1_1/src/kernel.cr:386:5 in 'main'

from falcon.

waghanza commented on May 21, 2024

@aemadrid Thanks for reminder, I've fix that on the-benchmarker/web-frameworks#481

from falcon.

ioquatix commented on May 21, 2024

Do you mind benchmarking the latest version? There have been quite a few performance enhancements since last time.

from falcon.

ioquatix commented on May 21, 2024

Also at this point I'd be happy for you to merge in falcon support.

from falcon.

noraj commented on May 21, 2024

I hope I am not mistaken here; but I don't believe the setup laid out in the config.ru is correct, and in my testing with rackup it defaulted to webbrick. @ioquatix may know how to do this without changing much.

Note: if I use rackup for the puma example in the repo without changing anything, it uses puma under the hood. So, I imagine this is a point of contention for sure, since this isn't the case for falcon.

@waghanza Right, I was wrong to use the rackup to test. I apologize for the confusion.

You can use this option

-s, --server SERVER      serve using SERVER

rackup -s puma
rackup -s falcon

from falcon.

waghanza commented on May 21, 2024

Maybe a good idea to simplify the project behind, but as I understand using bundle exec rackup is just a wrapper around bundle exec puma or bundle exec falcon

from falcon.

ioquatix commented on May 21, 2024

Actually rackup cannot take full advantage of the server design, so I advise against using it for proper benchmarks.

from falcon.

Benchmarking falcon alongside puma about falcon HOT 20 OPEN

Comments (20)

Puma Perf

Falcon Performance

Puma

Falcon

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent