Hi there,
First of all, thanks for you work on this tool. I've written my own script for benchmarking storage on Kubernetes : ksb.
My results are very differents from kubestr, so I tried to search where was the diff between our tests.
With kubestr cli, i've got between 734 and 849 IOPS in 3 differents run, here is the raw result of the best one :
> ./kubestr fio -s rook-ceph-block
PVC created kubestr-fio-pvc-g8gwx
Pod created kubestr-fio-pod-7tspk
Running FIO test (default-fio) on StorageClass (rook-ceph-block) with a PVC of Size (100Gi)
Elapsed time- 1m23.539474031s
FIO test results:
FIO version - fio-3.20
Global options - ioengine=libaio verify=0 direct=1 gtod_reduce=1
JobName: read_iops
blocksize=4K filesize=2G iodepth=64 rw=randread
read:
IOPS=849.953735 BW(KiB/s)=3416
iops: min=600 max=1053 avg=854.266663
bw(KiB/s): min=2400 max=4215 avg=3418.100098
As we can see in kubestr output, it used fio-3.20 with parameters :
- ioengine=libaio
- verify=0
- direct=1
- gtod_reduce=1
- blocksize=4K
- filesize=2G
- iodepth=64
- rw=randread
My Read IOPS benchmark is using exactly same parameters, but adding --time_based --ramp_time=2s --runtime=15s parameters. Results of 3 differents runs are between 10.6k IOPS and 12.9k IOPS. Here is the raw result of one run:
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
read_iops: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=33.2MiB/s][r=8486 IOPS][eta 00m:00s]
read_iops: (groupid=0, jobs=1): err= 0: pid=29: Wed Apr 21 10:46:22 2021
read: IOPS=12.9k, BW=50.4MiB/s (52.9MB/s)(757MiB/15012msec)
bw ( KiB/s): min=21904, max=98400, per=100.00%, avg=52198.45, stdev=19676.11, samples=29
iops : min= 5476, max=24600, avg=13049.48, stdev=4919.07, samples=29
cpu : usr=2.58%, sys=6.54%, ctx=203273, majf=0, minf=58
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=193816,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=50.4MiB/s (52.9MB/s), 50.4MiB/s-50.4MiB/s (52.9MB/s-52.9MB/s), io=757MiB (794MB), run=15012-15012msec
Disk stats (read/write):
rbd6: ios=212839/23, merge=0/3, ticks=769574/599, in_queue=567176, util=96.57%
Both docker images are based on alpine (ghcr.io/kastenhq/kubestr:latest and infrabuilder/iobench). So it does not comes from base OS difference. Anyway, my image is shipped with fio-3.25, and yours contains fio-3.20. To eliminate the image difference hyptohesis, I started a pod mounting a PVC from the exact same storageClass mounted on /root, using your image (ghcr.io/kastenhq/kubestr:latest) and overriding entrypoint with /bin/sh to let me do exec command in it. Here is the result of a manually launched fio command :
# fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=read_iops --filename=/root/fiotest --bs=4K --iodepth=64 --size=2G --readwrite=randread
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.20
Starting 1 process
read_iops: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [r(1)][100.0%][r=60.2MiB/s][r=15.4k IOPS][eta 00m:00s]
read_iops: (groupid=0, jobs=1): err= 0: pid=35: Wed Apr 21 10:25:15 2021
read: IOPS=10.8k, BW=42.4MiB/s (44.4MB/s)(2048MiB/48355msec)
bw ( KiB/s): min=12665, max=99328, per=99.89%, avg=43319.69, stdev=21317.01, samples=96
iops : min= 3166, max=24832, avg=10829.68, stdev=5329.23, samples=96
cpu : usr=1.70%, sys=5.80%, ctx=555542, majf=0, minf=71
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=42.4MiB/s (44.4MB/s), 42.4MiB/s-42.4MiB/s (44.4MB/s-44.4MB/s), io=2048MiB (2147MB), run=48355-48355msec
Disk stats (read/write):
rbd5: ios=519717/145, merge=0/27, ticks=2344134/3451, in_queue=1808204, util=95.04%
Here the result is very similar to the one I got on my benchmark : 10,8k IOPS (I've run it multiple times, results are between 9729 and 10.8k IOPS)
So to sum up :
- kubestr default : 734 to 849 IOPS (stddev unknown)
- ksb default : 10,6k to 12.9k IOPS (stddev 4919)
- kubestr image with manual fio command : 9729 to 10.8k IOPS (stddev 5329)
If my benchmark and in your image with manual command, the stddev is very high (about 5k) due to the fact that it is a used cluster and not a lab isolated dedicated to the bench, but even wih this deviation there is still a large gap between kubestr results and fio results. I may test in isolated lab, as I do for CNI benchmark, but for now I lack of time :)
Can you explain why kubestr and fio commands results differs so much, even with same image ?
Thanks.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent