earlephilhower / ezfio Goto Github PK

@earlephilhower, I saw your latest exceedance chart commit and realized that I did a poor job of advertising changes in fio's json+ output format.

http://git.kernel.dk/cgit/fio/commit/?id=129e4193ba3f6e840990fdd5f09c9742bb0dddfa actually changed the json+ output format to drop the indices for the raw clat data structure in favor of the actual latency durations

the recent nsec timing patches only changed the units from usec to nsec

• See the fio bug io reports verify: bad header rand_seed when using zoned random distribution
• Sitsofe recommend steering clear of randrw verification jobs - they are notorious for being difficult.
• The date verify is important,but run well at rand write test
• I used ezfio and found that bug below:

---Sustained Multi-Threaded Random Read Tests by Block Size---                                     
Sustained Multi-Threaded Random Read Tests by Block Size, BS=512            4.36     8,920  85241.6
Sustained Multi-Threaded Random Read Tests by Block Size, BS=1024           8.74     8,947  85069.5
Sustained Multi-Threaded Random Read Tests by Block Size, BS=2048          17.65     9,035  82437.5
Sustained Multi-Threaded Random Read Tests by Block Size, BS=4096          34.71     8,886  85614.8
Sustained Multi-Threaded Random Read Tests by Block Size, BS=8192          68.37     8,751  86823.2
Sustained Multi-Threaded Random Read Tests by Block Size, BS=16384        101.25     6,480 112674.6
Sustained Multi-Threaded Random Read Tests by Block Size, BS=32768        149.72     4,791 155977.1
Sustained Multi-Threaded Random Read Tests by Block Size, BS=65536        213.44     3,415 218972.7
Sustained Multi-Threaded Random Read Tests by Block Size, BS=131072       235.98     1,888 387823.7

---Sequential Write Tests with Queue Depth=1 by Block Size---                                      
Sequential Write Tests with Queue Depth=1 by Block Size, BS=512                   00:00:02         
FIO Error!
fio /tmp/tmp3nliobop --output-format=json+
STDOUT:
fio: pid=26829, err=84/file:io_u.c:2141, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
{
  "fio version" : "fio-3.27",
  "timestamp" : 1624370289,
  "timestamp_ms" : 1624370289118,

....
STDERR:
verify: bad header rand_seed 5513545861456270363, wanted 4503225908322601180 at file /dev/vdd offset 0, length 512 (requested block: offset=0, length=512)

Exception in thread Thread-21:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner

Hi,

I have tested the 970 PRO with both i5 8400 and the 32 Core 2990WX.

Could you please take a look at the results and tell if everyting is OK?

2290WX:
ezfio.zip

8400:

ezfio.zip

Regards

Hi Earle,

Currently, if /proc/sys/fs/aio-nr exeeds /proc/sys/fs/aio-max-nr, the script exits with fio error.

STDOUT:
time     3851  cycles_start=2971940857908
fio: pid=4077, err=11/file:engines/libaio.c:354, func=io_queue_init, error=Resource temporarily unavailable
fio: pid=4095, err=11/file:engines/libaio.c:354, func=io_queue_init, error=Resource temporarily unavailable
3;fio-2.13-28-g059b6;test;0;11;348512528;2904150;726037;120005;2;13707;13.613997;11.054928;1;24050;330.168617;161.437366;1.000000%=151;5.000000%=177;10.000000%=193;20.000000%=217;30.000000%=239;40.000000%=262;50.000000%=286;60.000000%=318;70.000000%=354;80.000000%=414;90.000000%=516;95.000000%=628;99.000000%=956;99.500000%=1112;99.900000%=1480;99.950000%=1640;99.990000%=2064;0%=0;0%=0;0%=0;39;24062;344.682689;161.719782;7363;12980;0.390511%;11341.037492;568.079527;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;2.382052%;7.343719%;87131642;0;8660;100.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.0%;0.01%;0.01%;0.01%;0.01%;0.01%;0.01%;35.12%;53.81%;8.38%;1.88%;0.79%;0.01%;0.01%;0.01%;0.01%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;nvme0n1;86976293;0;0;0;28073776;0;32065504;100.00%

STDERR:
fio: check /proc/sys/fs/aio-max-nr
fio: check /proc/sys/fs/aio-max-nr

Exception in thread Thread-40:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "./ezfio.py", line 632, in JobWrapper
    val = o['cmdline'](o)
  File "./ezfio.py", line 491, in <lambda>
    o['runtime'])})
  File "./ezfio.py", line 427, in RunTest
    raise FIOError(" ".join(cmdline), code, err, out)
FIOError

Sustained 4KB Random Read Tests by Number of Threads, Threads=256          ERROR     ERROR    ERROR
ERROR DETECTED, ABORTING TEST RUN.

Is this something we can skip values ?
Can this be notified to user (ex: with 0s) and proceeded to other runs ?
If yes, can we follow the same approach for all such instances ?

Hey there. Long time no see!

Just wanted to share my WD Black SN750 test.

ezfio.zip

Something about write values are not right. They look like 15K which does not seem normal to me.

The capacity of drive is different from NVMe list.

I use nvme list , the capacity is 6.4TB. The capacity of the same drive is 5961G.
Would you please do me favor to let me know the reason?

[root/ezfio-master]# ./ezfio.py -d /dev/nvme2n1 -u 100 --yes

ezFio test parameters:
Drive: /dev/nvme2n1
Model:
Serial:
AvailCapacity: 5961 GiB
TestedCapacity: 5961 GiB

After running for 9 hours and forgetting to take a screenshot, I clicked on Open Spread Sheet button and the results window disappered. I think it should stay open in the background until I clicked on close.

Hi,
I have found out cpuinfo parsing works differently for ppc64 architecture.
I have a patch ready for this. Will be sending a pull request.

Can I send it directly on the master branch ?

fail on CollectSystemInfo

Dear ezFIO team,

I get following warnings: -
./ezfio.py:1166: DeprecationWarning: isAlive() is deprecated, use is_alive() instead:20:17 ...
while job.isAlive():

I use Ubuntu 20.04.2 LTS (GNU/Linux 5.8.0-50-generic x86_64)

The tests that execute the multi-threaded Sequential Read and Random Read tests use asymmetric workloads (1 thread, 256 queue depth i the former case and 16 threads and 16 queue depth in the later). Is there a reason for this asymmetry:

testname = "Sustained Multi-Threaded Sequential Read Tests by Block Size"
seqrand = "Seq"
wmix=0
threads=1
runtime=shorttime
iops_log=False
iodepth=256
AddTestBSShmoo()

testname = "Sustained Multi-Threaded Random Read Tests by Block Size"
seqrand = "Rand"
wmix=0
threads=16
runtime=shorttime
iops_log=False
iodepth=16
AddTestBSShmoo()

• see that ods page

• For example Sustained Random 4K Mixed (70:30),the data belows:

• So Abscissa should be thread

Sometimes people only copy the .PS1/.BAT/.Py files and not the entire tree, missing the original.ods template which is required to actually build the final spreadsheet.

Should check this file exists before beginning the run, so that we don't have people waste several hours on testing only to fail out with a missing file error.

Running on a newer AMD server and FIO has some problem with affiinity. The result are lines written to stdout with the format:

CPU mask cannot bind CPUs (e.g. 64, 191) that are in different processor groups
CPU mask cannot bind CPUs (e.g. 64, 191) that are in different processor groups

This comes after the fio cmdline is written to the output file and before the fio jscon. The result if an error as ConvertFrom-Json sees improperly formatted text.

I fixed this in exfio.ps1 by adding

$LineSkip=0
foreach ($line in Get-Content $testfile) { if ($line -match '^{') {break} else {$LineSkip++} }

$j = ConvertFrom-Json "$(Get-Content $testfile | select -Skip $LineSkip)"

Around line 695. Instead of just skip 1, skip to the open json brace .

• See the fio bug io reports verify: bad header rand_seed when using zoned random distribution
• Sitsofe recommend steering clear of randrw verification jobs - they are notorious for being difficult.
• The date verify is important,but not run well at rand write test
• I used ezfio and found that bug below:

---Sustained Multi-Threaded Random Read Tests by Block Size---                                     
Sustained Multi-Threaded Random Read Tests by Block Size, BS=512            4.36     8,920  85241.6
Sustained Multi-Threaded Random Read Tests by Block Size, BS=1024           8.74     8,947  85069.5
Sustained Multi-Threaded Random Read Tests by Block Size, BS=2048          17.65     9,035  82437.5
Sustained Multi-Threaded Random Read Tests by Block Size, BS=4096          34.71     8,886  85614.8
Sustained Multi-Threaded Random Read Tests by Block Size, BS=8192          68.37     8,751  86823.2
Sustained Multi-Threaded Random Read Tests by Block Size, BS=16384        101.25     6,480 112674.6
Sustained Multi-Threaded Random Read Tests by Block Size, BS=32768        149.72     4,791 155977.1
Sustained Multi-Threaded Random Read Tests by Block Size, BS=65536        213.44     3,415 218972.7
Sustained Multi-Threaded Random Read Tests by Block Size, BS=131072       235.98     1,888 387823.7

---Sequential Write Tests with Queue Depth=1 by Block Size---                                      
Sequential Write Tests with Queue Depth=1 by Block Size, BS=512                   00:00:02         
FIO Error!
fio /tmp/tmp3nliobop --output-format=json+
STDOUT:
fio: pid=26829, err=84/file:io_u.c:2141, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character
{
  "fio version" : "fio-3.27",
  "timestamp" : 1624370289,
  "timestamp_ms" : 1624370289118,

....
STDERR:
verify: bad header rand_seed 5513545861456270363, wanted 4503225908322601180 at file /dev/vdd offset 0, length 512 (requested block: offset=0, length=512)

Exception in thread Thread-21:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner

(https://github.com/earlephilhower/ezfio/files/2908590/ezfio_results_28GB_4cores_3900MHz_nvme0n1_nvmeof_2019-02-26_17-31-20.ods.gz)

Why long term performance stability show as 0 iops?

Thanks.

ezFio test parameters:

           Drive: /dev/nvme0n1
           Model: SAMSUNG MZPLL1T6HEHP-00003
          Serial: S3HBNA0K303021
   AvailCapacity: 26 GiB
  TestedCapacity: 13 GiB
             CPU: Intel Core i3-7100 CPU @ 3.90GHz
           Cores: 4
       Frequency: 3900
     FIO Version: fio-3.8

Test Description BW(MB/s) IOPS Lat(us)

---Sequential Preconditioning---
Sequential Preconditioning Pass 1 DONE DONE DONE
Sequential Preconditioning Pass 2 DONE DONE DONE

---Sustained Multi-Threaded Sequential Read Tests by Block Size---
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=512 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=1024 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=2048 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=4096 1,049.27 268,613 952.7
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=8192 2,080.76 266,337 960.9
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=16384 3,757.09 240,453 1064.3
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=32768 5,852.74 187,288 1366.6
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=65536 5,845.64 93,530 2736.8
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=131072 5,835.88 46,687 5482.7

---Sustained Multi-Threaded Random Read Tests by Block Size---
Sustained Multi-Threaded Random Read Tests by Block Size, BS=512 0.00 0 0.0
Sustained Multi-Threaded Random Read Tests by Block Size, BS=1024 0.00 0 0.0
Sustained Multi-Threaded Random Read Tests by Block Size, BS=2048 0.00 0 0.0
Sustained Multi-Threaded Random Read Tests by Block Size, BS=4096 2,932.19 750,640 338.7
Sustained Multi-Threaded Random Read Tests by Block Size, BS=8192 5,331.03 682,372 373.1
Sustained Multi-Threaded Random Read Tests by Block Size, BS=16384 3,697.88 236,664 1080.0
Sustained Multi-Threaded Random Read Tests by Block Size, BS=32768 5,493.61 175,796 1455.6
Sustained Multi-Threaded Random Read Tests by Block Size, BS=65536 5,455.25 87,284 2931.9
Sustained Multi-Threaded Random Read Tests by Block Size, BS=131072 5,574.52 44,596 5739.5

---Sequential Write Tests with Queue Depth=1 by Block Size---
Sequential Write Tests with Queue Depth=1 by Block Size, BS=512 0.00 0 0.0
Sequential Write Tests with Queue Depth=1 by Block Size, BS=1024 0.00 0 0.0
Sequential Write Tests with Queue Depth=1 by Block Size, BS=2048 0.00 0 0.0
Sequential Write Tests with Queue Depth=1 by Block Size, BS=4096 237.95 60,915 16.0
Sequential Write Tests with Queue Depth=1 by Block Size, BS=8192 424.28 54,307 17.9
Sequential Write Tests with Queue Depth=1 by Block Size, BS=16384 684.19 43,788 22.3
Sequential Write Tests with Queue Depth=1 by Block Size, BS=32768 1,021.94 32,702 29.9
Sequential Write Tests with Queue Depth=1 by Block Size, BS=65536 1,416.31 22,661 43.1
Sequential Write Tests with Queue Depth=1 by Block Size, BS=131072 1,678.60 13,429 73.2

---Random Preconditioning---
Random Preconditioning DONE DONE DONE
Random Preconditioning DONE DONE DONE

---Sustained 4KB Random Read Tests by Number of Threads---
Sustained 4KB Random Read Tests by Number of Threads, Threads=1 43.63 11,169 88.4
Sustained 4KB Random Read Tests by Number of Threads, Threads=2 90.98 23,290 85.0
Sustained 4KB Random Read Tests by Number of Threads, Threads=4 169.53 43,398 90.9
Sustained 4KB Random Read Tests by Number of Threads, Threads=8 381.15 97,575 81.5
Sustained 4KB Random Read Tests by Number of Threads, Threads=16 730.29 186,955 85.1
Sustained 4KB Random Read Tests by Number of Threads, Threads=32 1,317.28 337,225 94.3
Sustained 4KB Random Read Tests by Number of Threads, Threads=64 2,010.83 514,773 122.5
Sustained 4KB Random Read Tests by Number of Threads, Threads=128 1,992.69 510,128 241.8
Sustained 4KB Random Read Tests by Number of Threads, Threads=256 1,553.91 397,800 607.0

---Sustained 4KB Random mixed 30% Write Tests by Threads---
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=1 60.32 15,443 85.1
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=2 112.98 28,923 90.3
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=4 173.27 44,358 112.6
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=8 298.74 76,478 129.5
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=16 536.49 137,342 147.2
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=32 825.57 211,345 192.3
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=64 1,143.85 292,825 283.4
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=128 1,458.41 373,354 447.0
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=256 1,496.10 383,002 781.4

---Sustained Perf Stability Test - 4KB Random 30% Write---
Sustained Perf Stability Test - 4KB Random 30% Write 1,490.16 381,480 783.5

---Sustained 4KB Random Write Tests by Number of Threads---
Sustained 4KB Random Write Tests by Number of Threads, Threads=1 231.21 59,191 16.4
Sustained 4KB Random Write Tests by Number of Threads, Threads=2 521.60 133,529 14.6
Sustained 4KB Random Write Tests by Number of Threads, Threads=4 897.79 229,833 17.1
Sustained 4KB Random Write Tests by Number of Threads, Threads=8 1,292.87 330,975 23.7
Sustained 4KB Random Write Tests by Number of Threads, Threads=16 1,602.42 410,219 38.5
Sustained 4KB Random Write Tests by Number of Threads, Threads=32 1,766.20 452,147 70.3
Sustained 4KB Random Write Tests by Number of Threads, Threads=64 1,730.65 443,047 143.8
Sustained 4KB Random Write Tests by Number of Threads, Threads=128 1,757.94 450,033 283.5
Sustained 4KB Random Write Tests by Number of Threads, Threads=256 1,574.54 403,082 633.4

---Sustained Multi-Threaded Random Write Tests by Block Size---
Sustained Multi-Threaded Random Write Tests by Block Size, BS=512 0.00 0 0.0
Sustained Multi-Threaded Random Write Tests by Block Size, BS=1024 0.00 0 0.0
Sustained Multi-Threaded Random Write Tests by Block Size, BS=2048 0.00 0 0.0
Sustained Multi-Threaded Random Write Tests by Block Size, BS=4096 1,877.82 480,723 531.8
Sustained Multi-Threaded Random Write Tests by Block Size, BS=8192 2,092.03 267,780 954.9
Sustained Multi-Threaded Random Write Tests by Block Size, BS=16384 2,084.45 133,405 1916.7
Sustained Multi-Threaded Random Write Tests by Block Size, BS=32768 2,094.95 67,039 3816.3
Sustained Multi-Threaded Random Write Tests by Block Size, BS=65536 2,089.59 33,433 7653.8
Sustained Multi-Threaded Random Write Tests by Block Size, BS=131072 2,098.95 16,792 15241.2

COMPLETED!
Spreadsheet file: /home/shu/src/ezfio/ezfio_results_28GB_4cores_3900MHz_nvme0n1_nvmeof_2019-02-26_17-31-20.ods

I don't know if this should be filled or not but the sustained 4K random write table is empty:

The files are in this attachment, this is the same file referred in #9.

This was used to parse the ezfio output csv files one by one, in order to generate the .ods files found in the linux dir within the first attachment.

The runs were executed in an ubuntu server 16.04 install , with the following arguments:
/usr/bin/python $HOME/script/.fio/ezfio.py -d /dev/sdb -o $OUT_DIR --yes

Even though Set-ExecutionPolicy -scope CurrentUser RemoteSigned command is applied, I kept getting this error:

./ezfio : File C:\ezfio\ezfio.ps1 cannot be loaded. The file C:\ezfio\ezfio.ps1 is not digitally signed. You cannot run
this script on the current system. For more information about running scripts and setting execution policy, see about_
Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1

• ./ezfio

•   + CategoryInfo          : SecurityError: (:) [], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess
  

The solution was to use this command:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

Regards

Hi,

We are having an issue where the .ods file is missing the Long Term Performance Stability Graph and the Timeseries tab has the computer information but is missing all the IOPS numbers. Everything else is working.

We are Running the test on Windows Server 2016 with PowerShell 5.1 and FIO 3.12.
Any help would be appreciated.

In original.ods template, Sustained 4K Read/Write chart x-axis labels are "Queue Depth". However, the data they point varies by Number of Threads so the charts seem to be wrongly labelled.

Yo Dr Earle!

Using fio 3.1 with your newest ezfio (but with longtime and shorttime set to 5 seconds (just to validate that everything is working. The test is also run with -u 1. This is just a sea trial!

It also failed using a ezfio version from last fall.

Is there a preferred version of fio?

Advice and counsel?

Thanks Earle!

---Random Preconditioning---
Random Preconditioning DONE DONE DONE
Random Preconditioning DONE DONE DONE

---Sustained 4KB Random Read Tests by Number of Threads---
Sustained 4KB Random Read Tests by Number of Threads, Threads=1 43.53 11,143 88.6
Sustained 4KB Random Read Tests by Number of Threads, Threads=2 87.76 22,467 88.0
Sustained 4KB Random Read Tests by Number of Threads, Threads=4 177.59 45,462 87.3
Sustained 4KB Random Read Tests by Number of Threads, Threads=8 356.54 91,274 87.0
Sustained 4KB Random Read Tests by Number of Threads, Threads=16 709.67 181,676 87.5
Sustained 4KB Random Read Tests by Number of Threads, Threads=32 1,347.17 344,877 92.2
Sustained 4KB Random Read Tests by Number of Threads, Threads=64 2,442.68 625,325 101.6
Sustained 4KB Random Read Tests by Number of Threads, Threads=128 3,054.59 781,974 162.6
Sustained 4KB Random Read Tests by Number of Threads, Threads=256 3,053.83 781,782 325.3

---Sustained 4KB Random mixed 30% Write Tests by Threads---
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=1 54.96 14,071 88.8
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=2 113.50 29,056 87.0
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=4 225.70 57,779 87.9
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=8 438.29 112,203 90.5
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=16 818.62 209,568 97.1
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=32 1,488.00 380,927 107.1
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=64 2,474.59 633,495 129.0
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=128 3,546.50 907,904 178.1
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=256 3,926.65 1,005,223 319.8

---Sustained Perf Stability Test - 4KB Random 30% Write---
Sustained Perf Stability Test - 4KB Random 30% Write Runtime 00:00:17 ...
FIO Error!
fio /tmp/tmpRtscjI --output-format=json+
STDOUT:

STDERR:
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion `!trylock' failed.

Exception in thread Thread-50:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File "./fast5.ezfio.py", line 1085, in JobWrapper
val = o'cmdline'
File "./fast5.ezfio.py", line 939, in
o['runtime'])})
File "./fast5.ezfio.py", line 832, in RunTest
raise FIOError(" ".join(cmdline), code, err, out)
FIOError

Sustained Perf Stability Test - 4KB Random 30% Write ERROR ERROR ERROR
ERROR DETECTED, ABORTING TEST RUN.
You have new mail in /var/spool/mail/root
fstsmtn10rt-1 ezfio-master #

Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=512 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=1024 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=2048 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=4096 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=8192 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=16384 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=32768 0.00 0 0.0

I've tried running ezfio.py on a linux system with an Intel NVMe. But it didn't output anything for blocksizes from 512-32k. Anything i'm doing wrong?

Hi Earle,

I've performed the latest run on the SSD with the latest ezfio and v3.3 FIO.

The test was done but one one of the entries is still showing as runnig, FYI:

I'm also sharing the ZIP so that you can investigate.

ezfio-master.zip

Regards

Trying to launch via BAT file, I get this:

W10 1703 x64

Last night, I used ezFIO to test two Intel DC P3700 U.2 NVMe SSDs, one 800GB, the other 1.6TB. The test session finished fine for both, but the generated ods files for each both showed incorrect Long Term Performance Stability plot. There is no line plot whatsoever, although the following statistics are there:

Average IOPS
227,896 IOPS

Standard Deviation
4,132.9 IOPS

Coefficient of Variation
2%

Environment:

• Drive /dev/nvme0n1
• Tested 745GB
• Model NVMEe INTEL SSDPE2MD80 0171
• Serial CVFT42040AP800HGN
• CPU Intel Xeon CPU E5-2620 v3 @ 2.4Ghz
• OS Linux fs00 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64
• FIO version fio-2.12

Please let me what I can do to get the plot generated correctly.

Regards,

-- Chin

Hello,

Thanks for the script, I've been figuring out fio cmdline parameters to troubleshoot some performance issues so this comes in handy and takes some work out of my hands.

Can you give me any indication how long a run should take? Any reference? I need to test 4 256GB NVME drives and test them on 2 systems, Asus ROG laptop with gen7 Core i7 on Windows 10 and a HPe ML10 Gen9 Xeon e3-v5 on Arch Linux (using PCE-E 3.0 x4 expansion card).

Should Windows results be comparable to Linux results?

Final Q I have is you mention issues with Bluestop FIO version 3.1, their latest is now 3.12. Are there any known issues running ezfio with this version?

Thank you.

See #56

Just wanted to share my 6:45 hours of 970 PRO tests. Anything wrong?

ezfio-master.zip

Earle Hi,

we have performed a test on Intel 760P 512 GB M.2 SSD. You can check the results if everything is OK.

Thanks.

details_512GB_12cores_3696MHz_PHYSICALDRIVE1_DESKTOP-FC2A4TG_2018-04-05_14-42-15.zip

[root@jaguar54 ezfio-master]# ./ezfio.py -d /dev/nvme1n1 -u 100
Install sdparm to allow model/serial extraction. Continuing.

WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING!
THIS TEST WILL DESTROY ANY DATA AND FILESYSTEMS ON /dev/nvme1n1

Please type the word "yes" and hit return to continue, or anything else to abort.
yes

ezFio test parameters:

           Drive: /dev/nvme1n1
           Model: UNKNOWN
          Serial: UNKNOWN
   AvailCapacity: 2980 GiB
  TestedCapacity: 2980 GiB
             CPU: Intel Xeon CPU E3-1246 v3 @ 3.50GHz
           Cores: 8
       Frequency: 3500

Test Description BW(MB/s) IOPS Lat(us)

---Sequential Preconditioning---
Sequential Preconditioning Pass 1 DONE DONE DONE
Sequential Preconditioning Pass 2 DONE DONE DONE

---Sustained Multi-Threaded Sequential Read Tests by Block Size---
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=512 00:02:00
Unexpected error while running FIO job.
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File "./ezfio.py", line 732, in JobWrapper
val = o'cmdline'
File "./ezfio.py", line 591, in
o['runtime'])})
File "./ezfio.py", line 526, in RunTest
j = json.loads(out)
File "/usr/lib64/python2.7/json/init.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=512 ERROR ERROR ERROR
ERROR DETECTED, ABORTING TEST RUN.
[root@jaguar54 ezfio-master]# nvme list

• see the ezfio.py inf line 214:

    # For cluster mode, we add a new physDriveList dict and fake physDrive
    if cluster:
        nodes = physDrive.split(",")
        for node in nodes:
            print(node)
            physDriveDict[node.split(":")[0]] = node.split(":")[1]
        physDrive = nodes[0].split(":")[1]

• see the readme

./ezfio.py --cluster --drive host1:/dev/dr1,host2:/dev/dr2/... ...

• in the multy disk condition:

./ezfio.py --cluster --drive host1:/dev/dr1,/dev/dr2,/dev/dr3,/dev/dr3,host2:/dev/dr2/,/dev/dr3,/dev/dr4,/dev/dr5,/dev/dr6

• It can't split string bitween disk and node in one word ","
• change is like that:

        nodes = physDrive.split("?")

./ezfio.py --cluster --drive host1:/dev/dr1?host2:/dev/dr2/... ...

Hi,

I just wanted to share my WD SN850 test results.

ezfio-master.zip

I believe some info here is also mssing:

[root@localhost ezfio]# ./ezfio.py -d /dev/nvme0n1 -u 1 --yes

ezFio test parameters:

           Drive: /dev/nvme0n1
           Model: STORESWIFT SSD2021
          Serial: APOLLO768540
   AvailCapacity: 2673 GiB
  TestedCapacity: 26 GiB
    TestedOffset: 0 GiB
             CPU: Intel Core i5-9400 CPU @ 2.90GHz
           Cores: 6
       Frequency: 2871
     FIO Version: fio-3.7

Test Description BW(MB/s) IOPS Lat(us)

---Sequential Preconditioning---
Sequential Preconditioning Pass 1 DONE DONE DONE
Sequential Preconditioning Pass 2 DONE DONE DONE

---Sustained Multi-Threaded Sequential Read Tests by Block Size---
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=512 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=1024 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=2048 0.00 0 0.0
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=4096 743.83 190,421 1344.1
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=8192 2,904.20 371,738 688.4
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=16384 4,490.70 287,405 890.4
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=32768 5,719.53 183,025 1398.4
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=65536 5,719.20 91,507 2797.3
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=131072 5,712.87 45,703 5601.0

---Sustained Multi-Threaded Random Read Tests by Block Size---
Sustained Multi-Threaded Random Read Tests by Block Size, BS=512 0.00 0 0.0
Sustained Multi-Threaded Random Read Tests by Block Size, BS=1024 0.00 0 0.0
Sustained Multi-Threaded Random Read Tests by Block Size, BS=2048 0.00 0 0.0
Sustained Multi-Threaded Random Read Tests by Block Size, BS=4096 3,856.63 987,297 258.7
Sustained Multi-Threaded Random Read Tests by Block Size, BS=8192 4,606.87 589,680 433.6
Sustained Multi-Threaded Random Read Tests by Block Size, BS=16384 5,340.78 341,810 748.3
Sustained Multi-Threaded Random Read Tests by Block Size, BS=32768 5,589.60 178,867 1430.7
Sustained Multi-Threaded Random Read Tests by Block Size, BS=65536 5,545.69 88,731 2884.6
Sustained Multi-Threaded Random Read Tests by Block Size, BS=131072 5,245.94 41,967 6099.4

---Sequential Write Tests with Queue Depth=1 by Block Size---
Sequential Write Tests with Queue Depth=1 by Block Size, BS=512 0.00 0 0.0
Sequential Write Tests with Queue Depth=1 by Block Size, BS=1024 0.00 0 0.0
Sequential Write Tests with Queue Depth=1 by Block Size, BS=2048 0.00 0 0.0
Sequential Write Tests with Queue Depth=1 by Block Size, BS=4096 491.34 125,784 7.6
Sequential Write Tests with Queue Depth=1 by Block Size, BS=8192 866.33 110,890 8.7
Sequential Write Tests with Queue Depth=1 by Block Size, BS=16384 1,322.93 84,668 11.4
Sequential Write Tests with Queue Depth=1 by Block Size, BS=32768 1,752.69 56,086 17.3
Sequential Write Tests with Queue Depth=1 by Block Size, BS=65536 2,015.75 32,252 30.2
Sequential Write Tests with Queue Depth=1 by Block Size, BS=131072 2,191.73 17,534 55.9

---Random Preconditioning---
Random Preconditioning DONE DONE DONE
Random Preconditioning DONE DONE DONE

---Sustained 4KB Random Read Tests by Number of Threads---
Sustained 4KB Random Read Tests by Number of Threads, Threads=1 44.62 11,422 86.0
Sustained 4KB Random Read Tests by Number of Threads, Threads=2 88.15 22,566 87.0
Sustained 4KB Random Read Tests by Number of Threads, Threads=4 174.25 44,609 88.0
Sustained 4KB Random Read Tests by Number of Threads, Threads=8 399.61 102,301 77.6
Sustained 4KB Random Read Tests by Number of Threads, Threads=16 793.57 203,155 78.4
Sustained 4KB Random Read Tests by Number of Threads, Threads=32 1,425.48 364,922 87.1
Sustained 4KB Random Read Tests by Number of Threads, Threads=64 2,392.79 612,554 104.0
Sustained 4KB Random Read Tests by Number of Threads, Threads=128 3,387.25 867,137 147.0
Sustained 4KB Random Read Tests by Number of Threads, Threads=256 3,656.47 936,056 272.9

---Sustained 4KB Random mixed 30% Write Tests by Threads---
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=1 58.12 14,879 85.6
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=2 113.17 28,971 87.8
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=4 214.96 55,029 92.5
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=8 475.34 121,688 89.6
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=16 808.11 206,876 106.3
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=32 1,274.10 326,169 135.6
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=64 1,877.73 480,699 183.5
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=128 2,582.30 661,068 270.3
Sustained 4KB Random mixed 30% Write Tests by Threads, Threads=256 3,440.67 880,811 401.3

---Sustained Perf Stability Test - 4KB Random 30% Write---
Sustained Perf Stability Test - 4KB Random 30% Write Runtime 00:20:59 ...
FIO Error!
fio /tmp/tmp_sbpfjk0 --output-format=json+
STDOUT:

STDERR:
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion `!trylock' failed.

Exception in thread Thread-50:
Traceback (most recent call last):
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "./ezfio.py", line 1129, in JobWrapper
val = o'cmdline'
File "./ezfio.py", line 983, in
o['runtime'])})
File "./ezfio.py", line 871, in RunTest
raise FIOError(" ".join(cmdline), code, err, out)
FIOError

Sustained Perf Stability Test - 4KB Random 30% Write ERROR ERROR ERROR
ERROR DETECTED, ABORTING TEST RUN.

Hi,

I'm running ezFio on windows 10 using fio-3.16 (source : https://bsdio.com/fio/ ). I'm consistently seeing this error related to !trylock. Any suggestion as to how to fix it for Windows ?

ezFio test parameters:
Drive : \.\PhysicalDrive0
Model : Samsung SSD 970 EVO Plus 250GB
Serial : 0025_385C_XXXX_XXXX.
AvailCapacity : 233 GiB
TestedCapacity : 35 GiB
CPU : Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz
Cores : 8
Frequency : 3501
FIO Version : fio-3.16

Test Description BW(MB/s) IOPS Lat(us)

Sequential Preconditioning DONE DONE DONE
Sequential Preconditioning DONE DONE DONE
---Sustained Multi-Threaded Sequential Read Tests by Block Size---
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=512 44.2 90469 1425.6
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=1024 91.4 93633 1375.7
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=2048 194.8 99752 1290.5
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=4096 377.4 96625 1334.3
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=8192 746.7 95578 1347.5
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=16384 1474.1 94343 1367.8
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=32768 2687.3 85992 2101.4
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=65536 2583.2 41331 6189.9
Sustained Multi-Threaded Sequential Read Tests by Block Size, BS=131072 2508.9 20071 12751.6
Program: C:\Program Files\fio\fio.exe
File: filelock.c, Line 182

Expression: !trylock
Assertion failed!

Program: C:\Program Files\fio\fio.exe
File: filelock.c, Line 182

Expression: !trylock
Assertion failed!

Program: C:\Program Files\fio\fio.exe
File: filelock.c, Line 182

Expression: !trylock
Assertion failed!

Program: C:\Program Files\fio\fio.exe
File: filelock.c, Line 182

Expression: !trylock
Assertion failed!

The link inside script that goes to Bluestop.org is non-HTTPS and it does not work. FYI.

Hi,

when you right click the batch file and run it as admin, PowerShell script is not called from the current path but rahter called from C:\Windows\System32. And this lead to nothing, as the script is not there.

I was unable to run ezFIO and was wondering what was the problem. Finally, after removing the hidden attribute from the batch file and recoding the screen, I was able to capture this:

I have copied all the files into System32 and it was working just fine after that moment.

OS: Windows 10 Build 1607

Hi,

Trying to run benchmark test against an SMB share mounted on Windows 2019 server. Does that scenario works with ezfio?

Thanks,
Virgil

Hello,
After running ezfio.py there was no exceedance data to look at or graph. Other data and values are present. No obvious errors or warnings in stdout.
ezfio_exceedance_exceedance30.csv was generated without any data in it. After browsing through the code, it seems some *.exc.read/write.csv files should have been created but are not present.

# cat ezfio_exceedance_exceedance30.csv
Drive,/dev/nvme0n1
Model,WUS4BA1A1DSP3X1
Serial,A06893D5
AvailCapacity,14306,GiB
TestedCapacity,14306,GiB
CPU,Intel Xeon Gold 6154 CPU @ 3.00GHz
Cores,72
Frequency,3000
OS,Linux fstlnsr650-8.fst.local 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64
FIOVersion,fio-3.7
QD1 Read Exceedance,,QD1 Write Exceedance,,,QD4 Read Exceedance,,QD4 Write Exceedance,,,QD16 Read Exceedance,,QD16 Write Exceedance,,,QD32 Read Exceedance,,QD32 Write Exceedance,,,
rdusec,rdpct,wrusec,wrpct,,rdusec,rdpct,wrusec,wrpct,,rdusec,rdpct,wrusec,wrpct,,rdusec,rdpct,wrusec,wrpct,,
,,,,,,,,,,,,,,,,,,,,

Command line:
./ezfio.py -d /dev/nvme0n1 --yes

Environment:

• CentOS Linux release 8.2.2004 (Core)
• Linux fstlnsr650-8.fst.local 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64
• Python 3.6.8
• fio-3.7
• ezfio commit 79909e6

Please let me know if I can provide more information.
Best Regards,
Calvin

fio can paint
ezfio also should do
some example:

many test no only to process
it perf down to 80%,is fail
and we need 24h or more test
save my eyes..

I am reading the "ezfio.py" file in the lines between 1013- 1020, the name and print info is "Multi-thread", but the thread parameter is 1. Is this a typo?
BTW, how about displaying the exact thread number instead of "multi-Threaded"? I guess most of the testers want to know the test parameter?

testname = "Sustained Multi-Threaded Sequential Read Tests by Block Size"
seqrand = "Seq"
wmix = 0
threads = 1
runtime = shorttime
iops_log = False
iodepth = 256
AddTestBSShmoo()

No idea what happen. Are there any tools I miss prior to run the script?

---Sustained Perf Stability Test - 4KB Random 30% Write---
Sustained Perf Stability Test - 4KB Random 30% Write 00:20:10
FIO Error!
fio /tmp/tmpgqkch6qd --output-format=json+
STDOUT:

STDERR:
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed. fio: filelock.c:182: __fio_lock_file: Assertion !trylock' failed.
fio: filelock.c:182: __fio_lock_file: Assertion `!trylock' failed.

Hi Earle,

I tried ezFIO on PCI NVMe adapters, which are formatted with 4k sector.
So, fio reports error on these devices for bs less than 4096.

Is this something that should be changed in the bs values in the ezfio script ?

I am reading the code. In the function of SetupFiles, original.ods is a must. The script will exit if the original.ods file does NOT exist.
Could u pls let me know what is the function of original.ods?

Hi,

here is one my latest runs (took about 30 hours I think). Sustained 4K Random Read & Write Tables look empty. Could you please check?

Regards

ezfio-master.zip

Hi There,
Would suggest adding AvailCapacity, TestedCapacity, TestedOffset in GB in RunAllTests function. That would be compatible with "NVMe list" Command. NVMe List adopts GB instead of GiB. I would like to submit a patch if you agree.

print(fmtinfo.format("AvailCapacity", str(physDriveGiB) + " GiB"))
print(fmtinfo.format("TestedCapacity", str(testcapacity) + " GiB"))
print(fmtinfo.format("TestedOffset", str(testoffset) + " GiB"))

Best，
Robert Q

The offset line in the CSV shifts the resulting CSV down by 1 row, but the ODS was not updated to the new locations.

Either fix the ODS to look 1 row down for all test results, or make the offset part of the size line (different column)