openebs / performance-benchmark Goto Github PK

Create a kubernetes job specification YAML template which can be updated by interested users to specify the benchmark test image of choice & take the test specification YAML as an argument (in case of the generic benchmark image with user-defined workloads). This job should run the tests on the kubernetes PV created on the desired storage backend and make the results available for visualisation and analysis.

Enable data visualisation of the benchmark test results with appropriate metrics & graphs. While a starting point could be to use simple plotting utilities such as gnuplot, which can derive necessary data (metrics) from the tool-specific log files, the need is to have a single source/webpage with all relevant plots to aid analysis and interpret the results.

This could be achieved using standard tools like grafana with metrics being collected from services like Prometheus

Create a generic benchmark test container image which can take a generic test specification YAML file (which contains workload characteristics, benchmark tool, storage backend) as input, converts that into desired tool-specific config files and runs the test on specified storage backend.

Create test images which can be used to benchmark any containerised storage solution for standard application workloads like (mailserver, database, video streaming etc.,) using workload generation tools like fio & vdbench.

Create automated performance tests that focus on identifying impact of specific workload parameters on the storage solution, such as :

transfer/block size
read/Write ratios
queue depth
data reduction ratios via dedupe, compression
etc..,

These tests must involve taking performance metrics for various values for each of these metrics and enabling visualisation of result data. Typically, these shall be executed and results compared in cases of important changes in the I/O path of the code.

Requirements

• fio workload (dbench - https://dbench.samba.org/)
• Scale from 1 to 2000 container instances, 1-3 replicas each, with detailed results recorded from each individual test
• Entire test itself is easily run with a single kubectl command
• Easily modifiable

Success:

• Single replica, no network overhead less than 10%
• As you add replicas, overhead is linear
• Jitter is managed to a tight, predictable window

The benchmark test suite is required to be run in an automated manner on every major build release on a setup/environment for which baseline numbers are identified.

The results of these periodic benchmark test runs need to be stored in a dedicated webserver with appropriate naming conventions & build details for ready reference