Comments (5)
@inevity response-time percentiles IMHO are a better indication of customer experience than throughput -- you can have the best throughput in the world but if the app frequently has a 5-second response time from storage, it may be a failure. We want to look at both throughput and response time, and mean response time is sort of irrelevant - what response time can the application count on? However, response-time percentiles are a much noisier metric , so you would have to be less sensitive to changes in this metric in order to use it. Make sense?
Also, I have to look at the arsenal/ link and see if that is up to date, I wasn't using it (maybe I should, I think it involves "jsonnet").
from benchmark-wrapper.
For comparisons, I suggest something like a text based tool like benchmark-comparison (aka Touchstone).
Regarding dashboards, please use https://github.com/cloud-bulldozer/performance-dashboards
from benchmark-wrapper.
The arsenal/ link is not up to date, but the fio percentiles is same as ycsb template from cloud-bulldozer/performance-dashboards which use the jsonnet. Nowdays, the performance-dashboards did not have the fio template.
This is why i ask you for the redhat link to learn what method they use.
If i understand correctly, the performance dashboard is used on the daily monitoring scene, not proper to do benckmark compare.
Do find we can compare percentiles with some confide interval on the internet, but it need another work to do. so just omit it.
from benchmark-wrapper.
@inevity, I posted the JSON for the OCS fio grafana dashboard here, hope this helps. It depends on data sources and variables so you have to fill those in as well. Let me know if you need details or have questions.
inevity> If i understand correctly, the performance dashboard is used on the daily monitoring scene, not proper to do benckmark compare.
That's correct, the performance-dashboard is for looking at the results of a single test, not necessarily comparing two tests.
inevity> Do find we can compare percentiles with some confide interval on the internet, but it need another work to do. so just omit it.
So if you do this, you risk false positives (i.e. pulling the fire alarm for a regression when there was no regression) or false negatives (i.e. concluding that there is no significant difference when in fact there is). But if you monitor the std deviation in your samples, you can get a rough idea of whether this is a problem or not by comparing sum of these to the difference in means. If the sum is << difference in means then you're probably ok to conclude that they are really different. It's just not that precise a method.
from benchmark-wrapper.
Thank your explaination.
from benchmark-wrapper.
Related Issues (20)
- Add CLI Option to Purge Empty Fields in ES
- Determine and Implement Versioning Schema
- ElasticSearch Unified Index HOT 3
- Addition of Collectors and Sample Abstraction Request HOT 2
- Create Signal Protocol HOT 4
- Extend Common Benchmark Fields HOT 4
- [Proposal] Index Environment Metadata for every benchmark HOT 4
- Quay trigger sometimes fails HOT 2
- snafu base image HOT 16
- 2 things I don't like about current cache dropping HOT 1
- backlog of ES results results in failure to get yielded results to ES?
- Inconsistent Benchmark and Quay Image Names HOT 10
- run_snafu does not protect against bad JSON data in certain failures HOT 2
- FIO and SF failed to run HOT 8
- Uperf test fails but does not exit with error HOT 2
- Elasticsearch: credential authentication HOT 5
- centos image resources .repo points to wrong URL
- scale_openshift_wrapper should expose parameters to target machineset from different labels HOT 1
- uperf Docker Image creation fails on ppc64le HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchmark-wrapper.