Comments (6)
piyush286
commented on July 30, 2024
Design for Aggregating Perf Data
For perf, we'll have 2 kinds of builds: Parent and Child Builds. We'll be doing similar aggregating for both of them with minor differences.
Child Build
For each child build, we'll aggregate the data of all iterations run inside that Jenkins build. We aggregate only the good data and ignore the iterations for which values might be null.
Raw Data for Child:
"testData" : {
"metrics" : [
{
"name" : "Footprint in kb",
"value" : [
167268,
168804,
169748,
168884,
167272,
167888,
169628,
165008
]
},
{
"name" : "Startup time in ms",
"value" : [
4207,
4298,
4112,
4161,
4575,
4213,
4606,
4406
]
}
],
"javaVersion" : "<javaVersion>",
"jdkBuildDateUnixTime" : 1554206400000.0
}
Aggregated Data Structure for Child
"aggregateInfo" : [
{
"benchmarkName" : "LibertyStartupDT",
"benchmarkVariant" : "17dev-4way-0-256-qs",
"benchmarkProduct" : "Build-20190402_03",
"metrics" : [
{
"name" : "Footprint in kb",
"value" : {
"mean" : 168062.5,
"max" : 169748,
"min" : 165008,
"median" : 168346,
"stddev" : 1564.4682346225,
"CI" : 0.0077836281768634,
"iteration" : 8
}
},
{
"name" : "Startup time in ms",
"value" : {
"mean" : 4322.25,
"max" : 4606,
"min" : 4112,
"median" : 4255.5,
"stddev" : 188.005888965517,
"CI" : 0.0363703702021615,
"iteration" : 8
}
}
]
}
],
Parent Build
For each parent build, we'll aggregate the "raw" (and not "aggregated") data of all child builds that were launched by that parent build. Each child may carry different weight if they have different number of valid data points as compared to other child jobs. Hence, we'll use weighted average to get the most accurate results.
For each parent build, we'll aggregate the "aggregated" (and not "raw") data of all child builds that were launched by that parent build. Since all child jobs are identical configurations for perf testing, we weigh each child job equally even if one child job may have different # of good values to aggregate due to some unexpected failures, something that isn't very common. We need weigh child jobs equally due to thermal and other factors.
Raw Data for Parent
Doesn't keep any raw data.
Aggregated Data for Parent
"aggregateInfo" : [
{
"benchmarkName" : "LibertyStartupDT",
"benchmarkVariant" : "17dev-4way-0-256-qs",
"benchmarkProduct" : "<JDKName>",
"metrics" : [
{
"name" : "Footprint in kb",
"value" : {
"mean" : 168801.25,
"max" : 169342.5,
"min" : 168260,
"median" : 168801.25,
"stddev" : 765.443090634438,
"CI" : 0.0407409453425256,
"iteration" : 2
}
},
{
"name" : "Startup time in ms",
"value" : {
"mean" : 4296.3125,
"max" : 4305.5,
"min" : 4287.125,
"median" : 4296.3125,
"stddev" : 12.9930871043028,
"CI" : 0.0271712951513653,
"iteration" : 2
}
}
]
}
]
from aqa-test-tools.
piyush286
commented on July 30, 2024
Sample Parent Job
{
"_id" : ObjectId("5ccb3473865c57c13bbff657"),
"url" : "<JenkinsURL>",
"buildName" : "PerfNext-Parent",
"buildNum" : 16,
"buildDuration" : 725926,
"buildResult" : "SUCCESS",
"timestamp" : 1554733203557.0,
"type" : "Perf",
"status" : "Done",
"artifactory" : null,
"buildData" : {},
"buildOutputId" : ObjectId("5ccb347843ed61c13e693338"),
"buildUrl" : "<JenkinsURL>/job/PerfNext-Parent/16/",
"hasChildren" : true,
"machine" : null,
"parserType" : "ParentBuild",
"startBy" : "user <email>",
"aggregateInfo" : [
{
"benchmarkName" : "LibertyStartupDT",
"benchmarkVariant" : "17dev-4way-0-256-qs",
"benchmarkProduct" : "<JDKName>",
"metrics" : [
{
"name" : "Footprint in kb",
"value" : {
"mean" : 168801.25,
"max" : 169342.5,
"min" : 168260,
"median" : 168801.25,
"stddev" : 765.443090634438,
"CI" : 0.0407409453425256,
"iteration" : 2
}
},
{
"name" : "Startup time in ms",
"value" : {
"mean" : 4296.3125,
"max" : 4305.5,
"min" : 4287.125,
"median" : 4296.3125,
"stddev" : 12.9930871043028,
"CI" : 0.0271712951513653,
"iteration" : 2
}
}
]
}
]
}
from aqa-test-tools.
piyush286
commented on July 30, 2024
Sample Child Job
{
"_id" : ObjectId("5ccb347943ed61c13e69333a"),
"url" : "<JenkinsURL>",
"buildName" : "PerfNext-Child",
"buildNameStr" : "PerfNext-Child",
"buildNum" : 27,
"parentId" : ObjectId("5ccb3473865c57c13bbff658"),
"type" : "Perf",
"status" : "Done",
"aggregateInfo" : [
{
"benchmarkName" : "LibertyStartupDT",
"benchmarkVariant" : "17dev-4way-0-256-qs",
"benchmarkProduct" : "Build-20190402_03",
"metrics" : [
{
"name" : "Footprint in kb",
"value" : {
"mean" : 168062.5,
"max" : 169748,
"min" : 165008,
"median" : 168346,
"stddev" : 1564.4682346225,
"CI" : 0.0077836281768634,
"iteration" : 8
}
},
{
"name" : "Startup time in ms",
"value" : {
"mean" : 4322.25,
"max" : 4606,
"min" : 4112,
"median" : 4255.5,
"stddev" : 188.005888965517,
"CI" : 0.0363703702021615,
"iteration" : 8
}
}
]
}
],
"artifactory" : null,
"buildDuration" : 1087621,
"buildResult" : "SUCCESS",
"buildUrl" : "<JenkinsURL>/job/PerfNext-Child/27/",
"hasChildren" : false,
"machine" : "kermit",
"parserType" : "BenchmarkParser",
"startBy" : "upstream project \"PerfNext-Master\" build number 15",
"tests" : [
{
"_id" : ObjectId("5ccb348c70da0cc141677686"),
"testOutputId" : ObjectId("5ccb348c70da0cc141677685"),
"testResult" : "PASSED",
"testIndex" : 1,
"benchmarkName" : "LibertyStartupDT",
"benchmarkVariant" : "17dev-4way-0-256-qs",
"benchmarkProduct" : "<JDKName>",
"testData" : {
"metrics" : [
{
"name" : "Footprint in kb",
"value" : [
167268,
168804,
169748,
168884,
167272,
167888,
169628,
165008
]
},
{
"name" : "Startup time in ms",
"value" : [
4207,
4298,
4112,
4161,
4575,
4213,
4606,
4406
]
}
],
"javaVersion" : "<javaVersion>",
"jdkBuildDateUnixTime" : 1554206400000.0
}
}
],
"timestamp" : 1554322240238.0
}
from aqa-test-tools.
llxia
commented on July 30, 2024
-
How parentBuild
aggregateInfois calculated? Are parentBuild max, mean, median, stddev, etc calculated based on max, mean, median, stddev, etc in all childBuilds aggregateInfo?
For example,- parentBuild max = max (all childBuilds max values in aggregateInfo)
- parentBuild mean = mean (all childBuilds mean values in aggregateInfo)
- parentBuild stddev = stddev (all childBuilds stddev values in aggregateInfo)
...
If this is the case, this will cause inaccurate results. For example,
- childBuild1 has 5 valid runs (childBuild1Mean = sum of 5 runs / 5)
- childBuild2 has 10 valid runs (childBuild2Mean = sum of 10 runs / 10)
- aggregate the "aggregated" => parentBuildMean = (childBuild1Mean + childBuild2Mean) /2
This parentBuildMean is not the same as ( sum of 5 runs + sum of 10 runs) / (5 + 10)
-
We should add detail about how
PerfComparewill look like -
Should Perf Dashboard be updated to reflect this design?
from aqa-test-tools.
piyush286
commented on July 30, 2024
@llxia It's useful to take all child with equal weights when we are interleaving. Just so 2 interleave builds for baseline and test have similar weight for each iteration since similar factors would be affecting the same iteration for both.
But you're right! It's more accurate to take weighted averages so that we divide the mean by the valid # of data points. @sophiaxu0424 Could you please update your changes? Thanks!
I'll create another issue for updating Dashboard & Perf Compare.
from aqa-test-tools.
piyush286
commented on July 30, 2024
Closing this since all related work to this issue has been completed.
from aqa-test-tools.
Related Issues (20)
- Review "Pass %" metric and consider a new "Pipeline %" metric
- TRSS does not list the new release-openjdkNN-pipeline jobs and tests HOT 1
- New release pipeline builds/smoke tests do not display well in Grid view HOT 4
- Some test jobs not displaying in Grid & tree views HOT 3
- Further support rerun builds in UI
- Rerun build may cause Release Summary Report to be long HOT 1
- Monitoring option does not seem to work
- Add queue time information HOT 4
- Added pipelines for monitoring seems to duplicate themselves HOT 1
- [Enhancement] Collapse All button for TRSS buildTreeView HOT 1
- Release Summary report Rerun link causing a 'comment is too long' issue HOT 1
- Add Netlify attribution to TRSS client page
- Since we upgrade the node, we should upgrade to 18. 16 is out of active support and security support for 16 will end in Sep 2023. HOT 2
- Enhance docker compose in TRSS HOT 1
- Set TRSS link in the Jenkins job HOT 2
- Uncontrolled data used in path expression
- Store and display ML user feedback counts HOT 1
- Bug: JDK21 Build links missing from TRSS pipeline grid view HOT 2
- TRSS server container OOM
- Chatbot for Efficient Issue Triage and Resolution
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aqa-test-tools.