Comments (8)
A new Issue was created by @christopheralanwest .
@davidlange6, @Dr15Jones, @smuzaffar, @fabiocos, @kpedro88 can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
from cms-bot.
Related to #937
It seems like we need a general scheme to generate, store, and track alternative comparison baselines.
from cms-bot.
@christopheralanwest , we have the results for PRs and I think with little modification we can use that as baseline. So if I understand correctly you will be doing
- cms-sw/cmssw#27651: please test
- Use latest IB and normal run normal tests
- Use IB as baseline
- cms-sw/cmssw#27698: please test baseline #27651
- Pickup IB used for testing #27651 PR and build and test #27698 and #27651
- Use baseline of #27651 which should have been uploaded before issuing this command.
- cms-sw/cmssw#27644: please test baseline #27651,#27698
- Pickup IB used for testing #27651 and build and test #27644, #27698 and #27651
- Use baseline of #27651,#27698 which should have been uploaded already.
Note that in this case one can only start the tests once a baseline of other PR is already available. Also you can not say e.g. for cms-sw/cmssw#27644 please test baseline #27698
as baseline for only #27698 was never generated and uploaded.
from cms-bot.
@smuzaffar What you propose isn't actually what I intended nor would it work for my use case involving autoCond updates. For the UL preparation, it is desirable to include finalized conditions in autoCond in master so that most of the updates are included in 11_0_X relval tests prior to their inclusion in 10_6_X. At the same time, there are PRs that include coupled code+conditions updates, and these should be tested separately from other condition updates. All of these PRs update the same lines in autoCond and so cannot be merged together. As backports are not merged as promptly as those in master, there may be three or four open PRs that modify the same line in autoCond.
In the scheme that I had in mind, your scheme would be written as, for example,
please test baseline #27651 with #27651
please test baseline #27651,#27698 with #27651,#27698
but that would not be sensible in the case of autoCond updates, in which global tags generally include all previous updates.
On another point: in your scheme one has to wait until the PR tests for the "earlier" PR(s) are completed prior to running the tests for the "later" PR. Since the tests take a non-negligible time to complete relative to the frequency of IB builds, the bot may use a different IB for the tests of the later PR. Could you instead ensure that when an alternate baseline is selected, the tests consistently use the same IB build?
from cms-bot.
@christopheralanwest , I am going to update bot which should allow us to generate baseline on demand (#1728 ). I think we can make use of this new feature to generate on demand baseline using extra PRs too e.g.
please test using baseline baseline-prs
please test with prs using baseline baseline-prs
can generate baseline using IB+baseline-prs
and use it to compare the results for IB+pr
or IB+pr+prs
. This also ensure that both baseline and PR tests use the same IB.
Question to you, do we still need this feature and will the above implementation is enough to cover your use case?
from cms-bot.
The feature requested in this PR would have been useful during the preparation for the Legacy production when new GT updates needed to be integrated before the PR including the previous set of updates had been merged. I don't know if GT changes are requested frequently enough nowadays that the feature is necessary. I'm no longer working on AlCa PRs so I don't need this feature myself, but I've added @cms-sw/alca-l2 to see if they would find it useful.
from cms-bot.
assign alca
- we'll need to digest a bit this before giving a coheren answer :)
from cms-bot.
New categories assigned: alca
@yuanchao,@francescobrivio,@malbouis,@tvami you have been requested to review this Pull request/Issue and eventually sign? Thanks
from cms-bot.
Related Issues (20)
- PR testing, memory info HOT 1
- parse and monitor JobReport.xml HOT 3
- Security Issue? HOT 12
- bot / github UI get confused if the same branch is used for multiple PRs HOT 3
- Add battibass as muon-dpg L2 HOT 4
- add step5 / NANO for reco release profiling HOT 8
- Test UBI8 and AlmaLinux8 HOT 7
- PF as a PR type HOT 3
- patching command HOT 2
- Integrate part of NANO test in cmsbot commands HOT 2
- Use cvmfs parallel publishers/gateway to deploy CMSSW IBs HOT 22
- Workflow `39634.21` undefined in `igprof` 12_6_X HOT 7
- add automatic labeling of PR types from package lists HOT 12
- Empty `Igprof Comparison cpu usage RECO produce methods` HOT 11
- Include CMSSW non-blocking dpg/pog labels to release notes/summary HOT 3
- Change default wfs for PR tests on GPU in `CMSSW_13_X_Y` HOT 6
- signature delegation on specific packages HOT 4
- Change default action to squash merge in cmssw repo. HOT 45
- [RFC] Code-style for cms-bot scripts HOT 5
- forcing merge of externals once cmssw PR is merged HOT 19
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cms-bot.