RelMon threshold to report a difference about cms-bot HOT 14 CLOSED

@nclopezo did you look into this?

from cms-bot.

Hi,
It seems that it was a glitch (cms-sw/cmssw#7565 (comment)). If this is happening again I can check it.

from cms-bot.

Hi David,

I'm not sure what was a glitch.
The state of RelMon is still the same, at least as of 2 days ago.

Since DAS is now a good generator of large differences in 140.53
I can use it as an example of missing differences in RelMon in 8752
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/7174/

RelMon result has essentially no differences
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/7174/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/
compare this to the slew of changes in the log file of RelMon (180MB log
file, you can dare to open)

vs
e.g. fwlite pt of tracks
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/7174/validateJR/all_OldVSNew_RunHI2011wf140p53/all_OldVSNew_RunHI2011wf140p53c_log10recoTracks_hiGeneralTracks__reRECO_obj_pt.png

    --slava

On 4/17/15 3:16 AM, David Mendez wrote:

Hi,
It seems that it was a glitch (cms-sw/cmssw#7565 (comment)
cms-sw/cmssw#7565 (comment)). If
this is happening again I can check it.

—
Reply to this email directly or view it on GitHub
#313 (comment).

from cms-bot.

(I can't reopen an issue closed by an admin)

from cms-bot.

Hi Slava,

I modified the threshold to be 0.05, here are the results:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/7242/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/
Are these results better? I can try with other values if you want.

from cms-bot.

Hi David,

As a check, could you please generate a diff with 0.005 as a threshold.
In majority of the cases of tests in jenkins we expect no changes at all,
since we are usually rerunning on the same events.
In this test case, where files are different, the number of failed
comparisons is still fairly small.
So, a looser selection would make sense.

(recall that for the alternative-comparisons we applied some logic of
changing
thresholds in diffs depending on how many are found; it's unclear if the
same can be applied here as well)

On 4/21/15 6:21 AM, David Mendez wrote:

Hi Slava,

I modified the threshold to be 0.05, here are the results:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/7242/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/
Are these results better? I can try with other values if you want.

—
Reply to this email directly or view it on GitHub
#313 (comment).

from cms-bot.

Here are the results with 0.005
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/20/

If I modify it to run first the JR comparison, the thresholds could be assigned in a similar way as for example: https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/7174/alternative-comparisons/assignedParameters.log

from cms-bot.

On 4/21/15 8:23 AM, David Mendez wrote:

Here are the results with 0.005
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/20/

Sorry, I misunderstood the direction of the threshold.
It's actually showing differences less probable that the threshold.
So, lets try 0.1 for this case with many differences
(this should be an equivalent of mod 2 in the alternative-comparisons).

The mapping to the choices done in alternative comparisons should be

• 0.999 for mod=3
• 0.1 for mod=2
• 0.999999999999 for mod=1
  I'm not sure if there is a switch for mod=0 (any bin difference) in RelMon

If I modify it to run first the JR comparison, the thresholds could be
assigned in a similar way as for example:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/7174/alternative-comparisons/assignedParameters.log

—
Reply to this email directly or view it on GitHub
#313 (comment).

from cms-bot.

Right now it is like this: https://github.com/cms-sw/cms-bot/blob/master/comparisons/analyzeFWComparison.py#L65

I can do the same with the thresholds for RelMon comparisons

from cms-bot.

David,
I think the same logic should work for RelMon.
Thank you.

from cms-bot.

Hi Slava,
Here you can see the results with the dynamic assignment of thresholds:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_4_X_2015-04-15-1100+8752/38/
You can see in the file RelMonAssignedParameters.log how the parameters were assigned.
And in this RelMonComparisonParams.txt a summary of the thresholds which is used by the script when it launches the RelMon comparison.

from cms-bot.

If this is ok I will merge this new feature to the cms-bot master branch.

from cms-bot.

I merged it and it should now be available for the next comparisons.

from cms-bot.

thank you.
Sorry, I couldn't check earlier.
Looking at the RelMonAssignedParameters.log , the chosen params make sense

from cms-bot.