DQM monitor not showing memory reduction about cms-bot HOT 9 CLOSED

@schneiml FYI

from cms-bot.

A new Issue was created by @fabiocos Fabio Cossutti.

@davidlange6, @Dr15Jones, @smuzaffar, @fabiocos can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

from cms-bot.

i notice is that the dqm size check was done on only 8 files

SUMMARY DQMHistoSizes: Histogram memory added: 0.0 KiB( 8 files compared)

likely due to something going wrong with the dqmMemoryStats call

Weird output /136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/step2_inDQM.root
('', 'Traceback (most recent call last):\n File "/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-04-18-2300/bin/slc6_amd64_gcc630/dqmMemoryStats.py", line 132, in \n for (fn, v) in reader.read_objects():\n File "/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-04-18-2300/python/DQMServices/FileIO/DQM.py", line 39, in read_objects_dqmio\n object_type = self.DQMIO_TYPES[indices.Type]\nIndexError: list index out of range\n')
('', 'Traceback (most recent call last):\n File "/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-04-18-2300/bin/slc6_amd64_gcc630/dqmMemoryStats.py", line 132, in \n for (fn, v) in reader.read_objects():\n File "/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-04-18-2300/python/DQMServices/FileIO/DQM.py", line 39, in read_objects_dqmio\n object_type = self.DQMIO_TYPES[indices.Type]\nIndexError: list index out of range\n')

from cms-bot.

@davidlange6 right, checking into this as well

from cms-bot.

A standalone test on 136.8311 does not show any error message (and 0 difference in the content), compatible with no B-tag specific folder

from cms-bot.

Looks like I definitly did screw up a regexp here [1] (minus missing), though that does not entirely explain the problem yet...

[1] https://github.com/cms-sw/cms-bot/pull/972/files#diff-4cc3e6a66c2b94c385ce39318e327cbaR167

from cms-bot.

... but the comparison in question was done using the old cms-bot code, looking at inDQM files. The "8 files compared" can be due to how files were counted back then (only those with changes or sth. in that direction).

The failure of dqmMemoryStats on these inDQM files is interesting and surprising, maybe @dmitrijus has an idea there. Either ways, the new code looks at harvesting output (which is in a different format), and should not show this problem (but maybe new ones).

from cms-bot.

The weird output reported by David should be fixed now.

from cms-bot.

this is addressed by #980

from cms-bot.