shalinmangar / clean-room Goto Github PK

We have a blame.py which works on only 1 test at a time. We either need to write a new script or modify the existing blame script to accept a given test_date, find the report.json for that date and run blame for each demotion.

Right now, there are separate scripts and configurations for identifying flaky tests using bare metal beasting and jenkins failure reports. But conceptually both are really just sources of data that must be combined together. For example, if we use jenkins failure reports as the source, we must still run the beasting filters to see if we can reproduce the failure. If the beasting filters cannot reproduce the failure then it is impossible to perform a git bisection. Perhaps we can just fold the use of jenkins failure report as another filter before/after beasting and simplify the code base?

Integrate @markrmiller's Lucene/Solr beasting script for better isolation and test seed selection.

A git bisect requires that we differentiate between a pass, fail, skip and abort. However, the current filters only return a pass or fail value. The fail can mean a real failure or a build failure. This can lead to wrong bisection and must be fixed. The corresponding source files are room_filter.py

We should perhaps have a separate html page (either static or JS/dynamic) for each test date so that information specific to each date can be visualized. Examples of such information are room data, promotions, demotions, blame info etc.

The current html report is rather basic. It should show the list of tests in the clean room and detention in a tabular format. Perhaps the tabulator JS widget can be used here.

The promote_if_not_failed_days configuration is used to promote tests to the clean room if they haven't failed in the last N days. But promotion should happen only if the test can pass all filters. Otherwise tests which are disabled and not run by the jenkins jobs might also be promoted by mistake.

Blame only prints the output on stdout and logs it to a text file. It should modify the given test date's report.json and store blame info so that it can be processed or visualised later by the reporting script.

We use the jenkins data to demote tests to detention and each such demotion is to be bisected to find the offending commit. However, for bisection to work we must be able to reproduce the test failure (unless it is a new test, in which case blame is a quick git log). We should run each demoted test against the configured filters and run blame only if the filters fail for this test. If the test passes our filters then we should mark the test somehow and indicate in the report{.json,.html} and skip running blame.

sh is a very nice replacement for invoking commands using the subprocess module. We can use it to simplify our code. See https://amoffat.github.io/sh/

We have support for a promote_if_not_failed_days configuration parameter which is used to promote tests to the clean room provided they have not appeared in the jenkins failure reports for the last N days and can pass all filters. However, we do not have a corresponding feature to demote tests only if they failed in the last N days or weeks or failed K times in a given date interval.

Write documentation on how to install and run the project components.