m-lab / bigsanity Goto Github PK

Current implementation in table_names is that tables like 2014_06 might have tests with log_times in 2014_05 or 2014_07, but upon closer inspection of our data, this bug only causes spillover forward (i.e. only 2014_07). We need to fix table_names to avoid adding unnecessary tables to the queries.

BigQuery queries can fail due to a variety of reasons (connection error, query error). In theory, these are all being encapsulated by the query_execution module into query_execution.Error exceptions.

TableEquivalenceChecker should catch query execution errors and gracefully handle them (probably by retrying 3x and then moving on).

Right now a BigQuery error will completely crash the BigSanity process.

Probably including something like total errors found

Instead of this:

import os
import sys

sys.path.insert(1, os.path.abspath(os.path.join(
    os.path.dirname(__file__), '../bigsanity')))
import intervals

We can do this:

from __future__ import absolute_import

from bigsanity import intervals

query_execution.BqFailedError: bq failed when attempt to execute query

Should be attempting

paris_traceroute data uses log_time rather than web100_log_entry.log_time since Paris Traceroute is not web100 based. We need to adjust the query construction to address this.

We are currently using tz-naive datetimes. This can probably bite us in unexpected ways around month borders, so let's switch everything to just explicitly use UTC time.

We'll run BigSanity in non-batch mode to get initial validation quickly, but going forward, we'll switch to batch (or make this configurable) so that we can minimize impact on other queries.