m-lab / bigsanity Goto Github PK
View Code? Open in Web Editor NEWM-Lab’s Big Data Sanity Checker
License: Apache License 2.0
M-Lab’s Big Data Sanity Checker
License: Apache License 2.0
Current implementation in table_names is that tables like 2014_06 might have tests with log_time
s in 2014_05 or 2014_07, but upon closer inspection of our data, this bug only causes spillover forward (i.e. only 2014_07). We need to fix table_names to avoid adding unnecessary tables to the queries.
BigQuery queries can fail due to a variety of reasons (connection error, query error). In theory, these are all being encapsulated by the query_execution
module into query_execution.Error
exceptions.
TableEquivalenceChecker
should catch query execution errors and gracefully handle them (probably by retrying 3x and then moving on).
Right now a BigQuery error will completely crash the BigSanity process.
Probably including something like total errors found
Instead of this:
import os
import sys
sys.path.insert(1, os.path.abspath(os.path.join(
os.path.dirname(__file__), '../bigsanity')))
import intervals
We can do this:
from __future__ import absolute_import
from bigsanity import intervals
query_execution.BqFailedError: bq failed when attempt to execute query
Should be attempting
paris_traceroute data uses log_time
rather than web100_log_entry.log_time
since Paris Traceroute is not web100 based. We need to adjust the query construction to address this.
We are currently using tz-naive datetimes. This can probably bite us in unexpected ways around month borders, so let's switch everything to just explicitly use UTC time.
We'll run BigSanity in non-batch mode to get initial validation quickly, but going forward, we'll switch to batch (or make this configurable) so that we can minimize impact on other queries.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.