Plugin - Custom Bigquery

Hooks

CustomBigQueryHook

This is an extension of the default BigQueryHook here that enables the json key for google cloud to be directly read in from the connection.

Useful for cloud hosted type set ups.

As of Airflow 1.9 and above this behavior will be default as per this issue .

Operators

CustomBigQueryOperator

This operator is an extension of the default BigQueryOperator here with the addition of:

• A query to the relevant __TABLES__ meta table to log destination table info before (if relevant) and after the main task of the operator (uses pandas.read_gbq() and pandas_gbq.GbqConnector.schema()).
• Sending a custom event type record for 'before' and 'after' to a dag and destination dataset specific table in an "airflow" dataset in BQ (This is useful for backing out task stats where direct access to the airflow db is not possible).