Goal

Automate sample data generation to csv file.

Acceptance

When generate_data.py executed, it will produce a barbeque_sales.csv with schema as shown at #1 :

• id : integer, auto increment transaction id
• brand_id : string, brand of the barbeque
• amount : integer, sales amount
• _PARTITIONTIME : timestamp, when the sales happen
  For now, load to BQ will be done manually via UI

Out of scope

Automatically load data to BQ using script

Goal

Create bigquery sql builder from config and cmd param

Acceptance

Given an object of job config and cmd param, necessary sql could be generated
Example:
See command and yaml at #1
It will produce BigQuery Standard SQL as per below:

SELECT
  DATE(_PARTITIONTIME) AS day,
  brand_id,
  SUM(amount) AS amount_sum,
  COUNT(*) AS cnt
FROM `barbeque.sales`
WHERE _PARTITIONTIME >= "2018-06-17 00:00:00" AND _PARTITIONTIME < "2018-06-30 00:00:00"
GROUP BY 1, 2

Out of Scope

• config reader from file is out of scope, config and date param will be an in memory object
• fire sql to bq is out of scope

Goal

Stitching all components created on a main flow

• read the job config range param
• generate sql
• fire sql to bq

Acceptance

Equal to #1

Out of scope

Nice error handling and logging is still out of scope

Goal

Read config from file.

Acceptance

Given config in yaml file format, it could be read when user execute barbeque sales.yml

Out of scope

Handling daterange parameter

Goal

Enable integration to BQ

Acceptance

Given certain SQL, we could fire the SQL to BQ and load the SQL result to a destination table with partition config, in replace manner

Out of scope

Keep track of job status in bq

Goal

MVP of barbeque, creating partition preserving summary, with assumptions:

• Only support day-partitioned table
• Partition time equal to event time. Any out of sync timestamp will be discarded. No feature for custom time field yet.
• Only support count and sum aggregation. Count is there by default.

Acceptance

Given this command: bbq sales.yml --start_dt="2018-06-17" --end_dt="2018-06-30" barbeque will read source table and write into target table (in replace mode) with name equal to job name, with partition preserved.

sales.yml :

name: sales_count_by_brand_id_day
type: day_partition_preserving
table: barbeque.sales
keys:
- brand_id
aggr:
- field: amount
  func: sum

Sample data source: (timestamp is partition time)

Sample data result:

Out of Scope

Additional feature planned (written here for self-note purpose):

• condition_sql: "entity_id IS NOT NULL AND entity_id <> 0"
• partition_padding: day_1 // assume partition time and timestamp field in sync, or assume +- n day