Mozart - Business logic for Search Ads 360

Table of Contents

• How it works
• Architecture
• Set-up
  • Pre-requisites
  • Composer set-up
• DataFlow set-up
• Cloud Storage set-up

Mozart is a framework for automating tasks on Search Ads 360 (SA360). Mozart lets advertisers and agencies apply their own business logic to SA360 campaigns by leveraging well-known technologies such as Apache Beam.

Mozart is designed to be deployed in an Airflow+Beam platform. The rest of this documentation assumes Google Cloud Platform (GCP) is used for deployment. Composer is the name of GCP's managed version of Airflow, whereas DataFlow is the name of the managed version of Beam.

How it works

Mozart leverages SA360 Reporting API and SA360 Bulksheet uploads to perform the automation tasks.

The sequence of high-level operations is:

1. Reports are downloaded from SA360 API. These reports must include all entities to be processed (e.g.: Keywords, Ads).
2. Downloaded reports are analyzed, applying the custom business logic. The output of this logic is a CSV file containing updated information about the entities (Keywords, Ads). For example: a new Max CPC value for certain keywords
3. CSV files with updated values are uploaded into SA360 using sFTP Bulksheet upload

Architecture

Mozart consists of two main modules:

1. An Airflow DAG
2. A Beam Pipeline

Set-up

This guide describes how to set-up Mozart on Google Cloud Platform.

For the rest of the guide, it is assumed that you have created a Google Cloud project, enabled billing and that you have Admin access to the project via console.cloud.google.com.

For instructions on how to create a Google Cloud project and enable billing please refer to Google Cloud Platform documentation.

You must also enable certain Google Cloud Platform APIs to be able to use Mozart. In order to enable all of them in a single step, click on enable APIs and follow the instructions.

Note: The link to enable APIs might take a while to load. Please be patient.

Pre-requisites

You must have the following software installed on your computer:

• Google Cloud SDK
• Apache Maven

Composer set-up

First step is to set-up Google Cloud Composer. In order to do so, follow the steps:

1. Create a new Composer environment

   1. Go to console.cloud.google.com/composer

   2. Check 'Enable beta features'

      

   3. Click on 'Create'

      

   4. Type in the following options:

      • Name: Enter a descriptive name
      • Location: select a location close to you (e.g.: 'europe-west1)
      • Scopes:
        • https://www.googleapis.com/auth/bigquery
        • https://www.googleapis.com/auth/devstorage.read_write
        • https://www.googleapis.com/auth/doubleclicksearch
        • https://www.googleapis.com/auth/compute
        • https://www.googleapis.com/auth/cloud-platform
      • Disk size: 20
      • Airflow version: 1.10.0

   5. Click on 'Create' at the bottom of the page

   Tip: Creating the environment takes a while. We suggest you to continue with the other sections and come back later to finish environment configuration.

2. Once the environment is created, go to the Composer page and open the 'Airflow webserver' for the newly created environment.

   

3. Click on 'Admin' > 'Variables'

   

4. Create the following variables:

DataFlow set-up

1. Create a Service account
   1. In the GCP Console, go to the Create service account key page.
   2. From the Service account drop-down list, select 'New service account'.
   3. In the Service account name field, enter a name .
   4. From the Role drop-down list, select 'Project > Owner'
      • Use 'Project > Owner' for testing purposes. For a production system, you should select a more restrictive role.
   5. Click 'Create'. A JSON file that contains your key downloads to your computer.
2. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the absolute path of the JSON file downloaded in the previous step

Cloud Storage set-up

Mozart's DataFlow pipeline works with files on Google Cloud Storage. You need to create a bucket where these files will be stored:

1. Go to console.cloud.google.com/storage

2. Create a bucket

   1. Click on 'Create bucket'

   2. Choose a name for the bucket

   3. Choose a location that matches the location you used for the Composer configuration

      Note: A configuration based on 'Regional' storage class and the same location as the one used for Composer is suggested. However, you may want to use other options if you plan on using the bucket for storing custom data. Check the Cloud Storage docs for more info on all the options.

3. Create a lifecycle rule for the bucket

   1. In the bucket list view, click on the Lifecycle column value

      

   2. Click on 'Add rule'

   3. Select 'Age' condition, and set it to 30 days

   4. As an action, select 'Delete'

   5. Save the rule

   Note: Lifecycle rules help you decrease Cloud Storage costs by deleting old elements. We suggest setting this 30-day policy, but you should adjust this if you wish to keep items for longer, or if you plan on storing other data in the same bucket.

4. Create the following folders:

   1. staging
   2. templates
   3. sa360_reports
   4. sa360_upload