Deploys an end-to-end Data Analytics technical walkthrough on Google Cloud. All the services are connected, configured and ready to run. The deployed artifacts can be run in any order letting you set your own path through the system. The system has 70 to 700+ million rows of data so that you can show "true life" performance, scale and integration of various services. The system orchestates itself using Airflow, communicates over private IP addresses and has working code to demonstrate how to build an end to end system.

Prior Months

• September 2023
• August 2023
• July 2023
• June 2023
• May 2023

• The demo costs about $20 USD a day to run or $2 USD per day if you remove Composer.
• Several of the demos start up a service and then shut it down after 4 hours via Airflow automation to save on costs.
• To save on costs you can delete Composer since it costs ~$17 USD per day.
  • Once you remove Composer cost drops to ~$2 USD per day.
  • You will lack the automation Composer provides.
  • You would want to run any DAGs you are interested in deploying before removing Composer.

After the repo is cloned you would type source deploy.sh to deploy.

• Deployment Video
• Post Deployment Verification Video
• The following IAM roles are required to deploy the solution
  • Prerequisite: Billing Account User (to create the project with billing)
  • Prerequisite: Organization Administrator (to create all assets)
  • Prerequisite: Organization Policy Administrator (to change org policies)
  • Optional: Folder Editor (if you alter the script to place the project in a folder)

1. Open a Google Cloud Shell: http://shell.cloud.google.com/
2. Type: git clone https://github.com/GoogleCloudPlatform/data-analytics-golden-demo
3. Switch the prompt to the directory: cd data-analytics-golden-demo
4. Run the deployment script: source deploy.sh
5. Authorize the login (a popup will appear)
6. Follow the prompts: Answer “Yes” for the Terraform approval.

• Deployment Video
• Post Deployment Verification Video
• The following items are required to deploy the solution
  • Prerequisite: You will need a project created for you (IT can do this for you)
  • Prerequisite: You will need to be an Owner (IAM role) of the project to run the below script
  • Prerequisite: You will need an Organization Policy Administrator to disable the following Org Policies (IT can do this for you)
    • requireOsLogin = false
    • requireShieldedVm = false
    • allowedIngressSettings = allow all
    • allowedPolicyMemberDomains = allow all
    • restrictVpcPeering = allow all

1. Open a Google Cloud Shell: http://shell.cloud.google.com/
2. Type: git clone https://github.com/GoogleCloudPlatform/data-analytics-golden-demo
3. Switch the prompt to the directory: cd data-analytics-golden-demo
4. Update the hard coded values in deploy-use-existing-project-non-org-admin.sh
5. Run source deploy-use-existing-project-non-org-admin.sh
6. Your Organization Policy Administrator can then reenable the following Organization Policies
   • (DO NOT RE-ENABLE) requireOsLogin = false
   • (RE-ENABLE) requireShieldedVm = false
   • (RE-ENABLE) allowedIngressSettings = allow all
   • (RE-ENABLE) allowedPolicyMemberDomains = allow all
   • (RE-ENABLE) restrictVpcPeering = allow all

1. By default the solution deploys to us-west{x} region and US (multi-region)
2. To deploy to another region review the code in deploy-europe-region.sh
3. You can run either of the above deployment methods. Copy the Terraform "region" parameters to either of the above scripts.

• Open Cloud Composer
• Open the Airflow UI
• You will see the Run-All-Dags DAG running
  • This will run the DAGs needed to seed the project with data
  • Once the DAG is complete, you can run the solution

1. If the script fails to enable a service or timeouts, you can rerun and if that does not work, run source clean-up.sh and start over
2. If the script has a security type message (unauthorized), then double check the configure roles/IAM security. Your user account is probably not an Owner of the project or Org Admin if creating the project.
3. When using Cloud Shell: If you get a "networking error" with some dial tcp message [2607:f8b0:4001:c1a::5f], then your cloud shell has a networking glitch, not the Terraform network. Restart the deployment "source deploy.sh". (e.g. Error creating Network: Post https://compute.googleapis.com/compute/beta/projects/bigquery-demo-xvz1143xu9/global/networks?alt=json: dial tcp [2607:f8b0:4001:c1a::5f]:443: connect: cannot assign requested address). This typically happens on WiFi.

• cloud-composer
  • dags - all the DAGs for Airflow which run the system and seed the data
  • data - all the bash and SQL scripts to deploy
  • dataflow
• dataflow
  • Dataflow job that connects to the public Pub/Sub sample streaming taxi data. You start this using composer.
• dataproc
  • Spark code to that is used to process the initial downloaded data
• colab-enterprise
  • Sample notebooks that can be run in Vertex AI. To create the managed notebook, use the DAG in composer.
  • sql-scripts
• sql-scripts
  • The BigQuery SQL sample scripts. These are currently deployed as stored procedures. You can edit each stored procedure and run the sample code query by query.
• terraform
  • The entry point for when deploying via cloud shell or your local machine. This uses service account impersonation.
• terraform-modules
  • api - enables the GCP apis
  • org-policies - sets organization policies at the project level that have to be "disabled" to deploy the resources.
  • org-policies-destroy - commented out org policies so we can revert the org policies back to their original state.
  • project - creates the cloud project if a project number is not provided
  • resources - the main set of resources to deploy
  • service-account - creates a service account if a project numnber is not provided. The service account will be impersonated during the deployment.
  • service-usage - enables the service usage API as the main user (non-impersonated)
  • sql-scripts - deploys the sql scripts