Git Product home page Git Product logo

poweroff-google-cloud-cap-billing's Introduction

Automatic cost control by capping Google Cloud billing

Badge: Google Cloud Badge: Terraform Badge: CI Badge: GitHub

With this repo you can cap costs and stop usage for a Google Cloud project by disabling Cloud Billing automatically. Removing the billing account from a project will cause all Google Cloud services in the project to terminate, including free-tier services.

Image: Architecture

You might want cap costs because you have a hard limit on how much money you can spend on Google Cloud. This is typical for students, researchers, or developers working in sandbox environments. In these cases you want to stop the spending and might be willing to shutdown all your Google Cloud services and usage when your budget limit is reached.

Warning When you remove Cloud Billing from your project, all resources are shut down. The resources may not shut down gracefully and be irretrievably deleted. There is no gracefully recovery if you disable Cloud Billing. You can re-enable Cloud Billing, but there is no guarantee that the service will be restored and manual configuration is required.

Everything is based on the original Google Cloud documentation.

This repo has the advantage that everything is deployed automatically thanks to Terraform. You don't have to set up all the steps each time for additional projects.

It also creates a separate custom role that can only unlink the billing account from the project, but not enable it. This has the advantage that only a billing administrator can enable the billing back and not the project itself.

Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays.

🏃 Deploying

Run all tasks in the free Google Cloud Shell. All necessary tools (gcloud and terraform) are already installed.

Open in Cloud Shell

Trust repo:

Screenshot: Cloud Shell trust repo

You need to be the Owner and Billing Account Administrator of the project.

1️⃣ Clone

Clone this repo and initial setup:

git clone https://github.com/Cyclenerd/poweroff-google-cloud-cap-billing.git
cd poweroff-google-cloud-cap-billing
terraform init

2️⃣ Set Project

Set the project that should be stopped when a certain amount is exceeded:

gcloud auth login
gcloud config set project YOUR-GOOGLE-CLOUD-PROJECT

3️⃣ Enable APIs

Enable required APIs and services:

bash enable-services.sh

4️⃣ Deploy

Now you can create a budget alert and Cloud Function for this project:

# Stop billing if 1000 USD are exceeded
terraform apply \
  -var="project_id=$GOOGLE_CLOUD_PROJECT" \
  -var="target_amount=1000"

In detail the following is added to the project:

  1. Service account : sa-cap-billing@...
  2. Custom role : myCapBilling
  3. Pub/Sub topic : cap-billing-alert
  4. Pub/Sub subscription : cap-billing-alert-pull
  5. Budget alert : Unlink YOUR-GOOGLE-CLOUD-PROJECT from billing account
  6. Storage bucket for Cloud Function source code : RANDOM-UUID
  7. Cloud Function with Pub/Sub event trigger : cap-billing-RANDOM-HEX

Variables

You can customize the setup with the following Terraform input variables:

Variable Description Default
project_id The project ID for the resources and budget alert
pubsub_topic Name of the Pub/Sub topic cap-billing-alert
target_amount Set maximum monthly budget amount (currency as in billing account) 1000
region Region for the resources us-central1

Examples

Via command:

# Stop billing if 5 USD are exceeded and deploy everything in europe-west4
terraform apply \
  -var="project_id=$GOOGLE_CLOUD_PROJECT" \
  -var="target_amount=5" \
  -var="region=europe-west4"

Or via config file terraform.tfvars:

# Project ID
project_id=bla-fa-123
# Target amount
target_amount=10
# Pub/Sub topic
pubsub_topic=stop-billing-alert

Apply with config file:

terraform apply

» Terraform Help

💥 Test

You can perform a test. The billing account will be removed. Do it only if you are sure and the project is not important.

Check active billing account:

gcloud beta billing projects describe "$GOOGLE_CLOUD_PROJECT" | grep billingAccountName

Send a message that triggers the Cloud Function and disables billing:

gcloud pubsub topics publish "cap-billing-alert" --message='{ "costAmount" : 2, "budgetAmount": 1 }'

Wait a while... Billing should then be disabled.

gcloud beta billing projects describe "$GOOGLE_CLOUD_PROJECT" | grep billingAccountName

💸 Enable Billing

Enable billing for an existing project:

Screenshot: Enable billing

» Google documentation

📎 Prerequisites

To run the commands described in this repo, you need the following:

  1. Install the Google Cloud SDK version 319.0.0 or later
  2. Install Terraform version 1.1.9 or later.
  3. Set up a Google Cloud billing account and project.

❤️ Contributing

Have a patch that will benefit this project? Awesome! Follow these steps to have it accepted.

  1. Please read how to contribute.
  2. Fork this Git repository and make your changes.
  3. Create a Pull Request.
  4. Incorporate review feedback to your changes.
  5. Accepted!

📜 License

All files in this repository are under the Apache License, Version 2.0 unless noted otherwise.

Please note:

  • No warranty
  • No official Google product

poweroff-google-cloud-cap-billing's People

Contributors

cyclenerd avatar dependabot[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

poweroff-google-cloud-cap-billing's Issues

Document how well this handles latency in the alert and load test results?

When billing alerts from Google Cloud (or other services) are mentioned, there's usually the caveat around latency between the alert and shut down. For example https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage says this:

Note: There is a delay between incurring costs and receiving budget notifications. Google Cloud products report usage and cost data to Cloud Billing processes at varying intervals, so you might incur additional costs for usage that hasn't arrived at the time that all services are stopped.
Following the steps in this capping example is not a guarantee that you will not spend more than your budget.
Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays.

How well does this handle that issue? Have you tried stress testing it by running a quick lambda function or something and seeing how many invocations happen before the billing is cut?

I was reading on this a while back and saw this thread https://news.ycombinator.com/item?id=25376768 where someone said this:

As a former victim to the same issue as OP, I am furious every time I see a Googler promote that as a solution.
In our case, we racked up a $10000 bill on BigQuery in ~6 hours, when a job was failing and auto-retrying.

We had set up every alert correctly and our reaction time was about 5 minutes (about $100 of usage, no big deal). So how did we get a $5000 bill?

Google's alert was 6 hours late (according to them, this was root-caused to us, because we were submitting jobs continuously). They pointed to their TOS and said they don't guarantee on-time delivery of the alert. I had to write up a blog post with fancy graphs and prepare it for social media before they finally agreed to eat the bill.

I saw the discussion at https://www.reddit.com/r/googlecloud/comments/va0cc0/automating_cost_control_by_capping_google_cloud/ but it didn't seem like anyone asked this question.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.