Comments (20)
I made brief research and figured out that the most efficient way for us to have the historical billing usage data is BigQuery export (https://cloud.google.com/billing/docs/how-to/export-data-bigquery); it also allows us to visualize the data with Data Studio - https://cloud.google.com/billing/docs/how-to/visualize-data.
An alternative method is exporting to CSV/JSON file, which is easier, but less-feature-rich (https://cloud.google.com/billing/docs/how-to/export-data-file).
PS. BigQuery export requires a specific bigquery.datasets.create
permission on GCP.
from k8s.io.
cc @hh who is going to work on the Bigquery + DataStudio utilization.
from k8s.io.
Brought #193 under this umbrella so we know where our storage costs are going.
Thank you @justinsb for bringing this up! (and for keeps kops infra up until the transition) 💯 🥇
from k8s.io.
Tracking implementation of per-namespace billing at #194
from k8s.io.
My ideal outcome for this:
- at the start of each wg-k8s-infra meeting we can look at generated reports
- for cluster-based infra, we can see per-namespace billing info (#194)... ie: if we added the "gcsweb" service, would we know how much extra that is costing us
- for artifact storage, we can see how much a given subproject / sig is costing us (#193)
- for non-cluster based infra, we can see per-project billing info... ie: how much is dns costing us, how much is bigquery costing us, etc.
from k8s.io.
In the most recent wg-k8s-infra meeting, Tim was able to read out "the default GCP billing report." This allowed him to break down costs over the past two weeks as follows:
- vm (compute, ram, ssd pv)
- dns (queries)
- load balancing
- dns-zone
- inter-zone egress
- (no storage bills as of yet)
Next step: demo of data studio analysis of data in bigquery #242
from k8s.io.
- Want parity of number between tim's readout and justin's readout
- Answer on billing-per-bucket
- Find more info on billing-per-namespace
- Possibly send daily reports to google group
from k8s.io.
Justin is getting a daily e-mail, will send to cluster admins google group, still have questions about parity
from k8s.io.
Justin to look at sharing Data Studio report with other users, possibly use another group as defined in groups.yaml
TBD whether we can export / import Data Studio reports to use some kind of gitops driven model
from k8s.io.
/assign @justinsb
Now sending billing report out to a google group, still not sure how to allow others to edit the report
from k8s.io.
Would like to continue to cross check this billing report against tim's billing report
from k8s.io.
Send report weekly to public google group https://groups.google.com/forum/#!forum/kubernetes-wg-k8s-infra
Send report daily to existing google group of k8s-infra-gcp-accounting
from k8s.io.
Backing up / sharing the work:
- lets open up the Data Studio project to the k8s-infra-gcp-account group
- may want to consider dumping sql query out, or finding a way to make the data studio report public read-only, etc.
from k8s.io.
/unassign
/assign @justinsb
from k8s.io.
@thockin unable to see report since july 1st
from k8s.io.
Latest report august 5th: https://groups.google.com/d/msg/kubernetes-wg-k8s-infra/uW7dEBaBWDY/-zXJc0QEBQAJ
from k8s.io.
would like to share edit access of the report to a group... k8s-infra-gcp-accounting can already edit it
would like some kind of README in an accounting directory
nice to have: other people provide other data studio addons
from k8s.io.
will continue to spot check as a recurring agenda item but we are ready to call this done
from k8s.io.
@justinsb update on this?
from k8s.io.
This is done. The billing report should be accessible to anyone in the k8s-infra group. To make it truly world-readable we'd need to recreate it on a non google-com email. Maybe a kubernetes.io gsuite account for publish docs like this? Will open an issue
from k8s.io.
Related Issues (20)
- k8s-infra-prow-build: Deploy KubeCost and integrate it with the existing KubeCost installation on EKS HOT 5
- image-promo job is hitting quota limits HOT 9
- Create secret for triage party HOT 3
- eks-prow-build-cluster: Decommission the canary cluster HOT 10
- eks-prow-build-cluster: Replace FluxCD with ArgoCD HOT 5
- peribolos jobs using incorrect token HOT 3
- k8s infra GCE pool is frequently exhausted HOT 6
- Manage the DNS for etcd.io using GitOps
- DNS REQUEST: dash.etcd.io HOT 1
- Umbrella Issue: Kubernetes Infrastructure Tech Debt HOT 3
- Migrate Infra code to separate repo called k/infrastructure HOT 4
- Detected cluster health issues in your AWS account 808842816990 HOT 4
- Unable to run terraform due to lack of permissions on custom roles HOT 5
- Migrate to tenv from tfswitch HOT 1
- Account used for aws-ebs-csi-driver ECR public repository HOT 14
- Update role in EBS CSI Public ECR Account (`406135939448`) to grant `ecr-public:SetRepositoryPolicy` HOT 4
- DNS REQUEST: kubebuilder.io HOT 9
- `kubeadm, kubelet` install file for MacOS exists in CDN? HOT 2
- https://pkgs.k8s.io/ does not support ipv6 HOT 16
- Migration to new vSphere environment HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k8s.io.