googlecloudplatform / cloud-foundation-toolkit Goto Github PK
View Code? Open in Web Editor NEWThe Cloud Foundation toolkit provides GCP best practices as code.
License: Apache License 2.0
The Cloud Foundation toolkit provides GCP best practices as code.
License: Apache License 2.0
The Kubernetes pipeline is an example of how to parallelize test suites.
Relevant templates: dm/templates/dataproc
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/dns_managed_zone
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Our current deployment looks something like this:
Two deployments in two projects.
The current template has a few problems with this:
It would be great if cross project sinks + some workaround for the dataset could be made. Perhaps it can do some project id matching inside the template and the same template can be deployed in both projects and it will be smart enough to know which resource to deploy in which project?
Relevant templates: dm/templates/kms
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/cloud_spanner
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Notes:
Relevant templates: dm/templates/pubsub
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Notes:
There would be an error "The reference 'masterAuth.clientKey' is not found" when users do not specify "issueClientCertificate" in config or the value of "issueClientCertificate" is false. That feature actually forces users to enable client certificate, which is not expected.
Support for Folder and Org level policies.
Request virtual binding endpoints.
org_policy
Relevant templates: dm/templates/dns_records
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
The terraform forseti template can configure the server config, while the DM one requires the user to create the bucket and upload the config themselves before calling the forseti template.
Add "compute.googleapis.com" to activate_apis in project/project/py
Hi,
I would like to request a CFT template for https://cloud.google.com/resource-manager/reference/rest/v1/projects/setIamPolicy.
It is a non-trivial update as it involves updatemasks and policy etags. Our main use case is to ensure all possible audit logs are being generated.
Currently we implement it via custom template: https://github.com/GoogleCloudPlatform/healthcare/blob/master/deploy/templates/data_project.py#L539
The following template is trying to create dataset, table, and view at the same time.
imports:
- path: templates/bigquery/bigquery_dataset.py
name: bigquery_dataset.py
- path: templates/bigquery/bigquery_table.py
name: bigquery_table.py
resources:
- name: pun-bq-dataset
type: bigquery_dataset.py
properties:
name: pun_bq_dataset
location: US
access:
- role: OWNER
userByEmail: <YOUR EMAIL>
- name: pun-bq-table
type: bigquery_table.py
properties:
name: pun_bq_table
datasetId: $(ref.pun-bq-dataset.datasetId)
schema:
- name: firstname
type: STRING
- name: lastname
type: STRING
- name: age
type: INTEGER
- name: pun-bq-view
type: bigquery_table.py
metadata:
dependsOn:
- pun-bq-table
properties:
name: pun_bq_view
datasetId: $(ref.pun-bq-dataset.datasetId)
view:
description: pun_bq_view
useLegacySql: false
query: "SELECT firstname, age FROM `<PROJECT_ID>.pun_bq_dataset.pun_bq_table`"
I'm not sure if this is the template issue or API issue but 75% of the time the deployment fails with error:
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1559665247835-58a81dd10d54c-8bfdbf01-df6ae411]: errors:
- code: RESOURCE_ERROR
location: /deployments/pun-bq-create1/resources/pun_bq_view
message: '{"ResourceType":"bigquery.v2.table","ResourceErrorCode":"404","ResourceErrorMessage":{"code":404,"errors":[{"domain":"global","message":"Not
found: Table <PROJECT_ID>:pun_bq_dataset.pun_bq_table","reason":"notFound"}],"message":"Not
found: Table <PROJECT_ID>:pun_bq_dataset.pun_bq_table","statusMessage":"Not
Found","requestPath":"https://www.googleapis.com/bigquery/v2/projects/<PROJECT_ID>/datasets/pun_bq_dataset/tables","httpMethod":"POST"}}'
When i run update
after the failure all resources get created (missing view gets created on retry).
I also tried changing
metadata:
dependsOn:
- pun-bq-table
to
metadata:
dependsOn:
- <PROJECT_ID>:pun_bq_dataset.pun_bq_table
but that didn't help either. It seems that dependOn
doesn't always behave as expected.
I've created this issue to track process of code base update.
Template/schema checklist:
Template batch 1:
Template batch 2:
Template batch 3:
Template batch 4:
Relevant templates: dm/templates/ip_reservation
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
roles/serviceusage.serviceUsageAdmin is needed for the DM SA.
Relevant templates: dm/templates/bastion
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/iam_member
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Notes:
Relevant templates: dm/templates/route
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/project
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Notes:
Relevant templates: dm/templates/bigquery
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/cloud_sql
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Notes:
Relevant templates: dm/templates/managed_instance_group
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Notes:
Relevant templates: dm/templates/route
Main issue: #47: DM templates refactoring: template and schema improvements
Context: #120 (comment)
Relevant templates: dm/templates/firewall
Main issue: #47: DM templates refactoring: template and schema improvements
Relevant: Firewall template broken
TODO list:
to golang impementation add --preview option support https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/master/dm/docs/userguide.md#the-update-action
Relevant templates: dm/templates/cloud_function
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Notes:
Relevant issue: Cloud functions python template error uploading functions
gcp-type for cloud function has a bit different logic for source fields. Sourcing logic that should be implemented:
Relevant templates: dm/templates/cloud_router
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/instance
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/autoscaler
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
e.g. sourceRanges is in the example but not in the fields, so the additionalProperties check causes it to fail.
cc @gruihuang
Relevant templates: dm/templates/network
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/backend_service
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
We should skip removing VPC if compute api is not in the list
There is a bug in activate_apis in project/project.py
Relevant templates: dm/templates/instance_template
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/healthcheck
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Currently go implementation support project id from command line --project and deployment yaml project: element, according to user guidee it should be:
Creating dataset & tables in a single run has a high chance for "dataset not found error" to occur.
dependsOn does not help
Relevant templates: dm/templates/gcs_bucket
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/gke
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
Relevant templates: dm/templates/folder
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
the sample diskImage in schema documentation does not work (e.g. "specify family/debian-9 to use the latest Debian 9 image projects/debian-cloud/global/images/family/debian-9").
I had to use the full projects/ path.
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1559621223205-58a779cfe4aba-8b821d49-371b680c]: errors:
- code: RESOURCE_ERROR
location: /deployments/data-protect-toolkit-resources/resources/work-machine-1
message: "{"ResourceType":"compute.v1.instance","ResourceErrorCode":"400"
,"ResourceErrorMessage":{"code":400,"errors":[{"domain":"global","
message":"Invalid value for field 'resource.disks[0].initializeParams.sourceImage':
\ 'family/debian-9'. The URL is malformed.","reason":"invalid"}],"message"
:"Invalid value for field 'resource.disks[0].initializeParams.sourceImage': 'family/debian-9'.
\ The URL is malformed.","statusMessage":"Bad Request","requestPath":"
https://www.googleapis.com/compute/v1/projects/umairidris-test42/zones/us-east1-a/instances\"\
,"httpMethod":"POST"}}"
Forseti now supports cloud asset inventory. The DM template does not have support for setting this however.
The terraform module (https://github.com/forseti-security/terraform-google-forseti) supports cai through the following vars:
Relevant templates: dm/templates/gke
Main issue: #47: DM templates refactoring: template and schema improvements
TODO list:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.