Relevant templates: dm/templates/iam_member

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs

Notes:

• There is no docs for "gcp-types/cloudresourcemanager-v1:virtual.projects.iamMemberBinding"

Our current deployment looks something like this:

• Log sink deployed on project-foo
• Get log sink writer and set in dataset
• Deploy dataset in project-bar

Two deployments in two projects.

The current template has a few problems with this:

• it doesn't allow cross project sinks
• due to the writer service account being in the form seviceAccount:[email protected] the bigquery access field cannot be set.

It would be great if cross project sinks + some workaround for the dataset could be made. Perhaps it can do some project id matching inside the template and the same template can be deployed in both projects and it will be smart enough to know which resource to deploy in which project?

The terraform forseti template can configure the server config, while the DM one requires the user to create the bucket and upload the config themselves before calling the forseti template.

Relevant templates: dm/templates/kms

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Add support for cross-project resource creation
• Fix resource names
• Add missing algorithms

Relevant templates: dm/templates/gcs_bucket

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Add uniqueItems: true to lists
• Switch to using type provider
• Add cross-project creation support
• Add additionalProperties: false for nested object
• Fix "bindings" schema
• Add support for "requesterPays"
• Add support for "acl", "billing", "cors", "defaultEventBasedHold", "defaultObjectAcl", "encryption", "iamConfiguration", "retentionPolicy": docs
• Fix resource name

to golang impementation add --preview option support https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/master/dm/docs/userguide.md#the-update-action

Relevant templates: dm/templates/instance_template

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add cross-project creation support
• Add additionalProperties: false for nested objects
• Add support for "networkInterfaces[].accessConfigs[]", "disks", "scheduling", "minCpuPlatform", "guestAccelerators", "shieldedInstanceConfig", "sourceInstance", "sourceInstanceParams": [docs]
• Fix resource name

Relevant templates: dm/templates/network

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation (both network and subnetwork)
• Add oneOf check for subnets: autoCreateSubnetworks should be exclusive with subnet list
• Fix network & subnetworks resources names
• Add support for "description", "routingConfig" to network
• Fix "secondaryIpRanges" definition in subnetwork
• Add basic schema unit tests

There is a bug in activate_apis in project/project.py

Relevant templates: dm/templates/autoscaler

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add cross-project creation support
• Add additionalProperties: false for nested object
• Fix resource name

The Kubernetes pipeline is an example of how to parallelize test suites.

Relevant templates: dm/templates/route

Main issue: #47: DM templates refactoring: template and schema improvements

Context: #120 (comment)

Currently go implementation support project id from command line --project and deployment yaml project: element, according to user guidee it should be:

1. The --project command-line option. If a project is specified via this option, all configs in the run use that project. This is a way of quickly overriding the project specified in a config file, which should be used with caution.
2. The project directive in the config file.
3. The CLOUD_FOUNDATION_PROJECT_ID environment variable.
4. The "default project" configured with the GCP SDK.

Add "compute.googleapis.com" to activate_apis in project/project/py

roles/serviceusage.serviceUsageAdmin is needed for the DM SA.

see GoogleCloudPlatform/deploymentmanager-samples#363

Relevant templates: dm/templates/dns_records

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Fix resource names

Creating dataset & tables in a single run has a high chance for "dataset not found error" to occur.

dependsOn does not help

We should skip removing VPC if compute api is not in the list

https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/master/dm/templates/firewall/firewall.py.schema

e.g. sourceRanges is in the example but not in the fields, so the additionalProperties check causes it to fail.

cc @gruihuang

Relevant templates: dm/templates/cloud_router

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Add support for "description"
• Switch to using type provider
• Add support for cross-project resource creation
• Add missing fields:
  • bgp (only asn is supported now)
  • nats
  • bgpPeers
  • interfaces
• Add basic schema unit tests

There would be an error "The reference 'masterAuth.clientKey' is not found" when users do not specify "issueClientCertificate" in config or the value of "issueClientCertificate" is false. That feature actually forces users to enable client certificate, which is not expected.

Support for Folder and Org level policies.
Request virtual binding endpoints.
org_policy

Relevant templates: dm/templates/instance

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add cross-project creation support
• Add additionalProperties: false for nested objects
• Add support for "description", "networkInterfaces[].accessConfigs[]", "networkInterfaces[].aliasIpRanges[]", "disks", "scheduling", "labels", "minCpuPlatform", "guestAccelerators", "deletionProtection", "hostname", "shieldedInstanceConfig", "shieldedInstanceIntegrityPolicy": docs
• Fix resource name

Relevant templates: dm/templates/route

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Fix resource names
• Fix arrays, objects and required fields (check for routeType + make old scheme optional)
• Add missing fields: "description", "nextHopInstance", "nextHopInstance", "nextHopNetwork", "nextHopGateway", "nextHopVpnTunnel"

Relevant templates: dm/templates/ip_reservation

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Add missing fields: "prefixLength", "networkTier"
• Fix field checks

Relevant templates: dm/templates/project

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Add "labels" support
• Add resource name prefix
• Update compute-beta to compute-v1
• Fix deprecated gcp-types/compute-v1:compute.subnetworks.setIamPolicy bindings
• Refactor default resources removal: remove code duplication
• Add usageExportBucket name prefix

Notes:

• There is no "gcp-type" for cloudbilling. There is only legacy deploymentmanager.v2.virtual.projectBillingInfo
• We should probably use gcs_bucket.py template for usage + add additional fields
• We should probably have a separate template for service accounts

Hi,

I would like to request a CFT template for https://cloud.google.com/resource-manager/reference/rest/v1/projects/setIamPolicy.

It is a non-trivial update as it involves updatemasks and policy etags. Our main use case is to ensure all possible audit logs are being generated.

Currently we implement it via custom template: https://github.com/GoogleCloudPlatform/healthcare/blob/master/deploy/templates/data_project.py#L539

cc @ocsig @morgante

the sample diskImage in schema documentation does not work (e.g. "specify family/debian-9 to use the latest Debian 9 image projects/debian-cloud/global/images/family/debian-9").

I had to use the full projects/ path.

ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1559621223205-58a779cfe4aba-8b821d49-371b680c]: errors:

• code: RESOURCE_ERROR
  location: /deployments/data-protect-toolkit-resources/resources/work-machine-1
  message: "{"ResourceType":"compute.v1.instance","ResourceErrorCode":"400"
  ,"ResourceErrorMessage":{"code":400,"errors":[{"domain":"global","
  message":"Invalid value for field 'resource.disks[0].initializeParams.sourceImage':
  \ 'family/debian-9'. The URL is malformed.","reason":"invalid"}],"message"
  :"Invalid value for field 'resource.disks[0].initializeParams.sourceImage': 'family/debian-9'.
  \ The URL is malformed.","statusMessage":"Bad Request","requestPath":"
  https://www.googleapis.com/compute/v1/projects/umairidris-test42/zones/us-east1-a/instances\"\
  ,"httpMethod":"POST"}}"

I've created this issue to track process of code base update.

Template/schema checklist:

• Version "1.0.0" is added to the schema file
• Legacy DM resources are changed to GCP type providers
• Cross-project resource creation is supported
• Name field is optional. DM resource name is used if it is missing
• API versions are upgraded to the latest ones (@ocsig should be notified in the case of major upgrade)
• All properties are checked for compatibility: there are differences between legacy DM resources and type providers, some fields are missing, etc.
• All APIs calls have links to relevant API documentation page. Schema info field has list of all methods used. Add list of APIs that need to be enabled
• Schemas provide extended validation:
  • All objects with schema have additionalProperties: false
  • Arrays have uniqueItems: true where applicable
  • ENUMs are up to date and present where necessary
  • Default values are revised
  • String values with strict checks have regex patterns - too difficult to do for now
• Resource names are correctly prefixed: some templates define too common names for resources that could interfere with each other. This might be something that we don't want though but users are struggling with
• Unit tests for schemas are added - too difficult to do for now
• Integration tests pass
• Manual resource creation is done (including cross-project creation)

Template batch 1:

• autoscaler
• cloud_function
• cloud_router
• cloud_spanner
• cloud_sql
• folder
• gcs_bucket
• instance
• instance_template
• managed_instance_group
• network
• network/subnetwork
• project

Template batch 2:

• backend_service
• bastion
• bigquery
• firewall
• gke
• healthcheck
• iam_custom_role
• iam_member
• ip_reservation
• nat_gateway
• kms
• pubsub
• route

Template batch 3:

• dataproc
• dns_managed_zone
• dns_records
• external_load_balancer
• forwarding_rule
• haproxy
• internal_load_balancer
• logsink
• network_peering
• org_policy
• shared_vpc_subnet_iam
• ssl_certificate
• target_proxy
• url_map

Template batch 4:

• cloud_tasks
• cloudbuild
• forseti
• interconnect
• interconnect_attachment
• runtime_config
• stackdriver_metric_descriptor
• vpn

The following template is trying to create dataset, table, and view at the same time.

imports:
  - path: templates/bigquery/bigquery_dataset.py
    name: bigquery_dataset.py
  - path: templates/bigquery/bigquery_table.py
    name: bigquery_table.py

resources:
  - name: pun-bq-dataset
    type: bigquery_dataset.py
    properties:
      name: pun_bq_dataset
      location: US
      access:
        - role: OWNER
          userByEmail: <YOUR EMAIL>
  - name: pun-bq-table
    type: bigquery_table.py
    properties:
      name: pun_bq_table
      datasetId: $(ref.pun-bq-dataset.datasetId)
      schema:
        - name: firstname
          type: STRING
        - name: lastname
          type: STRING
        - name: age
          type: INTEGER
  - name: pun-bq-view
    type: bigquery_table.py
    metadata:
      dependsOn:
      - pun-bq-table
    properties:
      name: pun_bq_view
      datasetId: $(ref.pun-bq-dataset.datasetId)
      view:
        description: pun_bq_view
        useLegacySql: false
        query: "SELECT firstname, age FROM `<PROJECT_ID>.pun_bq_dataset.pun_bq_table`"

I'm not sure if this is the template issue or API issue but 75% of the time the deployment fails with error:

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1559665247835-58a81dd10d54c-8bfdbf01-df6ae411]: errors:
- code: RESOURCE_ERROR
  location: /deployments/pun-bq-create1/resources/pun_bq_view
  message: '{"ResourceType":"bigquery.v2.table","ResourceErrorCode":"404","ResourceErrorMessage":{"code":404,"errors":[{"domain":"global","message":"Not
    found: Table <PROJECT_ID>:pun_bq_dataset.pun_bq_table","reason":"notFound"}],"message":"Not
    found: Table <PROJECT_ID>:pun_bq_dataset.pun_bq_table","statusMessage":"Not
    Found","requestPath":"https://www.googleapis.com/bigquery/v2/projects/<PROJECT_ID>/datasets/pun_bq_dataset/tables","httpMethod":"POST"}}'

When i run update after the failure all resources get created (missing view gets created on retry).

I also tried changing

    metadata:
      dependsOn:
      - pun-bq-table

to

    metadata:
      dependsOn:
      - <PROJECT_ID>:pun_bq_dataset.pun_bq_table

but that didn't help either. It seems that dependOn doesn't always behave as expected.

Relevant templates: dm/templates/dataproc

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Fix resource names
• Add missing fields: "labels", ".imageUri", ".isPreemptible", ".accelerators", "softwareConfig.optionalComponents", "encryptionConfig"
• Merge master, nodes and secondary nodes

Relevant templates: dm/templates/backend_service

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Add new fields: "backends[].maxRatePerEndpoint, .maxConnectionsPerEndpoint", "iap", "customRequestHeaders[]"
• Change healthCheck to healthChecks[]
• Add checks for INTERNAL/EXTERNAL LB schemes for all relevant fields
• Add checks for backends[].balancingMode

Relevant templates: dm/templates/healthcheck

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Add support for missing fields: "description", "name"

Relevant templates: dm/templates/cloud_spanner

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Add support for "labels"
• Switch to using type provider
• Make "name" optional, correctly handle it
• Add support for cross-project resource creation
• Update "instanceConfig" enum with new values
• Add basic schema unit tests
• Add additionalProperties: false for nested objects

Notes:

• TODO: check database "createStatement" & "extraStatements" fields
• TODO: check IAM policy for possible additional fields (both instance and database)

Relevant templates: dm/templates/cloud_sql

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add enum for "databaseVersion", "region", "settings.pricingPlan", "settings.replicationType"
• Add additionalProperties: false for nested object
• Fix "users" schema
• Add basic schemas unit tests

Notes:

• Some fields are missing in insert docs: "serverCaCert", "serviceAccountEmailAddress", "settings.databaseReplicationEnabled", "settings.settingsVersion", "onPremisesConfiguration", "maxDiskSize"
• Additional documentation is required as sometimes people don't try to use our template as examples:
  • SQL instance is being created in DM project
  • cloudsql - postgres

Relevant templates: dm/templates/gke

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Fix resource names
• Remove deprecated "nodeConfig", switch to "nodePools[].config"
• Remove deprecated "initialNodeCount", switch to "nodePools[].initial_node_count"
• Remove deprecated "privateCluster", "masterIpv4CidrBlock" -> "privateClusterConfig"
• Update parameters: "masterAuth", "loggingService", "monitoringService", "addonsConfig",
  "ipAllocationPolicy" and others
• Add support for "nodePools[]", "binaryAuthorization", "autoscaling", "networkConfig", "defaultMaxPodsConstraint", "resourceUsageExportConfig", "authenticatorGroupsConfig", "verticalPodAutoscaling", "tierSettings", "workloadIdentityConfig", "nodeConfig->diskType,sandboxConfig,shieldedInstanceConfig", "enableTpu", "databaseEncryption"
• Add uniqueItems: true and additionalProperties: false

Relevant templates: dm/templates/firewall

Main issue: #47: DM templates refactoring: template and schema improvements

Relevant: Firewall template broken

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Upgrade compute-beta to compute-v1
• Fix resource names
• Add schema to "rules": "name", "description", "priority", "sourceRanges", "destinationRanges", "sourceTags", "targetTags", "sourceServiceAccounts", "targetServiceAccounts", "allowed", "denied", "direction", "logConfig", "disabled"

See documentation here https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/blob/master/dm/docs/userguide.md#name

Relevant templates: dm/templates/pubsub

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Fix resource names
• Add topic fields: "labels"
• Add subscription fields: "pushConfig", "retainAckedMessages", "messageRetentionDuration", "labels", "expirationPolicy"

Notes:

• Type provider is broken:
  • It requires "topic" field as input for topic, while api schema and documentation state, that we should use "name"
  • It requires "subscription" field as input for subscription, while api schema and documentation state, that we should use "name"
• Setting "gcpIamPolicy" does not work for gcp-type
• Cross-project creation is not supported

GoogleCloudPlatform/deploymentmanager-samples#470

Forseti now supports cloud asset inventory. The DM template does not have support for setting this however.

The terraform module (https://github.com/forseti-security/terraform-google-forseti) supports cai through the following vars:

• enable_cai_bucket
• bucket_cai_lifecycle_age
• bucket_cai_location
• cai_api_timeout

Relevant templates: dm/templates/gke

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Add support for cross-project resource creation
• Add support for "stage" field

Relevant templates: dm/templates/dns_managed_zone

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Fix resource names

Relevant templates: dm/templates/cloud_function

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Upgrade API version: v1beta2 -> v1
• Add cross-project creation support (including upload.py)
• Add support for "labels", "environmentVariables" fields
• Rename "region" to "location" with backward compatibility + there is oneOf check
• Add new runtimes, change default to nodejs10
• Fix function upload code (it is not redeployed after code upload when using other project)
• Update source upload logic (see below)

Notes:

• Type provider is broken. It requires "parent" and "function" fields as input, while api schema and documentation state, that we should use "location" and "name"
• It does not support correct cross-project resource creation/update, cloudfunctions API must be enabled on the seed project due to API limitations
• Cloud build is triggered on the seed project, projectId parameter is ignored. It also has the same limitations as cloudfunctions
• Due to the fact that cloudbuild is scheduled on the seed project, cloudbuild.gserviceaccount.com service account should have storage.buckets.create on the target project
• "|| true" on bucket creation step hides all errors
• Any non-asci characters (especially binary files) break uploads

New source upload logic

Relevant issue: Cloud functions python template error uploading functions

gcp-type for cloud function has a bit different logic for source fields. Sourcing logic that should be implemented:

• If "sourceRepository" field is present, use it as-is
• If "sourceRepositoryUrl" field is provided, change it to "sourceRepository" format
• If "sourceArchiveUrl" is provided, use it as-is
• If "localUploadPath" is present, create cloudbuild build for function zip file. Use "sourceArchiveUrl" as target or generate it automatically (providing generated path via ""sourceArchiveUrl")
• If "sourceUploadUrl" is present, use it as-is
• All other parameter combinations should be restricted

Relevant templates: dm/templates/managed_instance_group

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add cross-project creation support
• Add additionalProperties: false for nested objects
• Fix resource name

Notes:

• Instance template config duplicates one from instance_template. See relevant issue
• We are using compute-beta for setAutoHealingPolicies

Relevant templates: dm/templates/bastion

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Fix resource names
• Fix instance.py outputs (it lacks internalIp/externalIP). Relevant commit
• Fix endless loops in tests

Relevant templates: dm/templates/bigquery

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• Add version, links to docs
• Switch to using type provider
• Add support for cross-project resource creation
• Add missing fields to datasets: "friendlyName", "defaultPartitionExpirationMs", "labels", "access"
• Add missing fields to tables: "description", "labels", "clustering", "requirePartitionFilter", "externalDataConfiguration", "encryptionConfiguration"
• Fix resource names
• Add uniqueItems: true and additionalProperties: false

Relevant templates: dm/templates/folder

Main issue: #47: DM templates refactoring: template and schema improvements

TODO list:

• add version, links to docs
• fix folder resource names: use base resource name as a template
• add oneOf check for folderId/orgId + fix incorrect schema for array
• fix "displayName" regex
• make "name" field optional
• use the same format for parent as in project + fixed name prefix to plural