👋 Loving the work and development going on!

Working on a project which is going to utilise APIGEE and would be interested in utilising these modules.

Is there an item on the backlog to version control these? Ideally I don't want to be pulling from main 🚫

I've utilised the Release Please GitHub Action to achieve this before.

Using the samples/x-l7xlb template, I tried to setup an Apigee environment in my GCP organization.
I used the sample settings in https://github.com/apigee/terraform-modules/blob/main/samples/x-l7xlb/x-demo.tfvars, only filling the necessary project variables to use my existing project.

Running terraform apply showed an error after creating the Apigee instance, saying:

Error: Invalid index

  on main.tf line 75, in module "apigee-x-bridge-mig":
  75:   endpoint_ip = module.apigee-x-core.instance_endpoints[each.key]
    ├────────────────
    │ each.key is "euw1-instance"
    │ module.apigee-x-core.instance_endpoints is map of string with 1 element

The given key does not identify an element in this collection value.

Looking how the values flow and that the phrase "euw1-instance" is only mentioned as key in the apigee_instances map variable, this key is apparently lost in the process of creating the instance and is not a valid key in the map of instances eventually returned by apigee-x-core module.

Checking my GCP Console, I noticed that the instance is called instance-europe-west1. which is weird, since obviously, the instance is expected to be names like the key here. Digging into the code, I think I found a bug in where the name is retrieved from, after creating the instance.

1. x-l7xlb#main.tf#L64 passes the apigee_instances object right into the module, so the map is name => instance object
2. apigee-x-core's main.tf#L19 transforms this into a map of region => instance object. Do note, that this instance object does NOT transfer the key of apigee_instances into a name field, which becomes important, once the data is passed into the apigee module.
3. local.instances is passed into the apigee module here in line 83
4. Module apigee main.tf#L78 is a loop to create one Apigee instance per object in the var.instances retrieved from outside. Using coalesce() is checks whether the each.value object has a name attribute and if not, assigns the default name instance-${each.key}, which causes the default name instance-europe-west1 observed earlier.
5. Instance creation works fine nonetheless, since the module accounts for the lack of an explicit instance name - however, at this point, the instances all retrieved a standard name instead of the key value "euw1-instance" previously set in the configuration file, so these pieces of information diverged.
6. output.tf of apigee-x-core now returns the instance objects (correctly containing the standard names in the instance objects) as output.

Bug happens now:x-l7xlb's main.tf looks like this:

module "apigee-x-bridge-mig" {
  for_each    = var.apigee_instances
  source      = "../../modules/apigee-x-bridge-mig"
  project_id  = module.project.project_id
  network     = module.vpc.network.id
  subnet      = module.vpc.subnet_self_links[local.subnet_region_name[each.value.region]]
  region      = each.value.region
  endpoint_ip = module.apigee-x-core.instance_endpoints[each.key]
}

It utilizes var.apigee_instances to perform a loop. The desired instance names (like "euw1-instance") are the keys and the instance object is the value. However, the endpoint_ip line now tries to retrieve the endpoint_ip of the actual instances, which were just created, by their name in the return value of module.apigee-x-core whose keys are filled with the instances' names - the standard names.

Since the desired instance names from the configuration file were omitted between the apigee-x-core and apigee modules, causing the names to be set by the apigee module itself, the keys in var.apigee_instances cannot be used to retrieve the actual instance objects from the map.

There are multiple ways to fix this.

1. for_each uses module.apigee-x-core.instance_map (region =>instance object) as iterator. This changes the subnet and region lines, as each.value.region is now retrievable as each.key (may also be contained in the instance object; didn't check). Also, endpoint_ip uses module.apigee-x-core.instance_endpoints[each.value.name] as index.
2. Make apigee-x-core pass the keys of the apigee_instances object into the instance objects as name field. This should cause apigee to properly recognize the explicit assignment of an instance name and prevent it from falling back to the default instance name. In addition, this prevents the desired instance name from apigee_instances' keys and the actual instance names from diverging.

I prefer approach 1, since it sources the instance names from the module that creates the instances itself and returns their final names, instead of relying on the instances having a specific name from a config file key. It's just receiving the data directly from the source.
Edit: After testing this, I realized that terraform plan will fail, since the for_each argument is dynamic and cannot be determined at plan-time. So I prefer approach 2 of adding name = key in apigee-x-core#main.tf @ line 20 to pass the keys of apigee_instances as name field in the local.instances values.

Edit2: I also noticed that the output value instance_endpoints of apigee-x-core is documented to be a map of instance name => instance endpoint IP (see here). However, after applying solution 2 and using the hostname as a key, I still received the error saying that "the key doesn't identify an object in the map". Printing the object revealed that instance_endpoints is actually mapping region => instance endpoint IP, contrary to what the documentation says.

I stumbled upon this when testing for a project of mine, so I'll add a pull request, when I have the time and can properly test it. As I'm quite new to Terraform with GCP and this module family, I couldn't provide one along with this report.
If I overlooked something and caused the bug myself, I'm happy to be corrected :)

Edit3: Pull request added.

In some customer environments a single demo GCP project might be provisioned for experimenting with these terraform modules.

KMS keyrings cannot be deleted and live as long as the hosting GCP project.

When doing a terraform destroy and re apply the re apply will fail as the key rings already exist.

Consider use of terraform data keyword and boolean kms create flags so that the second time around the existing key rings are re-used.

Below suggestions are from Chat GPT-4 and NOT validated,

Quote:

"In this scenario, you can use a combination of data and resource blocks, along with a variable, to conditionally create the keyring or use an existing one. This way, when you run terraform apply after terraform destroy, the configuration will not attempt to create a new keyring if it already exists.

First, add a new boolean variable in your Terraform configuration file to decide whether to create a new keyring or use an existing one:

variable "create_keyring" { description = "Whether to create a new keyring or use an existing one." type = bool default = true }

Next, add the following data block to fetch the existing keyring:

data "google_kms_key_ring" "existing_keyring" { count = var.create_keyring ? 0 : 1 project_id = var.project_id location = var.keyring_location name = var.keyring_name }

Then, update the google_kms_key_ring resource block to conditionally create the keyring using the count parameter:

resource "google_kms_key_ring" "default" { count = var.create_keyring ? 1 : 0 project_id = var.project_id location = var.keyring_location name = var.keyring_name }

Now, create a local variable to hold the keyring ID, which will be used by other resources in the configuration:

locals { keyring_id = merge( google_kms_key_ring.default[*].id, data.google_kms_key_ring.existing_keyring[*].id )[0] }

Replace any reference to google_kms_key_ring.default.id in your configuration with local.keyring_id.

Finally, update the create_keyring variable in your Terraform configuration, setting it to false if you want to use the existing keyring:

module "kms-inst-disk" {
  # ...
  create_keyring = false
  # ...
}
`
" End Quote

Had an issue running samples/x-nb-psc-xlb on a newly formed GCP project. Terraform failed due to the default subnetwork already existing.

Unsure if this subnet exists in every new GCP project, but removing the creation of the vpc-ingress module in the sample fixed it in my case (see commit)

Hi,
I am trying to use mTLS envoy and I generated a self-signed cert as follows:

openssl req -newkey rsa:2048 -nodes -keyform PEM -keyout example-ca.key -x509 -days 3650 -outform PEM -out example-ca.crt openssl genrsa -out example.key 2048 openssl req -new -key example.key -out example.csr openssl x509 -req -in example.csr -CA example-ca.crt -CAkey example-ca.key -set_serial 100 -days 365 -outform PEM -out example.crt

for envoy I am using:

common_tls_context: validation_context: trusted_ca: filename: /opt/apigee/certs/example-ca.crt tls_certificates: - certificate_chain: filename: /opt/apigee/certs/example.crt private_key: filename: /opt/apigee/certs/example.key

for client:

openssl genrsa -out example-cli.key 2048 openssl req -new -key example-cli.key -out example-cli.csr openssl x509 -req -in example-cli.csr -CA example-ca.crt -CAkey example-ca.key -set_serial 101 -days 365 -outform PEM -out example-cli.crt

For all certs I used "app1.test.com" as a FQDN.

When I am trying to reach envoy from my internal VM I am receiving an error:

curl "https://app1.test.com/test" --cert example-cli.crt --key example-cli.key --cacert example.crt -vv

`* Expire in 3 ms for 1 (transfer 0x5619776d10f0)

• Trying 10.100.0.28...
• TCP_NODELAY set
• Expire in 200 ms for 4 (transfer 0x5619776d10f0)
• Connected to app1.test.com (10.100.0.28) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• successfully set certificate verify locations:
• CAfile: example.crt
  CApath: /etc/ssl/certs
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
• TLSv1.3 (IN), TLS handshake, Request CERT (13):
• TLSv1.3 (IN), TLS handshake, Certificate (11):
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
• TLSv1.3 (IN), TLS handshake, Finished (20):
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.3 (OUT), TLS handshake, Certificate (11):
• TLSv1.3 (OUT), TLS handshake, CERT verify (15):
• TLSv1.3 (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server did not agree to a protocol
• Server certificate:
• subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=app1.test.com
• start date: Mar 28 08:08:56 2022 GMT
• expire date: Mar 28 08:08:56 2023 GMT
• common name: app1.test.com (matched)
• issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=app1.test.com
• SSL certificate verify ok.
• SSL_write() returned SYSCALL, errno = 32
• Failed sending HTTP request
• Connection #0 to host app1.test.com left intact
  curl: (55) SSL_write() returned SYSCALL, errno = 32`

curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0

Below is the apigee_instances code and running it gives below error:

from variables.tf:

variable "apigee_instances" {
description = "Apigee Instances (only one instance for EVAL)."
type = map(object({
region = string
ip_range = string
environments = list(string)
}))
default = {
region = "us-central1"
ip_range = "10.x.x.x/24"
environments = [ "terraform-env1" ]
}
}

error:

Error: Invalid default value for variable
│
│ on variables.tf line 86, in variable "apigee_instances":
│ 86: default = {
│ 87: region = "us-central1"
│ 88: ip_range = "10.x.x.x/24"
│ 89: environments = [ "terraform-env1" ]
│ 90: }
│
│ This default value is not compatible with the variable's type constraint: element "region": object required.
╵

Hi,
I am using your code for psc/dns but for more than one dns peering to the apigee I am not able to use 7.x.x.x A record on apigee. Is there any limitation?

Some organizations have a GCP Org policy that requires VPC Flow Logs be used.

The samples here use older versions (v16.0.0) of net-vpc which do not support VPC flow logs.

The latest (master branch) does support VPC flow logs [2]

[2] https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/blob/09945a24ebb55124c6f39daee4dc067ef1b1a2ba/modules/net-vpc/subnets.tf#L116

The apigee-x-bridge-mig module uses cloud-foundation-fabric/compute-vm for bridge mig template formation. The bridge mig template gets its name from local variable bridge_name, which itself gets formed using the name of the GCP region to which the Apigee Instance belongs.

Some of the GCP regions have too long names, making the bridge_name even more longer. The bridge_name formed is utilized to create service_account in the upstream cloud-foundation-fabric/compute-vm module; specifically the account_id, which has a character limit of 28 characters, utilizes the bridge_name. If the bridge_name is too long, the Terraform module errors out with character limit exception.

A potential fix is to have the bridge_name shortened for GCP regions having long names.

Unable to pass multiple ssl certificate in module mig-l7xlb. URL: https://github.com/apigee/terraform-modules/blob/main/modules/mig-l7xlb/main.tf#L56
it accept string type variable, so how can I give multiple ssl_certificate in this module?

Currently if we try to deploy an Apigee X instance using the samples then there is no support for Hub and spoke architecture where the network components are in host project and the MIG and LB is in service project.

We can an error similar to this while configuring the MIG resource-

module.mig-l7xlb.google_compute_backend_service.mig_backend: Creating...
╷
│ Error: Error creating BackendService: googleapi: Error 400: Invalid value for field 'resource.backends[0].group': 'https://www.googleapis.com/compute/v1/projects/REDACTED/regions/us-central1/instanceGroups/apigee-bridge-mig-us-central1'. Cross project referencing is not allowed for this resource., invalid
│
│   with module.mig-l7xlb.google_compute_backend_service.mig_backend,
│   on modules/mig-l7xlb/main.tf line 26, in resource "google_compute_backend_service" "mig_backend":
│   26: resource "google_compute_backend_service" "mig_backend" {

https://github.com/apigee/terraform-modules/blob/main/samples/hybrid-gke/main.tf#L212

should read

iam_project_roles = { "${module.project.project_id}" = [ ... ] }

in order to extract the Project ID correctly from the module.

Fix warnings and errors with respect to Mega Linter linters.

Hi, I have set up the Apigee using the provided modules and the L7XLB module to have a load balancer. Everything is fine so far; the only issue is that the Integration section in the Develop menu is unavailable. So just wondering if you know what's the answer to that.
Thanks.

Provide a sample for SB PSC as documented here: https://cloud.google.com/apigee/docs/api-platform/architecture/southbound-networking-patterns-endpoints#create-the-psc-attachments

Could we include the option to use existing VPCs? I'd like it for the Northbound PSC Backend sample to start.
Looking at the Cloud Foundations module they support it via the the variable vpc_create

I added vpc_create = false to main.tf of the Northbound PSC sample and it worked.

I don't see the KMS keys are going through rotation or a value is set to enable that. Will ApigeeX support the KMS key rotation if we chance the module from apigee-x-core ?

PSC_Ingress_module is missing. Please add it to the module

Demonstrate how Apigee egress traffic can be routed via a firewall appliance.

TL;DR

Recently ip_range field was added to google_apigee_instance Terraform resource from google provider version 4.8.0.

This is in regards to the new ipRange field being added to organizations.instance API. The peeringCidrRange field is being deprecated.

Cloud foundation fabric v14.0.0 release

Cloud foundation fabric v14.0.0 was released in which the cidr_mask variable was replaced with ip_range variable for apigee-x-instance module. So need to update the modules in this repo to point to v14.0.0 of cloud foundation fabric modules.

Hello,
I have checked the mTLS configuration on the bridge VM and seems that something is missing in the envoy yaml file. Envoy doesn't check CA cert. You can try curl https://xxxxx --cert xxx.crt --key xxx.key -k and traffic will be forwarding but this is not mTLS. we have to also add --cacert xxx.crt. I assume we have to add something like this:

          request_headers_to_add:
          - header:
              key: "x-apigee-tls-client-raw-cert"
              value: "%DOWNSTREAM_PEER_CERT%"

The psc internal load balancer module creates a subnet, but there are scenarios were subnets are provisioned ahead of time by the team that manages the network.

For example, our org heavily relies on shared VPCS and the networking team provisions the subnets for the dev teams, and from my understanding, only one regional managed proxy can be active in a given region for a VPC.

It would be nice to support passing in a subnet for the proxy and skip the creation of it.

Issue

Currently, the Apigee X Organization's KMS Key Ring name is hard-coded to apigee-x-org. Because of this hard-coding, terraform apply fails when re-creating (after an Apigee X Organization has been destroyed) the Apigee X Organization using Terraform.

The reason for this behaviour is that a KMS Key Ring cannot be deleted. When Apigee X Organization and related resources are created using modules/apigee-x-core, KMS Key Rings are created for Apigee X Organization and each Apigee X Runtime Instance.

create a pattern that uses northbound PSC (preview) as documented in the Apigee Docs https://cloud.google.com/apigee/docs/api-platform/system-administration/psc-overview

It times out when creating the instance. Had to use gtimeout/timeout cmd. Please let me know if that's not the case with your testing.

Also, after the initial timeout, I wasn't able to delete the KMS keys and rest of the resource with 'destroy'. Had to create a new project for a fresh start.

hi folks, thanks for the awesome work on this.
I believe it would be incredibly helpful if each of the sample comes with a diagram explaining the component.

Why does the Northbound PSC ingress sample use two VPCs [1][2] ?

As per [3], only one VPC is needed to implement northbound ingress with PSC.

[1]

With the new PAYGv2 pricing model, the environment object has changed to include a new parameter - EnvironmentType (https://cloud.google.com/apigee/docs/reference/apis/apigee/rest/v1/organizations.environments#environmenttype) . Also, for PAYGv2, nodes will become irrelevant (https://cloud.google.com/apigee/docs/reference/apis/apigee/rest/v1/organizations.environments#nodeconfig). I understand we depend on the upstream Google TF provider, but these changes might change some TF scripts, depending on what they are using/configuring.

In order to associate Cloud Armor security configurations with the layer 7 load balancer created by the mig-x7xlb module, we would need to add a parameter not currently present in this module's interface.

To implement this, we would need to have access to define a variable to populate the security_policy field in the compute_backend_service resource.

This use case may be common enough to consider supporting at this level without the need to fork or create other workarounds.

Hi,

I have tried executing the below sample terraform code to provision ApigeeX evaluation stack. But, I notice that "Access Routing" is not being configured on executing this. Am I missing something? Please suggest.

https://github.com/apigee/terraform-modules/tree/main/samples/x-l7xlb

Attached are screenshots and terraform logs for reference.


terraform_apply_output.txt

Some breaking changes were introduced with the Google Provider 4.0.0 version (e.g. self_links removed).

We should bump our versions and dependencies as well.

See https://github.com/terraform-google-modules/cloud-foundation-fabric/releases/tag/v8.0.0 for more details.

Release Version for the Cloud Foundation Fabric Dependencies to be bumped to the latest stable.

Client (TLS) -> L4 XLB -> Proxy VM/Pods (mTLS) -> Apigee X (TLS)

Fix PSC Southbound Endpoint Attachment Module

(Issue)[https://github.com/apigee/terraform-modules/blob/9c99a5107a1c1a30e078894124276841f60a3945/modules/sb-psc-attachment/main.tf#L30]

organizations/ prefix needs to be added.

Client -> L7 XLB -> Proxy VM/Pods (Region 1) -> Apigee X (Runtime Instance Region 1)
Client -> L7 XLB -> Proxy VM/Pods (Region 2) -> Apigee X (Runtime Instance Region 2)