chainguard-dev / terraform-infra-common Goto Github PK

This came up as a concern here: #328

The schema for containers is fairly large and is used a number of places:

• regional-go-service
• cron
• Github bots
• regional-service (in the linked PR)

There are subtle (mostly intentional) differences across these today, but these are becoming an ongoing maintenance concern.

@k4leung4 I think that we should move the alert policy we have for this in mono into cloudevent-recorder (scoped to those DTS jobs). Can you link to where we configure these today and/or shoot over a PR 🤞

The basic idea here is that we use _iam_binding in places where we believe a resource is an implementation detail of a particular module, and therefore the module has complete information about how that resource should be accessed (thus the use of _iam_binding vs. _iam_member).

Build on this idea, we can actually add alert policies for each of these resources to detect and flag anomalous usage. As a proof-of-concept, I added one here (sorry, private repo): https://github.com/chainguard-dev/octo-sts/blob/c737ecae7dd57c2fc340f51bf6aa9e95adfbdd20/iac/main.tf#L124-L176

We should audit this repo for uses of _iam_binding (or cases that should be!) and add audit log alerting to each.

Something like this:

  message_storage_policy {
    allowed_persistence_regions = [each.key]
  }

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_topic#allowed_persistence_regions

https://cloud.google.com/pubsub/docs/resource-location-restriction

ref: #139

With this we should be able to check things like:

This was @rawlingsj's idea, to have a manually triggerable job for brokers that would re-publish all the items in the DLQ, to trigger a retry.

This trigger could support filtering what kinds of events to re-publish if we want.

We might want to let callers specify buckets somehow, so that requests that take >10s aren't all grouped together into the largest bucket

https://cloud.google.com/pubsub/docs/subscription-message-filter#boolean_operators_for_the_filter_expression

Today we have:

  filter        = { "type" : "dev.chainguard.bar" } // Only trigger on bar-type events
  filter_prefix = { "source": "baz/" }              // ...coming from baz

which translates down to

filter = "attributes.ce-type=\"dev.chainguard.bar\" AND hasPrefix(attributes.ce-source, \"baz/\")"

This means it's not possible to easily subscribe to 2+ event types, or 2+ prefixes, etc., even though this is possible from PubSub's side.

• https://cloud.google.com/monitoring/dashboards/show-events
• https://cloud.google.com/monitoring/dashboards/event-types#run-deployment

This will be useful for the cloudevent-recorder

We should trigger an alert whenever things get put onto a cloudevent trigger's DLQ.

The minimum number of retries is 5, which is a strong signal that the handler is unhappy with the event it received and we should make our handler handle things a bit more gracefully.

cc @tcnghia

You had some cool metrics that would be cool to include as tiles, and wrap into a section for inclusion in dashboards like the cloudevent-recorder.

GCS is able to publish to PubSub. We should have a lil service subscribe to those events and re-publish them (possibly without transformation?) to a cloudevent broker so that regional-go-services can subscribe in the standard way.

Inputs:

• bucket name
• broker name
• regions? Should it just exist in the bucket region?

resource "google_storage_notification" "notification" {
  bucket         = data.google_storage_bucket.bucket.name
  payload_format = "JSON_API_V1"
  topic          = google_pubsub_topic.topic.name
  event_types    = ["OBJECT_FINALIZE", "OBJECT_DELETE"]
}

This should have a depends_on:
https://github.com/chainguard-dev/terraform-cloudrun-glue/blob/345c3e95943d76fd3a0c6b8ee88dd9573679c6f6/regional-go-service/main.tf#L128-L137