An IAM Policy needs to be created with the policy document and then that needs to be attached to the role.

Workaround available: Merge something to main in covid-alert-metrics repo

in-app metrics and the keyserver should not live in the same VPC they can be thought of as two seperate services and should be isolated from each other.

Related to https://github.com/orgs/cds-snc/projects/9#card-64471053

Add a warning alarm action to the traffic change alarm.

Related #71

#42 is needed because we failed to wire up the network in #41. This caused the previous apply to fail.

We need to determine a way to prevent this in the future.

This is a Quality of Life enhancement.

aws_cloudwatch_metric_alarm.backoff_retry_average_duration will be updated in-place

~ resource "aws_cloudwatch_metric_alarm" "backoff_retry_average_duration" {
~ alarm_description = "This metric monitors average duration for the save_metrics lambda" -> "This metric monitors average duration for the backoff_retry lambda"
~ dimensions = {
~ "FunctionName" = "save-metrics" -> "backoff_retry"
}
id = "save-metrics-average-duration"
tags = {}
# (16 unchanged attributes hidden)
}

PR #80 added an API gateway traffic anomaly alarm with the goal of identifying traffic that is outside the realm of "normal" within a 24 hour day.

However, the alarm immediately began firing, and upon investigation, it was determined that the automatic band of "normal" traffic was not set appropriately:

Investigation is needed to determine how that low/high traffic band was determined and if it can be adjusted.

The server metrics ETL needs to be passed a metrics_token this token should come from an AWS Secret that is added during build time using Github Secrets.

The masked/unmasked metrics tasks take longer than 15 minutes to run, so won't work in the existing Lambdas. These should be moved to Fargate tasks in an ECS cluster.

It's no longer needed so we shouldn't pay for it.

"User: arn:aws:sts::005133826942:assumed-role/etl-st-cw-role/.... is not authorized to perform: iam:PassRole on resource: arn:aws:iam::005133826942:role/server_metrics_task_execution_role"

The secret value should not be in a container definition since the entire thing is a container definition.

Also the folder is named improperly.

Summary

The app will be sending new error metric payloads that need to be aggregated. Proposed payload is in cds-snc/covid-alert-metrics-payloads#5.

This metric includes a code and message:

Todo

• Determine how to aggregate code/message (unique identifier per error, new field or use existing field like state).
• Update covid-alert-metrics to a generate new CSVs based on error metrics.

Summary

• At 00:00:00 EST on 2021-06-27 the metrics API gateway 5xx alarm triggered.
• This was caused by a large traffic spike to the endpoint, which caused AWS to throttle 111 invocations of the save-metrics Lambda.
• Incident doc

Todo

• Determine if there was a reason for this traffic spike, or it if was a one-off.
• Determine if the 5xx metrics API gateway 5xx alarm threshold should be increased (is this a new normal traffic pattern).

This is part of server and should have proper labels

The API gateway can reject metrics payloads if they fail model validation. We should have an alarm if we start seeing this type of rejection.

TF_VAR_metrics_token is the only env variable besides AWS secrets

Rather than capturing save-metrics Lambda invocation errors in with the API Gateway 5xx errors, we should look for specific Lambda failures and create alarms based on their occurrence.

This will allow us to narrow down our focus to problems with the Lambda code and its supporting services.

Related #122, #124

To reproduce:

1. Add code change that does not rely on a tag version (e.g. terragrunt.hcl or Lambda code change).
2. Don't have that /env/production/$MODULE as part of the pull-request workflow matrix. As a result, there will be no plan run for the changes.
3. Merge the PR, which runs terragrunt run-all apply which will apply the changes from step 1.

We saw this happen in #185.

Possible workaround would be to have the Apply Production workflow only operate on tagged versions of the repo (rather than using main for non-module changes).

Ideally we should be able to use-detailed-exitcode to detect if the plan changed any resources or not but it looks like terragrunt doesn't support this option yet.

gruntwork-io/terragrunt#1166

We may be able to take advantage of OPA to determine if changes occurred and prevent commenting if formatting is valid and no changes are in the plan.

The new metric traffic pattern now causes throttled Lambda invocations at midnight EST. The current metic 5xx alarm threshold of 100 in 1 min is no longer sufficient.

Related #122

This shouldn't show up each time we do a PR.

# aws_default_network_acl.default will be updated in-place
~ resource "aws_default_network_acl" "default" {
      id                     = "acl-0a293e49ea86cc7ce"
    ~ subnet_ids             = [
        - "subnet-066a4cc52f615255c",
        - "subnet-07f256fc1e9b65732",
      ]
      tags                   = {
          "Name" = "metricsstaging_default_nacl"
      }
      # (5 unchanged attributes hidden)


      # (3 unchanged blocks hidden)
  }

When a plan fails we aren't commenting to the PR the results of the error.

We might want to do this for ease of use.

As per our Sev 1 Discussion this morning we need to add an alarm for the above metric.

If it hits above 3 in a day we should trigger a critical error
If it hits 1 in a day we should warn

• Public custom metric from aggregate lambda
• Create cloudwatch alarm based on above thresholds

Currently Falco is tightly coupled with the generic ecs task module. Ideally this should be able to be enabled/disabled on demand based on uses and to allow services to deploy if we run into issues with Falco.

Current 4xx CloudWatch threshold is too high and caused the project to not notice a spike in 4xx errors.

Related: incident-en-4xx-high

Fargate task definition doesn't support sourcePath; find an alternate means of declaring volumes.

The following repos are both managing the same policies:

The following policies are duplicated between repos:

aws_iam_policy.aggregate_metrics_update
aws_iam_policy.vpc_networking
aws_iam_policy.write_logs

Solution is to create a new backoff_retry policy and remove the above resources from remote state.

• Remove from staging
• Remove from production

Falco is the CNCF open source project for intrusion and abnormality detection for containers and cloud-native apps. Falco will generate security events when it finds abnormal behaviours, which are defined by a customizable set of rules.

Now that we have a nice clean Terragrunt repo for metrics we need to migrate over all the metrics repo from the keyserver.

• aggregate_metrics table
• raw_metrics table
• dead letter queue
• backoff retry lambda
• aggregate_metrics lambda
• raw_metrics lambda
• api_gateway
• cloudwatch alarms
• cloudwatch dashboard

This will allow the TF plan, OPA check and plan comment in the PR workflow to be more easily shared with other projects.

An example is this etl_lambda check. A terraform error was reported but the exit code 1 was hidden, likely by terragrunt.

Related #14

Add a new alarm that checks for a significant drop in traffic from one day to the next:

1. Aggregate previous day's traffic (24 hour period).
2. If traffic of current day has dropped by a significant percentage (5%, 10%?), trigger alarm.

This will catch cases where an app release has caused the metrics to drop unexpectedly.

Test in staging as Terragrunt still doesn't officially support Terraform v1.0.

Create new log metric filtering on Upload failed, unable to split large payload.

Example