Open source project to help accelerate and ease observability setup on AWS environments
Home Page: https://aws-observability.github.io/terraform-aws-observability-accelerator/
License: Apache License 2.0
There should be the ability to easily monitor the AMP deployments that are created throughout the use of this accelerator.
A grafana dashboard showing CloudWatch vended metrics of the AMP workspace (ingestion rate, active metrics, etc).
Provide X-Ray receiver and exporters in the collector configurations.
Customer will be able to visualize traces for their applications using X-Ray or OTEL SDKs
Please add your vote to influence this request ( 👍🏾 )
Setting up GH actions and unit tests
OTEL config typos/misconfigurations can be difficult and time-consuming to correct. Configs should be validated before being checked in to the repository.
Run the otel config validator against all otel configs submitted.
n/a
All adot components are now supported in the validator. This can be used to ensure config viability.
Is there some specific reason you don't support the latest stable release of Terraform? https://github.com/hashicorp/terraform/releases/tag/v1.3.0
https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/main/versions.tf
examples/eks-cluster-with-vpc to support variable instance types and count
Setup GH Actions and use-case testing scenarios
AMP remote write should be taken from hashicorp/terraform-provider-aws#26120 while supporting cross-region scenarios
In order to avoid issues from post-publish manipulation of tagged artifacts, pre-commit supports "freezing" hooks to a particular version, but pinning to a "frozen" SHA. This ensures if the sha disappears or changes, the precommit hook doesn't install unsavory payloads.
Automatic updating can be performed via: pre-commit autoupdate --freeze
More info at: https://pre-commit.com/#pre-commit-autoupdate
References: aws-ia/terraform-repo-template#8
v1.4.1
Terraform v1.3.3
Ecountering this error, linked to ADOT operator v0.62.1-eksbuild.1
│ Error: unexpected EKS Add-On (eks-cluster-with-vpc:adot) state returned during creation: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
│ * : AccessDenied: clusterroles.rbac.authorization.k8s.io "opentelemetry-operator-manager-role" is forbidden: user "eks:addon-manager" (groups=["system:authenticated"]) is attempting to grant RBAC permissions not currently held:
│ {APIGroups:["networking.k8s.io"], Resources:["ingresses"], Verbs:["create" "delete" "get" "list" "patch" "update" "watch"]}
│
│
│ [WARNING] Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration
│
│ with module.eks_observability_accelerator.module.operator[0].aws_eks_addon.adot,
│ on ../../modules/add-ons/adot-operator/main.tf line 28, in resource "aws_eks_addon" "adot":
│ 28: resource "aws_eks_addon" "adot" {
│
╵
No response
to:
variable "enable_kube_state_metrics" {
type = bool
default = true
}
variable "kube_state_metrics_config" {
type = object({
create_namespace = optional(bool, true)
k8s_namespace = optional(string, "kube-system")
helm_chart_name = optional(string, "kube-state-metrics")
helm_chart_version = optional(string, "4.9.2")
helm_release_name = optional(string, "kube-state-metrics")
helm_repo_url = optional(string, "https://prometheus-community.github.io/helm-charts")
helm_settings = optional(map(string), {})
helm_values = optional(map(any), {})
})
}
variable "enable_node_exporter" {
type = bool
default = true
}
variable "node_exporter_config" {
type = object({
create_namespace = optional(bool, true)
k8s_namespace = optional(string, "prometheus-node-exporter")
helm_chart_name = optional(string, "prometheus-node-exporter")
helm_chart_version = optional(string, "2.0.3")
helm_release_name = optional(string, "prometheus-node-exporter")
helm_repo_url = optional(string, "https://prometheus-community.github.io/helm-charts")
helm_settings = optional(map(string), {})
helm_values = optional(map(any), {})
})
}
variable "enable_dashboards" {
type = bool
default = true
}Setup sets of checks to run locally before pushing commits
Provide Alerts visualization in Managed Grafana and create the alertmanager datasource
As the TF Grafana provider is defined inline instead of being passed as a parameter to the module, the module cannot be used with TF depends_on
│ Error: Module is incompatible with count, for_each, and depends_on
│
│ on main.tf line 184, in module "gitlab_runner_monitoring":
│ 184: module.autoscaling-eks.cluster_id
│
│ The module at
│ module.gitlab_runner_monitoring.module.eks_observability_accelerator is a
│ legacy module which contains its own local provider configurations, and so
│ calls to it may not use the count, for_each, or depends_on arguments.
│
│ If you also control the module
│ "git::https://github.com/aws-observability/terraform-aws-observability-accelerator.git?ref=tags/v1.5.0",
│ consider updating this module to instead expect provider configurations to
│ be passed by its caller.
Pass the Grafana provider as parameter to the module.
Using depends_on is necessary to ensure correct order in cluster with monitoring creation
Add support for ECS workloads
Setup infrastructure monitoring module with associated example
Port module and create an example for Nginx based workloads on EKS
from https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/observability/adot-amp-grafana-for-nginx
Please add your vote to influence this request ( 👍🏾 )
Thanks for putting this tool together! It packages some fantastic dashboards. However, I find it lacking in explaining node health.
Could you provide OOTB dashboards for node CPU/MEM/Disk utilization? Standard infrastructure health metrics would help a Kubernetes operator monitor the health of their nodes.
Kube-prometheus-stack has some node / instance dashboards that I'm a big fan of. Would love to see if we can bring those into this project.
Would like to be able to deploy the Prometheus and Grafana resources in different Regions of choice.
As a customer, I should have an
The Grafana API key is stored in .tf files and is not secure. All users will start cloning the repo and storing keys in .tf files that will be in public (cloned) repos. It will be good to update the stack to move the keys and any sensitive information into AWS Secrets Manager and read from there. The EKS and ECS Blueprints already follow this approach.
Store API keys and secrets in AWS Secrets Manager and update the TF stack to read the value from AWS Secrets Manager.
Add any other context or screenshots about the feature request here.
Documentation for collecting custom metrics in the EKS infrastructure module
Github pages
Would like to compose infra + java + other modules. Currently, there's a namespace conflict when trying to assemble multiple ones
# deploys the base module
module "eks_observability_accelerator" {
# source = "aws-observability/terrarom-aws-observability-accelerator"
source = "../../"
aws_region = var.aws_region
eks_cluster_id = var.eks_cluster_id
# deploys AWS Distro for OpenTelemetry operator into the cluster
enable_amazon_eks_adot = true
# reusing existing certificate manager? defaults to true
enable_cert_manager = true
# creates a new Amazon Managed Prometheus workspace, defaults to true
enable_managed_prometheus = local.create_new_workspace
# reusing existing Amazon Managed Prometheus if specified
managed_prometheus_workspace_id = var.managed_prometheus_workspace_id
managed_prometheus_workspace_region = null # defaults to the current region, useful for cross region scenarios (same account)
# sets up the Amazon Managed Prometheus alert manager at the workspace level
enable_alertmanager = true
# reusing existing Amazon Managed Grafana workspace
enable_managed_grafana = false
managed_grafana_workspace_id = var.managed_grafana_workspace_id
grafana_api_key = var.grafana_api_key
tags = local.tags
}
# https://www.terraform.io/language/modules/develop/providers
# A module intended to be called by one or more other modules must not contain
# any provider blocks.
# This allows forcing dependency between base and workloads module
provider "grafana" {
url = module.eks_observability_accelerator.managed_grafana_workspace_endpoint
auth = var.grafana_api_key
}
module "workloads_infra" {
source = "../../modules/workloads/infra"
eks_cluster_id = module.eks_observability_accelerator.eks_cluster_id
dashboards_folder_id = module.eks_observability_accelerator.grafana_dashboards_folder_id
managed_prometheus_workspace_id = module.eks_observability_accelerator.managed_prometheus_workspace_id
managed_prometheus_workspace_endpoint = module.eks_observability_accelerator.managed_prometheus_workspace_endpoint
managed_prometheus_workspace_region = module.eks_observability_accelerator.managed_prometheus_workspace_region
# optional, defaults to 60s interval and 15s timeout
prometheus_config = {
global_scrape_interval = "60s"
global_scrape_timeout = "15s"
}
tags = local.tags
depends_on = [
module.eks_observability_accelerator
]
}
module "workloads_java" {
source = "../../modules/workloads/java"
eks_cluster_id = module.eks_observability_accelerator.eks_cluster_id
dashboards_folder_id = module.eks_observability_accelerator.grafana_dashboards_folder_id
managed_prometheus_workspace_id = module.eks_observability_accelerator.managed_prometheus_workspace_id
managed_prometheus_workspace_endpoint = module.eks_observability_accelerator.managed_prometheus_workspace_endpoint
managed_prometheus_workspace_region = module.eks_observability_accelerator.managed_prometheus_workspace_region
# optional, defaults to 60s interval and 15s timeout
prometheus_config = {
global_scrape_interval = "60s"
global_scrape_timeout = "15s"
scrape_sample_limit = 2000
}
tags = local.tags
depends_on = [
module.eks_observability_accelerator
]
}
The above config produces this error
╷
│ Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "otel-prometheus-role" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "adot-collector-java": current value is "adot-collector-kubeprometheus"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "adot-collector-java": current value is "adot-collector-kubeprometheus"
│
│ with module.workloads_java.module.helm_addon.helm_release.addon[0],
│ on .terraform/modules/workloads_java.helm_addon/modules/kubernetes-addons/helm-addon/main.tf line 1, in resource "helm_release" "addon":
│ 1: resource "helm_release" "addon" {
│
╵
We dont have a kubernetes native way using Kubernetes Native CRDs to create AMP, AMG, Grafana Dashboards.
Customers who prefer kubernetes native way to create AMP, AMG and Grafana Dashboards in conjunction with GitOps approach will love it. With CrossPlane and GitOps, customers would use Kubernets native way for Observability mechanisms.
Core module that deploys ADOT Operator, AMP and AMG
As a customer, I would like to create a Managed Grafana workspace from scratch without having to manually create an API Key
support of k8s jobs (eg: batch or triggered by apache airflow for example) dashboard... eg: how many jobs completed? time taken..resource used/perf etc.
kube-prometheus-stack comes with lots of useful Prometheus scrape configuration (via ServiceMonitor abstraction) and corresponding out-of-the-box Grafana dashboards. One of them is for kube-apiserver
AWS EKS exposes the kube-apiseever metrics endpoint in Prometheus format as described in the official AWS documentation here
In "EKS Best Practices Guides - Reliability" AWS is recommending using this metrics here
I propose adding this scrape endpoint and the corresponding Grafana dashboard to AWS Obervability Accelerator
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
A clear and concise description of what you want to happen.
A clear and concise description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
Create a Grafana dashboard that can be used to track ADOT collector health along with alerts. It should cover basic utilization metrics along with pipelines and components metrics.
Please add your vote to influence this request ( 👍🏾 )
When We clone the Git Repo we are getting unwanted terraform files in the root directory which needs to be cleaned up as it causes confusion. I ended up running terraform init and deplpy from wrong root directory instead of transversing to the correct examples location.
Valid working terraform-aws-observability-accelerator terraform files and setup are under https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples
If possible we can clean up below files which will prevent users from accidentally running terraform from wrong place.
=> ~/Observablity/terraform-aws-observability-accelerator =>ll *.tf
-rw-r--r-- 1 sunilcsi staff 2331 Nov 1 20:11 locals.tf
-rw-r--r-- 1 sunilcsi staff 1473 Nov 1 20:11 main.tf
-rw-r--r-- 1 sunilcsi staff 1194 Nov 1 20:11 outputs.tf
-rw-r--r-- 1 sunilcsi staff 1986 Nov 1 20:11 variables.tf
-rw-r--r-- 1 sunilcsi staff 314 Nov 1 20:11 versions.tf
11:26:32=> ~/Observablity/myfork/terraform-aws-observability-accelerator =>
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
A clear and concise description of what you want to happen.
A module that can scrape envoy metrics, ingest them to Amazon Prometheus and build dashboards on Amazon Grafana
A clear and concise description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
Currently, To create Grafana API keys, you can do it on the Grafana console or use AWS Control Plane APIs. There is also a Terraform support for Amazon Managed Grafana API keys.
This issues is to address its integration in an end-to-end automated solution
Please add your vote to influence this request ( 👍🏾 )
The default version of the node-exporter used in the infra module is 2.0.3, which is old and should be updated.
The current latest version is 4.8.0.
However, this cannot be done by upgrading from v3 to v4.
This should be added to the troubleshooting section of the documentation.
Update the default chart version and docs.
The current version of AWS tflint rules is quite dated and should be updated.
Update to latest (0.21.1 as of this issue).
Provide a complete example for Java using the java workload module
Please add your vote to influence this request ( 👍🏾 )
Documentation guide for tracing configuration with AWS X-Ray #27
Github pages end to end doc fo tracing
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Currently, Observability accelerator only support Amazon EKS. Documentation around how to deploy ADOT collector on one/or multiple EC2 instances and integration with Amazon Managed prometheus and Grafana is sparse and requiring much heavy lifting.
A clear and concise description of what you want to happen.
AWS Observability accelerator can provide terraform module to deploy ADOT collector on one or more existing EC2 instances and integrate with both Amazon Managed Prometheus and Grafana.
A clear and concise description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
Presently the documentation (both README.md and PR template) point to the eks blueprint repo (in this org, so invalid locations). This should be updated to reflect this repository.
Update doc and template to point to this repository.
Terraform v1.2.9
on linux_amd64
Your version of Terraform is out of date! The latest version
is 1.3.1. You can update by downloading from https://www.terraform.io/downloads.html
(ins)[hendry@t14g3 existing-cluster-with-base-and-infra]$ cat terraform.tfvars
# (mandatory) AWS Region where your resources will be located
aws_region = "ap-southeast-1"
# (mandatory) EKS Cluster name
eks_cluster_id = "hello-eks"
# (optional) Leave it empty for a new workspace to be created
managed_prometheus_workspace_id = ""
(ins)[hendry@t14g3 existing-cluster-with-base-and-infra]$ terraform plan
data.aws_eks_cluster_auth.this: Reading...
data.aws_eks_cluster.this: Reading...
module.eks_observability_accelerator.data.aws_region.current: Reading...
data.aws_eks_cluster_auth.this: Read complete after 0s [id=hello-eks]
module.eks_observability_accelerator.data.aws_eks_cluster.eks_cluster: Reading...
module.eks_observability_accelerator.data.aws_partition.current: Reading...
module.eks_observability_accelerator.data.aws_caller_identity.current: Reading...
module.eks_observability_accelerator.data.aws_region.current: Read complete after 0s [id=ap-southeast-1]
module.eks_observability_accelerator.data.aws_partition.current: Read complete after 0s [id=aws]
data.aws_eks_cluster.this: Read complete after 0s [id=hello-eks]
module.eks_observability_accelerator.data.aws_eks_cluster.eks_cluster: Read complete after 0s [id=hello-eks]
module.eks_observability_accelerator.module.operator[0].data.aws_eks_addon_version.this: Reading...
module.eks_observability_accelerator.module.operator[0].data.aws_eks_addon_version.this: Read complete after 1s [id=adot]
module.eks_observability_accelerator.data.aws_caller_identity.current: Read complete after 1s [id=160071257600]
module.eks_observability_accelerator.module.operator[0].module.cert_manager[0].data.aws_iam_policy_document.cert_manager_iam_policy_document: Reading...
module.eks_observability_accelerator.module.operator[0].module.cert_manager[0].data.aws_iam_policy_document.cert_manager_iam_policy_document: Read complete after 0s [id=2873133587]
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│
│ on .terraform/modules/eks_observability_accelerator.operator.cert_manager/modules/irsa/versions.tf line 4, in terraform:
│ 4: experiments = [module_variable_optional_attrs]
│
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│
│ (and 3 more similar warnings elsewhere)
╵
╷
│ Error: Invalid index
│
│ on ../../locals.tf line 28, in locals:
│ 28: amg_ws_endpoint = var.managed_grafana_workspace_id == "" ? "https://${module.managed_grafana[0].workspace_endpoint}" : "https://${data.aws_grafana_workspace.this[0].endpoint}"
│ ├────────────────
│ │ module.managed_grafana is empty tuple
│
│ The given key does not identify an element in this collection value: the collection has no elements.
No response
Port module and create an example for HAProxy based workloads on EKS
from https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/observability/adot-amp-grafana-for-haproxy
Please add your vote to influence this request ( 👍🏾 )
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I tried using promtail example of aws-eks-blueprints and never work except with installing loki-stack helm charts, great if we can use loki with AMG. thanks
A clear and concise description of what you want to happen.
Add Promtail and Loki
A clear and concise description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
I really enjoy when helm charts I install give me a link to jump to the dashboard/UI that I can see after I install the helm chart.
The terraform output after the installation is complete today just shows you the AMP workspace (if it creates one). Ideally, it should give you a link to a Grafana workspace where the dashboards are installed, so you can click on the output and see the dashboards.
I'd like for the Observability Accelerator to be quicker to install. Right now it takes > 1-2 min to install, and ideally it would be great if it could be installed in 30s or less.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
A clear and concise description of what you want to happen.
Add kubecost add-ons
A clear and concise description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
A clear and concise description of what you want to happen.
A clear and concise description of any alternative solutions or features you've considered.
Add any other context or screenshots about the feature request here.
I am trying to deploy Observability Accelerator inside a fully private EKS cluster (that is, private endpoints for control plane and worker nodes deployed in private subnets). In this limited connectivity scenario the accelerator components cannot be deployed, since image pulling does not work and pods go into error.
Support for fully private EKS clusters.
The only alternative I have identified is to import all images that compose the solution into a private Docker repository which is reachable from within the EKS cluster using VPC Endpoints and then modifying the manifests to point to this new private images.
Have an example to setup Managed Grafana with SSO from scratch
We can benefit from a GitOps approach to create Grafana Datasource and Grafana Dashboards from GitOps repo on remote AMG. We can also think about deploying Grafana Operator to achieve this.
A GitOps approach to create Grafana Datasource and Grafana Dashboards from GitOps repo on remote AMG. This will help customers to manage a Single repo to create Dashboards on same kind across multiple clusters. This Decouples the Grafana Dashboard work for Ops Personas from Cloud Infra personas who creates this accelarator infrastructure. Using GitOps approach helps customers who wants to run 100s and 1000s of clusters to create Grafana Dashs.
Grafana Operator
Module and Example for Logs with OpenSearch
Please add your vote to influence this request ( 👍🏾 )
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.