A sample project that deploys an EKS Cluster following a set of best practices with options to install additional addons. Easy deployment of the EBS CSI Driver, EFS CSI Driver, FluentBit Centralized Logging using Cloudwatch, Cluster Autoscaler, ALB Ingress Controller, Secrets CSI Driver and Network Policy Engine.
- AWS CDK >= 2.3.0 check Getting Started with AWS CDK to setup your CDK environment. Run
cdk --version
to check the CLI version.
git clone https://github.com/aws-samples/amazon-eks-using-cdk-typescript.git
# install dependant packages
npm install
# If you have not used cdk before, you may be advised to create cdk resources
cdk bootstrap aws://ACCOUNT_ID/REGION
# check the diff before deployment to understand any changes, on first run all resources will created
cdk diff
# Deploy the stack, you will be prompted for confirmation for creation of IAM and Security Group resources
cdk -c cluster_name=myfirstcluster deploy --all
Go to the EC2 Console and look for an instance Name: EKSBastionHost
. Right click on the instance and select connect
Select Session Manager and click the Connect
button
In the new window run the following command to configure kubectl with EKS Cluster. Replace CLUSTER_NAME and REGION_NAME with the proper values.
aws eks update-kubeconfig --name CLUSTER_NAME --region REGION_NAME
To destroy the cluster run the command below
cdk destroy -c cluster_name=myfirstcluster --all
cdk deploy -c cluster_name=mycluster --all
cdk deploy -c cluster_name=mycluster EKSStack
cdk deploy -c cluster_name=mycluster -c stack_prefix="dev-" --all
All cdk stacks will contain prefix ex. dev-EKSStack dev-EKSK8sBaseline
Tag existing VPC with proper EKS Tags for ALB/NLB auto discovery
cdk deploy -c use_vpc_id=vpc-1234567 -c cluster_name=mycluster --all
Requires VPC to have private subnets
cdk deploy -c cluster_name=mycluster --all --parameter EKSK8sBaseline:ebsDriverParameter=true --parameter EKSK8sBaseline:albDriverParameter=false
If using stack_prefix context, add prefix to parameters. ex dev-EKSK8sBaseline:ebsDriverParameter=true
Parameters can be configured via Cloudformation console or via cdk deploy --parameter STACK_NAME:PARAMETER_NAME=PARAMETER_VALUE
Stack | Parameter | Default | Values | Description |
---|---|---|---|---|
EKSStack | k8sVersion | 1.21 | Valid Values | Kubernetes Version to deploy |
EKSStack | eksLoggingOpts | api,audit,authenticator, controllerManager,scheduler |
Valid Logging Types | List of options to enabled, "" to disable control plane logs |
EKSNodeGroups | nodegroupMax | 10 | Number | Max number of worker node to scale up for nodegroup |
EKSNodeGroups | nodegroupCount | 2 | Number | Desired number of worker nodes for nodegroup |
EKSNodeGroups | nodegroupMin | 2 | Number | Min number of worker node to scale down for nodegroup |
EKSNodeGroups | nodegroupInstanceType | t2.medium | String | Instance Type to be used with EKS Managed Nodegroup ng-1 |
EKSNodeGroups | nodeAMIVersion | 1.21.2-20210722 | Valid Values | AMI Release Version |
EKSK8sBaseline | ebsDriver | false | true,false | Deploy EBS CSI Driver |
EKSK8sBaseline | albDriver | true | true,false | Deploy ALB Ingress Controller |
EKSK8sBaseline | efsDriver | false | true,false | Deploy EFS CSI Driver |
EKSK8sBaseline | fluentBit | true | true,false | Deploy FluentBit driver to output logs to centralized location |
EKSK8sBaseline | secretsDriver | false | true,false | Deploy AWS Secret/Parameter CSI Driver |
EKSK8sBaseline | networkPolicyEngine | false | true,false | Deploy Calico Network Policy Engine |
EKSK8sBaseline | clusterAutoscaler | true | true,false | Deploy Cluster Autoscaler |
EKSK8sBaseline | containerInsights | false | true,false | Deploy Container Insights using OpenTelemetry |
EKSK8sBaseline | metricServer | true | true,false | Deploys Metric Server |
Context can only be configured at synthesis time via cdk deploy -c CONTEXT_NAME=CONTEXT_VALUE
or editing cdk.json file.
Context | Default | Values | Description |
---|---|---|---|
eks-addon-vpc-cni-version | v1.9.0-eksbuild.1 | AWS CLI* | Amazon VPC Container Network Interface (CNI) plugin |
eks-addon-kube-proxy-version | v1.21.2-eksbuild.2 | AWS CLI* | K8s Network Proxy |
eks-addon-coredns-version | v1.8.4-eksbuild.1 | AWS CLI* | K8s Cluster DNS |
cluster-autoscaler-helm-version | 9.10.3 | Values | Management of k8s worker node scaling |
aws-load-balancer-controller-helm-version | 1.2.3 | Values | AWS Ingress Controller (ALB/NLB) |
aws-ebs-csi-driver-helm-version | 2.0.1 | Values | EBS CSI Driver |
aws-efs-csi-driver-helm-version | 2.1.4 | Values | EFS CSI Driver |
secrets-store-csi-helm-version | 0.1.0 | Values | Secrets CSI Driver |
aws-calico-helm-version | 0.3.5 | Values | Calico Network Policy Engine |
cluster_name | myfirstcluster | String | Name of EKS Cluster |
use_vpc_id | String | (Optional) Use existing VPC to deploy resources | |
stack_prefix | String | (Optional) Prefixes to add to underlying Cloudformation stack names |
*Run aws eks describe-addon-versions --kubernetes-version 1.21
to see full compatible list
Provides an example of EKS best practices in Infrastructure as Code
- Make the EKS Cluster Endpoint private
- IAM Roles for Service Accounts(IRSA) for deployable addons (ex. ALB Ingress Controller)
- Update the aws-node daemonset to use IRSA via EKS AddOns
- Option to enable Audit Logs
- Option to enable network policies using Calico engine
- Use AWS KMS for envelope encryption of kubernetes secrets
- Minimize access to worker node use SSM Manager
- Deploy workers onto private subnets*
- ECR Private Endpoints*
- Option to enable Cluster Autoscaler
- EKS Console Read Only Group called eks-console-dashboard-full-access-group
- EBS Encryption of Worker and Bastion Instances
*Use of vpc created by stack
Helm Chart versions can be configured by editing the cdk.json before deployment. The versions can be listed by using helm commands. Example for Cluster Autoscaler:
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm search repo autoscaler --versions
Another way to identity Helm Chart versions is looking at that index.yaml of Chart repositories for version information.
Cluster Autoscaler - Scalability
Deploys Cluster Autoscaler chart version 9.10.3 with app version 1.21.0 using Helm chart. A full list of chart versions with app versions can be found here
Default configuration below, additional configurations can be edited in k8s-baseline.ts:
extraArgs: {
// https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca
'skip-nodes-with-system-pods': false,
'skip-nodes-with-local-storage': false,
'balance-similar-node-groups': true,
//How long a node should be unneeded before it is eligible for scale down
'scale-down-unneeded-time' : '300s',
//How long after scale up that scale down evaluation resumes
'scale-down-delay-after-add':'300s'
}
ALB Ingress Controller - Scalability
Deploys ALB Ingress Controller chart version 1.2.3 with app version 2.2.1 using Helm. A full list of chart versions with app versions can be found here
EBS CSI Driver - Persistent Data
Deploys EBS CSI Driver Chart version 2.0.1 with app version 1.1.1 using Helm. A full list of chart versions with app versions can be found here
Needs user to deploy EBS StorageClass for use with pods. Example steps in EBS CSI Driver docs
EFS CSI Driver - Persistent Data
Deploys EFS CSI Driver chart version 2.1.4 with app version 1.3.2 using Helm. A full list of chart versions with app versions can be found here
Needs user to create EFS Volume and provision a storageclass for use with pods. Example steps in EFS CSI Driver docs
Secrets CSI Driver - Security
Deploys the Secrets CSI Driver chart version 0.1.0 with app version 0.1.0 using Helm.A full list of chart versions with app versions can be found here
Requires pods to have ServiceAccount with proper permissions to get values from AWS Secrets Manager or AWS Parameter Store. Example steps in AWS Blog post, continue from step 4.
Automated rotation for the driver using the rotation reconciler is turned off by default. Uncomment the below lines in secretsCsiHelmChart
k8s-baseline.ts
values: {
grpcSupportedProviders: 'aws',
// alpha feature reconciler feature
rotationPollInterval: 3600
enableSecretRotation: true
},
Network Policy Engine - Security
Deploys Calico Network Policy Engine chart version 0.3.5 with app version 1.15.1 using Helm.
Increases default cpu/memory limits to be
resources: {
limits: {
memory: '256Mi',
cpu: '500m',
},
},
Container Insights - Observability
Deploys Container Insights on Amazon EKS Cluster by using the AWS Distro for OpenTelemetry (collectoramazon/aws-otel-collector:v0.11.0). The visualized metrics can be found on Cloudwatch -> Insights -> Container Insights. The logs can be found at /aws/containerinsights/CLUSTER_NAME/performance. List of available metrics that can be collected available here
Configurations are applied via manifest, edits can be made following the documentation
FluentBit - Observability
Deploys FluentBit daemonset using manifests. The configuration is found in the manifest. The FluentBit conf files are stored in ConfigMap k8s definition. Additional configuration like log_retention_days, log_format, role_arn(cross-account) can be configured by editing conf files.
IAM permission to read/write permission to log group: /aws/containerinsights/${CLUSTER_NAME}
. Editable in k8s-baseline.ts
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/containerinsights/${CLUSTER_NAME}/*",
"arn:aws:logs:${REGION}:${ACCOUNT}:log-group:/aws/containerinsights/${CLUSTER_NAME}/*:log-stream:*"
],
"Effect": "Allow"
}
]
}
Include Logs from:
/var/log/containers/*.log
Exclude Logs from:
/var/log/containers/aws-node*
/var/log/containers/kube-proxy*
Output to:
Cloudwatch Log group /aws/containerinsights/${CLUSTER_NAME}/application
Include Logs from:
systemd:docker.service
systemd:kubelet.service
/var/log/containers/aws-node*
/var/log/containers/kube-proxy*
Exclude Logs from:
N/A
Output to:
Cloudwatch Log group /aws/containerinsights/${CLUSTER_NAME}/dataplane
Include Logs from:
/var/log/secure
/var/log/messages
/var/log/dmesg
Exclude Logs from:
N/A
Output to:
Cloudwatch Log group /aws/containerinsights/${CLUSTER_NAME}/host
A list of recommendations to further enhance the environment.
Least Privileged Access - Security
A readonly K8s Group is created called eks-console-dashboard-full-access-group
. Adding the below code snippet to the sample will add an existing Role to view eks-console-dashboard-full-access-group
RBAC group. See eks-cdk-js-stack.ts for example.
// Add existing IAM Role to Custom Group
this.awsauth.addRoleMapping(
Role.fromRoleArn(
this,
"Role_Admin",
`arn:aws:iam::${this.account}:role/Admin`
),
{
groups: ["eks-console-dashboard-full-access-group"],
username: `arn:aws:iam::${this.account}:role/Admin/{{SessionName}}`,
}
);
// Add existing IAM User to Custom Group
this.awsauth.addUserMapping(
User.fromUserArn(
this,
"Role_Admin",
`arn:aws:iam::${this.account}:user/Admin`
),
{
groups: ["eks-console-dashboard-full-access-group"],
username: `arn:aws:iam::${this.account}:user/Admin`,
}
);
The error below can be resolved by adding the iam user/role to eks-console-dashboard-full-access-group
k8s group.
Bastion Host Permissions - Security
The sample deploys a bastion host with outbound to 0.0.0.0
with outbound access on port 443
. The IAM role associated with the ec2 instance is registered as Admin role for EKS Cluster. SSM is able to log session data to Cloudwatch Logs or S3, see docs for more details. The end user permissions can be scoped to specific instance ids or tags.
Secrets CSI Driver - Security
Each pods requires a Service Account with IAM Role to access AWS Secrets Manager or AWS Parameter Store resources.
Sample permissions scoped to Secrets/Parameters with tags ekscluster:myclustername for dynamic access permissions.
const secretsManagerPolicyStatement = new PolicyStatement({
resources: ["*"],
actions: ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
conditions: {
StringEquals: {
"secretsmanager:ResourceTag/ekscluster": `${props.eksCluster.clusterName}`,
},
},
});
const parameterManagerPolicyStatement = new PolicyStatement({
resources: ["*"],
actions: ["ssm:GetParameters"],
conditions: {
StringEquals: {
"ssm:ResourceTag/ekscluster": `${props.eksCluster.clusterName}`,
},
},
});
const awsSA = new ServiceAccount(this, "awsSA", {
cluster: props.eksCluster,
name: "my-deployment",
namespace: "default",
});
const secretsManagerPolicy = new Policy(this, "secrets-manager-policy", {
statements: [parameterManagerPolicyStatement, secretsManagerPolicyStatement],
policyName: `${props.eksCluster.clusterName}-secrets`,
roles: [awsSA.role],
});
IMDSv2 Configuration - Security
The sample deploys Managed Node group with IMDSv1 and IMDSv2 enabled with hop count: 2 to ensure broadly levels of compatibility.
// Not all components are IMDSv2 aware. Ex. Fluentbit
metadataOptions: {
httpTokens: 'optional',
httpPutResponseHopLimit: 2,
},
To enforce IMDSv2 only, change to below code.
metadataOptions: {
httpTokens: 'required',
httpPutResponseHopLimit: 1,
},
The configuration for the FluentBit is set to create LogGroup and LogStreams with log retention=forever. The full set of options can be found here.
A sample output snippet for cross-account and log retention days (30 days).
[OUTPUT]
Name cloudwatch_logs
Match application.*
region ${AWS_REGION}
log_group_name /aws/containerinsights/${CLUSTER_NAME}/application
log_stream_prefix ${HOST_NAME}-
auto_create_group true
extra_user_agent container-insights
log_retention_days 30
role_arn arn:aws:iam::123456789:role/role-name
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.