Git Product home page Git Product logo

eks-airflow's Introduction

EKS Deployment

This repo provides Terraform templates for deploying a Kubernetes cluster against AWS using EKS

Requirements

  • Terraform
  • AWS CLI
  • Kubectl

Usage with Terraform CLI

Make sure environment variables AWS_PROFILE and AWS_DEFAULT_REGION point to the corresponding AWS credentials/region

  export AWS_PROFILE=sandbox 
  export AWS_DEFAULT_REGION=us-west-2
  1. Review terraform/variables.tf and adjust as needed

  2. Provision VPC and cluster

    	cd terraform
    	terraform init
    	terraform apply
  3. Authenticate to Cluster

    aws eks update-kubeconfig --name sandbox-eks-cluster --region $AWS_DEFAULT_REGION
    
  4. Register nodes to the cluster

    	terraform output config_map_aws_auth > config_map_aws_auth.yml
    	kubectl apply -f config_map_aws_auth.yml
  5. Validate by checking the created cluster nodes (it might take a minute to show 'Ready' status)

    	kubectl get nodes
    
  6. Provide access to other IAM users/roles that require it (see https://aws.amazon.com/premiumsupport/knowledge-center/amazon-eks-cluster-access/)

    	kubectl edit configmap aws-auth -n kube-system
    

    (IAM users should authenticate to the cluster after this step is completed)

    	aws eks --region $AWS_DEFAULT_REGION update-kubeconfig --name sandbox-eks-cluster
    

Note

This template does not currently include an Ingress controller. One can be deployed, eg for Nginx, using

	kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/aws/deploy.yaml
	kubectl get all -n ingress-nginx 

Notice that this is deployed outside of Terraform, so it needs to be manually deleted before removing the cluster.

Cleanup

โš  WARNING: You will delete your cluster with this steps.

If you want to destroy the cluster execute:

	terraform destroy
	

Airflow

See airflow/README.md for information about how to install airflow in this cluster.

Turn off EKS cluster

โš  WARNING: Your cluster will be turned off if you follow this steps.

If you want to turn off the instances go to Amazon's web console -> EC2 -> Autoscaling group and edit the configuration, setting up 0 instances for 'Desired capacity' and 'Minimum capacity' fields.

References:

eks-airflow's People

Contributors

mikaelapisani avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.