terraform-aws-elk

This is a basic example for making an ELK stack server. Updated to use Amazon linux2 Packer AMI, and Auto-scaling.

It's 100% Open Source and licensed under the APACHE2.

Introduction

Build you AMI's first. The example uses HCL2 which is in beta, you will have to add your own region,vpc_id and subnet_id. Your subnet needs to ssh'able.

In the Packer folder :

$ packer build .\packer\
....

$ packer build ./packer/
....

https://www.elastic.co/guide/en/elastic-stack-get-started/7.6/get-started-elastic-stack.html#install-elasticsearch

With an AMI built the Terraform module will pick up the latest build version from your account.

Usage

To use this module add module.elk.tf as below into your Terraform code:

module "elk" {
  source              = "JamesWoolfenden/elk/aws
  version             = "0.2.15"
  ami_name            = var.ami_name
  ingress_cidrs       = ["0.0.0.0/0"]
  ssh_cidrs           = ["0.0.0.0/0"]
  instance_type       = var.instance_type
  private_subnet_tag  = var.private_subnet_tag
  vpc_cidr            = var.vpc_cidr
}

Replace 0.0.0.0/0 CIDRS with your own ranges, for more implmentation details see the full example in /example/examplea.

Costs

Monthly cost estimate

Project: .

 Name                                                    Monthly Qty  Unit         Monthly Cost

 module.elk.aws_autoscaling_group.elk
 └─ module.elk.aws_launch_configuration.elk
    ├─ Instance usage (Linux/UNIX, on-demand, t2.large)            0  hours               $0.00
    ├─ EC2 detailed monitoring                                     0  metrics             $0.00
    └─ root_block_device
       └─ Storage (general purpose SSD, gp2)                       0  GB-months           $0.00

 module.elk.aws_elb.elk
 ├─ Classic load balancer                                        730  hours              $21.46
 └─ Data processed                                       Cost depends on usage: $0.0084 per GB

 PROJECT TOTAL                                                                           $21.46

Check instance

This instance[and my user] is set-up to use ec2-instance-connect.

aws ec2-instance-connect send-ssh-public-key --region us-west-2 --instance-id i-0aa77051c763cd094 --availability-zone us-west-2b --instance-os-user ec2-user --ssh-public-key file://mynew_key.pub
ssh -i mynew_key [email protected]

Requirements

No requirements.

Providers

Name	Version
aws	n/a

Modules

No modules.

Resources

Name	Type
aws_autoscaling_attachment.elk	resource
aws_autoscaling_group.elk	resource
aws_elb.elk	resource
aws_iam_instance_profile.elk	resource
aws_iam_role.elk	resource
aws_iam_role_policy_attachment.elk	resource
aws_launch_configuration.elk	resource
aws_security_group.elk	resource
aws_security_group.lb	resource
aws_ami.elk	data source
aws_caller_identity.current	data source

Inputs

Name	Description	Type	Default	Required
ami_name	Partial string to match the target AMI	`string`	n/a	yes
asg	Settings to ensure that there's always an instance	`map`	{ "max_size": 1, "min_size": 1, "name": "ELK server" }	no
elb_name	(optional) describe your variable	`string`	`"elk-elb"`	no
encrypted	Root block device encryption	`bool`	`true`	no
ingress_cidrs	A range that is allowed to access ELK stack	`list(string)`	n/a	yes
instance_type	The AWS instance size for the ELK server	`string`	n/a	yes
ssh_cidrs	A range that is allowed to ssh on to the ELK stack	`list(string)`	n/a	yes
subnet_private	n/a	`any`	n/a	yes
subnet_public	n/a	`any`	n/a	yes
vpc_id	n/a	`string`	n/a	yes

Outputs

Name	Description
autoscaling	Details of autoscaling
launch_conf	Details of Launch Configuration
lb	Details of the load balancer
security_group	Details of the Security Group
security_group_lb	Details of the Security Group of the load balancer

Policy

This is the policy required to build this project:

The Terraform resource required is:

resource "aws_iam_policy" "terraform_pike" {
  name_prefix = "terraform_pike"
  path        = "/"
  description = "Pike Autogenerated policy from IAC"

  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "autoscaling:AttachLoadBalancers",
                "autoscaling:CreateAutoScalingGroup",
                "autoscaling:CreateLaunchConfiguration",
                "autoscaling:DeleteAutoScalingGroup",
                "autoscaling:DeleteLaunchConfiguration",
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeScalingActivities",
                "autoscaling:DetachLoadBalancers",
                "autoscaling:UpdateAutoScalingGroup"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateSecurityGroup",
                "ec2:DeleteSecurityGroup",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeImages",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeSecurityGroups",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AttachLoadBalancerToSubnets",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateLoadBalancerListeners",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:ModifyLoadBalancerAttributes"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "iam:AddRoleToInstanceProfile",
                "iam:AttachRolePolicy",
                "iam:CreateInstanceProfile",
                "iam:CreateRole",
                "iam:DeleteInstanceProfile",
                "iam:DeleteRole",
                "iam:DetachRolePolicy",
                "iam:GetInstanceProfile",
                "iam:GetRole",
                "iam:ListAttachedRolePolicies",
                "iam:ListInstanceProfilesForRole",
                "iam:ListRolePolicies",
                "iam:PassRole",
                "iam:RemoveRoleFromInstanceProfile"
            ],
            "Resource": "*"
        }
    ]
})
}

Help

To check is using systemd or not

ps -p 1

Configuration

Ensure in elasticsearch.yml

network.host: 0.0.0.0

and in kibana.yml

server.host: <servers ip>

todo

app config in user data
get data https://www.elastic.co/blog/getting-aws-logs-from-s3-using-filebeat-and-the-elastic-stack?blade=kibanafeed
automate config above

Got a question?

File a GitHub issue.

Contributing

Bug Reports & Feature Requests

Please use the issue tracker to report any bugs or file feature requests.

Copyrights

License

See LICENSE for full details.

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Contributors

James Woolfenden

jameswoolfenden / terraform-aws-elk Goto Github PK