int128 / terraform-aws-nat-instance Goto Github PK

Hi,

I've had issues with this not working, although it used to work.

It seems that when it deletes the default route:

# switch the default route to eth1
ip route del default dev eth0

The nat instance then looses all internet connectivity.

Does this still work for you?

This is to help 'ping' and 'aws s3 cp' command work. I could create a PR for this.

Hello,
I've encountered the following issue:

1. The NAT ec2 instance needs to be replaced due to failure or spot termination.
2. The original instance is removed and the ASG is spawning a new one.
3. In the meantime the ENI that was used by the instance is still not available for reattachment.
4. The new instance starts but fails to attach the ENI and gets stuck in a loop while not forwarding traffic.

This happens because the aws ec2 attach-network-interface command in the runonce.sh script to fails, but it still moves on to starting the snat service.

In the snat.sh script (ran by the snat.service) we have the following loop:

while ! ip link show dev eth1; do
  sleep 1
done

Which will run forever as the eth1 interface will never be available.

Possible solutions:

1. Add a check after aws ec2 attach-network-interface to see that the interface was actually attached (or check return code), if not, fail somehow.
2. Make it so the loop won't run forever so an additional script can be added by the users of the module to detect this and handle this however they see fit.

Hi!
tfsec complains (rightly so) about the following:

Result #3 HIGH Launch template does not require IMDS access to require a token
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 .terraform/modules/nat_instance/main.tf Lines 67-115
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   67  │ resource "aws_launch_template" "this" {
   68  │   name_prefix = var.name
   69  │   image_id    = var.image_id != "" ? var.image_id : data.aws_ami.this.id
   70  │   key_name    = var.key_name
   71  │
   72  │   iam_instance_profile {
   73  │     arn = aws_iam_instance_profile.this.arn
   74  │   }
   75  │
   76  │   network_interfaces {
   77  │     associate_public_ip_address = true
   78  │     security_groups             = [aws_security_group.this.id]
   79  │     delete_on_termination       = true
   80  │   }
   81  │
   82  │   tag_specifications {
   83  │     resource_type = "instance"
   84  │     tags          = local.common_tags
   85  │   }
   86  │
   87  │   user_data = base64encode(join("\n", [
   88  │     "#cloud-config",
   89  │     yamlencode({
   90  │       # https://cloudinit.readthedocs.io/en/latest/topics/modules.html
   91  │       write_files : concat([
   92  │         {
   93  │           path : "/opt/nat/runonce.sh",
   94  │           content : templatefile("${path.module}/runonce.sh", { eni_id = aws_network_interface.this.id }),
   95  │           permissions : "0755",
   96  │         },
   97  │         {
   98  │           path : "/opt/nat/snat.sh",
   99  │           content : file("${path.module}/snat.sh"),
  100  │           permissions : "0755",
  101  │         },
  102  │         {
  103  │           path : "/etc/systemd/system/snat.service",
  104  │           content : file("${path.module}/snat.service"),
  105  │         },
  106  │       ], var.user_data_write_files),
  107  │       runcmd : concat([
  108  │         ["/opt/nat/runonce.sh"],
  109  │       ], var.user_data_runcmd),
  110  │     })
  111  │   ]))
  112  │
  113  │   description = "Launch template for NAT instance ${var.name}"
  114  │   tags        = local.common_tags
  115  │ }
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
          ID aws-autoscaling-enforce-http-token-imds
      Impact Instance metadata service can be interacted with freely
  Resolution Enable HTTP token requirement for IMDS

  More Information
  - https://aquasecurity.github.io/tfsec/v1.8.0/checks/aws/autoscaling/enforce-http-token-imds/
  - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#metadata-options

Following your documentation, I was getting an error that azs was empty for module.vpc. Maybe it's just a newer version of the VPC module? I was able to run things after this change:

data "aws_availability_zones" "available" {
  state = "available"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.17.0"

  name               = "test-vpc"
  azs                  = formatlist("%s",data.aws_availability_zones.available.names)
  cidr                 = "172.18.0.0/16"
  private_subnets      = ["172.18.64.0/20", "172.18.80.0/20", "172.18.96.0/20"]
  public_subnets       = ["172.18.128.0/20", "172.18.144.0/20", "172.18.160.0/20"]
  enable_dns_hostnames = true
}

it works some what but not exactly......
in a private subnet instance I can telnet to google.com 443 and connect but when i traceroute from there it doesn't work

traceroute to google.com (142.250.66.110), 30 hops max, 60 byte packets
 1  ip-173-80-5-183.ap-east-1.compute.internal (173.80.5.183)  0.659 ms  0.638 ms  0.624 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

when I check the nat instance i get below

[ec2-user@ip-173-80-8-231 ~]$ systemctl status snat
● snat.service - SNAT via ENI eth1
   Loaded: loaded (/etc/systemd/system/snat.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2022-02-17 05:18:20 UTC; 3min 58s ago
  Process: 2438 ExecStart=/opt/nat/snat.sh (code=exited, status=0/SUCCESS)
 Main PID: 2438 (code=exited, status=0/SUCCESS)

Feb 17 05:18:12 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: + sysctl -q -w net.ipv4.conf.eth1.send_redirects=0
Feb 17 05:18:12 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: + iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
Feb 17 05:18:13 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: + rm -f /etc/sysconfig/network-scripts/ifcfg-eth0
Feb 17 05:18:13 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: + ip route del default dev eth0
Feb 17 05:18:13 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: + curl --retry 10 http://www.example.com
Feb 17 05:18:13 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Feb 17 05:18:13 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: Dload  Upload   Total   Spent    Left  Speed
Feb 17 05:18:13 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--...erver
Feb 17 05:18:13 ip-173-80-8-231.ap-east-1.compute.internal snat.sh[2438]: + systemctl restart amazon-ssm-agent.service
Feb 17 05:18:20 ip-173-80-8-231.ap-east-1.compute.internal systemd[1]: Started SNAT via ENI eth1.
Hint: Some lines were ellipsized, use -l to show in full.`

but i do have internet access from subnet

Could you please cut a release of the terraform registry module? I raised a PR #55 last month (thank you!) but I am having to reference the github repo rather than the terraform registry as the terraform registry does not have a release with this code in it?

Thanks again for all your work on this module, I appreciate you looking at this issue.

The default egress security group blocks UDP. This blocks any DNS queries made using Google, Cloudflare DNS etc.

https://github.com/int128/terraform-aws-nat-instance/blob/master/main.tf#L14

A rule should be added to allow UDP requests to fix this.

Exposing the autoscaling group name as an output would allow me to add an alarm on it.

The current snat.sh script doesn't work on Amazon Linux 2023.

Here's my first attempt at an alternative script for Amazon Linux 2023

#!/bin/bash -x

# wait for ens6
while ! ip link show dev ens6; do
  sleep 1
done

# NAT Instance Setup
# https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#NATInstance

# enable IP forwarding and NAT on ens6
sysctl -q -w net.ipv4.ip_forward=1
sysctl -q -w net.ipv4.conf.ens6.send_redirects=0
/sbin/iptables -t nat -A POSTROUTING -o ens6 -j MASQUERADE
service iptables save

# switch the default route to ens6

GATEWAY=$(ip route | awk '/default/ { print $3 }')
ip route add $GATEWAY dev ens6
ip route add default via $GATEWAY
ip route del default dev ens5

# wait for network connection
curl --retry 10 http://www.example.com

# re-establish connections
systemctl restart amazon-ssm-agent

There's a couple of areas which could use improvement such as:

1. Don't hardcode ens5 and ens6
2. Persist the routes after a reboot

If there is anyone else interested in having this module work with Amazon Linux 2023 comment here and i'll submit a PR.

i've been using this module for months now without any issues, but i am now receiving this quota error.

my full terraform code can be found here
https://github.com/aristosMiliaressis/pwnctl/blob/af81a6eeb7c8bac0d5474209b7da054cce2d704b/infra/vpc.tf#L89

i think this is a bug in the module cause i am not using any EC2 or spot instances anywhere else (aside from ECS Fargate) so i think only this module contributed to the specific service quota.

DueI noticed that this module creates two network interfaces & two public IPs. I guess the second network interface is for an elastic IP. Considering the new charges for public IPs, it would be nice to be able to not need the additional IP if possible. Not sure if it is.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update actions/checkout action to v4
• Update hashicorp/setup-terraform action to v3

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Hi!
another tfsec warning

Result #2 HIGH IAM policy document uses sensitive action 'ec2:AttachNetworkInterface' on wildcarded resource '*'
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 .terraform/modules/nat_instance/main.tf Line 199
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  194  │         {
  195  │             "Effect": "Allow",
  196  │             "Action": [
  197  │                 "ec2:AttachNetworkInterface"
  198  │             ],
  199  │             "Resource": "*"
  200  │         }
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
          ID aws-iam-no-policy-wildcards
      Impact Overly permissive policies may grant access to sensitive resources
  Resolution Specify the exact permissions required, and to which resources they should apply instead of using wildcards.

  More Information
  - https://aquasecurity.github.io/tfsec/v1.8.0/checks/aws/iam/no-policy-wildcards/
  - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

it can probably be solved by adding a condition to the policy (as per the ASG tag here) like

         "Condition": {
             "StringEquals": {
                "aws:resourceTag/Name": "nat-instance-${var.name}"
              }
          }

Hi! Thank you for the work!

But I am not be able to make this work though. The created nat instance does not have eth1.

Thanks!

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  name                 = "main_vpc"
  cidr                 = "10.1.0.0/16"
  azs                  = ["xxxxxxx"]
  private_subnets      = ["10.1.1.0/24"]
  public_subnets       = ["10.1.0.0/24"]
  enable_dns_hostnames = true
}


module "nat" {
  # https://registry.terraform.io/modules/int128/nat-instance/aws/latest?tab=inputs
  source  = "int128/nat-instance/aws"
  name = "nat"
  vpc_id                      = module.vpc.vpc_id
  public_subnet               = module.vpc.public_subnets[0]
  private_subnets_cidr_blocks = module.vpc.private_subnets_cidr_blocks
  private_route_table_ids     = module.vpc.private_route_table_ids
  enabled = true
}

resource "aws_eip" "nat" {
  network_interface = module.nat.eni_id
  tags = {
    "Name" = "nat-instance"
  }
}

Exposing an architecture variable will allow you to specify arm64 AMIs.

In data "aws_ami"...

  filter {
    name   = "architecture"
    values = [var.architecture]
  }

In variables.tf...

variable "architecture" {
  type = string
  default = "x86_64"
}

Happy to submit a PR if you think this will be additive to the project.

we've had a case where plans show up as changes in the resource "aws_autoscaling_group" "this"
resource, because the "name" tag in AWS, is like the 4th tag down, and in this module, the name tag is at the bottom. It then tries to update the tags, but has no effect.

I suggest using the for_each loop, instead of in the local in the variables.tf, to use it in the autoscaling group, and add multiple tag blocks , rather than a single tags attribute containing the array of tags.