gruntwork-io / terraform-aws-utilities Goto Github PK

View Code? Open in Web Editor NEW

212.0 23.0 77.0 10.84 MB

A collection of useful Terraform utilities

License: Apache License 2.0

HCL 97.89% Go 0.98% Python 1.05% Shell 0.08%

terraform-aws-utilities's Introduction

Terraform Utility Modules

This repo contains miscellaneous utility and helper modules for use with Terraform.

What is in this repo

This repo provides a Gruntwork IaC Package and has the following folder structure:

modules: This folder contains the main implementation code for this repository, broken down into multiple standalone modules.
examples: This folder contains examples of how to use the modules.
test: Automated tests for the modules and examples.

The following modules are available:

join-path: This module can be used to join a list of given path parts into a single path that is platform/operating system aware. (This module requires Python)
operating-system: This module can be used to figure out what operating system is being used to run Terraform. (This module requires Python)
require-executable: This is a module that can be used to ensure particular executables is available in the PATH. (This module requires Python)
run-pex-as-data-source: This module prepares a portable environment for running PEX files and runs them as an external data source. PEX files are python executables that contain all the requirements necessary to run the script. (This module requires Python)
run-pex-as-resource: This module prepares a portable environment for running PEX files and runs them as an local-exec provisioner on a null_resource. PEX files are python executables that contain all the requirements necessary to run the script. (This module requires Python)

The following modules were deprecated and removed:

intermediate-variable: This module has been superseded by terraform local values. To upgrade, switch usage of intermediate-variable with locals.
enabled-aws-regions: This module has been superseded by terraform aws_regions data source. To upgrade, switch the module block with:

data "aws_regions" "enabled_regions" {}

Then, you can get the list of enabled regions using data.aws_regions.enabled_regions.names.

Click on each module above to see its documentation. Head over to the examples folder for example usage.

What is a module?

A Module is a canonical, reusable, best-practices definition for how to run a single piece of infrastructure, such as a database or server cluster. Each Module is written using a combination of Terraform and scripts (mostly bash) and include automated tests, documentation, and examples. It is maintained both by the open source community and companies that provide commercial support.

Instead of figuring out the details of how to run a piece of infrastructure from scratch, you can reuse existing code that has been proven in production. And instead of maintaining all that infrastructure code yourself, you can leverage the work of the Module community to pick up infrastructure improvements through a version number bump.

Who maintains this Module?

This Module is maintained by Gruntwork. If you're looking for help or commercial support, send an email to [email protected]. Gruntwork can help with:

Setup, customization, and support for this Module.
Modules for other types of infrastructure, such as VPCs, Docker clusters, databases, and continuous integration.
Modules that meet compliance requirements, such as HIPAA.
Consulting & Training on AWS, Terraform, and DevOps.

How is this Module versioned?

This Module follows the principles of Semantic Versioning. You can find each new release, along with the changelog, in the Releases Page.

During initial development, the major version will be 0 (e.g., 0.x.y), which indicates the code does not yet have a stable API. Once we hit 1.0.0, we will make every effort to maintain a backwards compatible API and use the MAJOR, MINOR, and PATCH versions on each release to indicate any incompatibilities.

License

Please see LICENSE.txt and NOTICE for details on how the code in this repo is licensed.

terraform-aws-utilities's People

Contributors

Stargazers

Watchers

Forkers

willceitho ticketguardian dcc-op mthrynn aapaeste mundey mploski ubi-mirrors upenderadepu maxbrunet tenderitaf edgarsandi devopskev tidalmigrations ramesharun sijokappen cloud-admin-wafd newamsterdamlabs xiaoruiguo cristian-sr ssebastiandass savadev willerpp necroscope rynkowsg pathologywatch schysys sivaatluri alyragab cheredia2k12 asquared-capital jacobjohansen isabella232 rocky0001 jtdoepke arun9theja feistyeric hyposcaler rjgcp franklinharry jayvb4d4 sweetgreen gannino attractions-io marianod92 sivaatluri567 nrgene crunchyroll gravityforall dswashb-octo worthpointlabs tagsource roche-csi gitproperly shakahl eali-andros meganerddev doflo-gruntwork-io tamioom tnn-gruntwork-io princetonidentity-gruntwork passbase gcagle3 lkovacs-mf adevhammer jlok327 crater-io zitaratech rajbkrish gads-citron vanyanikolova shinchan79 ngoclong189 arunms003 shristi-satvik axios-hq imerit-io

terraform-aws-utilities's Issues

Python 3.11 [SSL: CERTIFICATE_VERIFY_FAILED] in download-dependency-if-necessary.py

Python 3.11 has backwards incompatible CAs?

If you use Python 3.11 to run download-dependency-if-necessary.py, it will generate this error:

/tmp/test 0 $ /usr/local/bin/python3 download-dependency-if-necessary.py --executable kubergrunt --download-url https://github.com/gruntwork-io/kubergrunt/releases/download/v0.11.3/kubergrunt --append-os-arch --install-dir /tmp/kubergrunt-test
2023-07-12 14:26:25,314 - __main__ - INFO - Downloading from https://github.com/gruntwork-io/kubergrunt/releases/download/v0.11.3/kubergrunt_darwin_amd64 to /tmp/kubergrunt-test/kubergrunt
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1286, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1332, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1281, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1041, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 979, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1458, in connect
    self.sock = self._context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1075, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/test/download-dependency-if-necessary.py", line 110, in <module>
    main()
  File "/private/tmp/test/download-dependency-if-necessary.py", line 55, in main
    executable_path = download_executable(args.executable, args.download_url, args.install_dir, args.append_os_arch)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/test/download-dependency-if-necessary.py", line 83, in download_executable
    urlretrieve(download_url, executable_path)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 241, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
                            ^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)>
/tmp/test 1 $ /usr/local/bin/python3 --version
Python 3.11.4

This configuration is meant to mimic the call in the eks-cluster-control-plane module:

 /usr/local/bin/python3 download-dependency-if-necessary.py --executable kubergrunt --download-url https://github.com/gruntwork-io/kubergrunt/releases/download/v0.11.3/kubergrunt --append-os-arch --install-dir /tmp/kubergrunt-test

Python 3.9 still works

/tmp/test 0 $ python3 --version
Python 3.9.6
/tmp/test 0 $ python3 download-dependency-if-necessary.py --executable kubergrunt --download-url https://github.com/gruntwork-io/kubergrunt/releases/download/v0.11.3/kubergrunt --append-os-arch --install-dir /tmp/kubergrunt-test-python-3.9
2023-07-12 14:29:58,861 - __main__ - INFO - Downloading from https://github.com/gruntwork-io/kubergrunt/releases/download/v0.11.3/kubergrunt_darwin_amd64 to /tmp/kubergrunt-test-python-3.9/kubergrunt
{"path": "/tmp/kubergrunt-test-python-3.9/kubergrunt"}
/tmp/test 0 $ ls ../kubergrunt-test-python-3.9 
kubergrunt*

Python pex scripts fail on windows machines without long path support

Python 3 has native long path support built in on windows, but it only supports Windows 10 and requires enabling certain settings that may not be accessible to end users.

The failure mode is that the pex zip file fails to unzip into the $HOME/.pex/install directory. This probably requires a fix within pex itself.

examples/pex: Fix Python 3.11 support

The pex example was recently updated to support Python 3.8-3.11. We are opening this issue to report a bug in Python 3.11 support due to the 'rU' mode being removed. Current error when running the pex example using Python 3.11:

│ The data source received an unexpected error while attempting to execute the program.
│ 
│ Program: python3
│ Error Message: Traceback (most recent call last):
│   File "/terraform-aws-utilities/examples/pex/sample-python-script/bin/sample_python_script_py3_env.pex/.bootstrap/pex/pex.py", line 349, in execute
│   File "/terraform-aws-utilities/examples/pex/sample-python-script/bin/sample_python_script_py3_env.pex/.bootstrap/pex/pex.py", line 276, in patch_sys
│   File "/terraform-aws-utilities/examples/pex/sample-python-script/bin/sample_python_script_py3_env.pex/.bootstrap/pex/pex.py", line 244, in minimum_sys
│   File "/terraform-aws-utilities/examples/pex/sample-python-script/bin/sample_python_script_py3_env.pex/.bootstrap/pex/pex.py", line 119, in _extras_paths
│   File "/terraform-aws-utilities/examples/pex/sample-python-script/bin/sample_python_script_py3_env.pex/.bootstrap/pex/util.py", line 228, in iter_pth_paths
    asample-python-script",
│ ValueError: invalid mode: 'rU'
│ Traceback (most recent call last):
│   File "<string>", line 6, in <module>
│   File "~/.pyenv/versions/3.11.0/lib/python3.11/subprocess.py"r line 413, in check_call
│     raise CalledProcessError(retcode, cmd)
│ subprocess.CalledProcessError: Command '['python3', './sample-python-script/bin/sample_python_script_py3_env.pex', '../../modules/prepare-pex-environment/entrypoint.py', 'sample_python_script.main:main',
│ '--is-data']' returned non-zero exit status 1.
│ 
│ State: exit status 1

PEX binary will need to be updated to correctly support Python 3.11.

Update pex example for Python compatibility

Describe the bug

The module pex example only supports up to Python 3.7, which will be EOL 27 Jun 2023. Need to update functionality to a newer supported version of Python.

Missing `aws` provider explicitly specified for this repo

While updating to tf14, we found that the upgrade script did not add any required_providers block. We then realised it's because there were no provider blocks in the original code.

It might be best to add a specific version provider for aws instead of relying on the default one, when not specified.

Context: See TicketJam notes & PR open for the TF 14 upgrade script fixes

Deprecation Notice: Replace distutils package

The distutils package has been deprecated for a little over 3 years now and is being phased out of systems now. There are 2 scripts in this repo that are dependent on this package and should be updated.

PEP 632

Dependent scripts:

Users are reporting errors for dependent modules such as the terraform-aws-eks module due to the package not being available during deployment.

EKS Issue

The require-executable module forces an external program lookup on the terraform running environment even not used

We are trying to use Terraform Cloud to plan and apply the eks-cluster-control-plane module and require-executable is a hard downstream dependency. It always fails because TFC runner has no python installed and there is no way we can disable terraform from checking the python executable even we are not using the required-executable functions.

Understand that we could just use the tf resources directly if not leveraging those basic validations gruntwork provided out of the box, but It is always good to provide options for users to work around without affecting the default behavior, especially for the use cases where people are migrating from self-hosted runners to terraform cloud.

Error Log:

with module.eks_cluster.module.require_kubergrunt.data.external.required_executable
on .terraform/modules/eks_cluster.require_kubergrunt/modules/require-executable/main.tf line 9, in data "external" "required_executable":
  program = ["python", "${path.module}/require_executable.py"]
The data source received an unexpected error while attempting to find the program.
Platform: linux
Program: python
Error: exec: "python": executable file not found in $PATH

Proposed Solution:
Using count to mute terraform plan if length(required_executable) == 0. (inspired by the executable-dependency module)

data "external" "required_executable" {
  count = length(var.required_executables) > 0 ? 1 : 0
  program = ["python3", "${path.module}/require_executable.py"]

  # Currently the external data source provider does not support list values in the query, so we convert the input list
  # to be a comma separated string.
  # See https://github.com/terraform-providers/terraform-provider-external/issues/2
  query = {
    required_executables = join(",", var.required_executables)
    error_message        = var.error_message
  }
}

Let me know what do you guys think, happy to contribute PRs.

Can we unpin the external provider pin now that the bug is fixed

This #72 was done because of this bug hashicorp/terraform-provider-external#193
Which is now fixed it would appear

Can we do the testing to see if this pin can be removed?

Thanks!

Upgrade all utilities modules to support aws provider v5.x

Describe the solution you'd like
Test and support version >= 5.0 for all utilities modules.

Describe alternatives you've considered

Additional context
https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md#500-may-25-2023