Git Product home page Git Product logo

terragrunt-infrastructure-modules-example's Introduction

Maintained by Gruntwork.io

Example infrastructure-modules for Terragrunt

This repo, along with the terragrunt-infrastructure-live-example repo, show an example file/folder structure you can use with Terragrunt to keep your Terraform and OpenTofu code DRY. For background information, check out the Keep your code DRY section of the Terragrunt documentation.

This repo contains the following example modules:

  • asg-alb-service: An example of how to deploy an Auto Scaling Group (ASG) with an Application Load Balancer (ALB) in front of it. We run a dirt-simple "web server" on top of the ASG that always returns "Hello, World".

  • mysql: An example of how to deploy MySQL on top of Amazon's Relational Database Service (RDS).

Note: This code is solely for demonstration purposes. This is not production-ready code, so use at your own risk. If you are interested in battle-tested, production-ready Terraform and OpenTofu code, check out Gruntwork.

How do you use these modules?

To use a module, create a terragrunt.hcl file that specifies the module you want to use in the source URL as well as values for the input variables of that module in the inputs block:

# Use Terragrunt to download the module code
terraform {
  source = "git::[email protected]:gruntwork-io/terragrunt-infrastructure-modules-example.git//path/to/module?ref=<VERSION>"
}

# Fill in the variables for that module
inputs = {
  foo = "bar"
  baz = 3
}

(Note: the double slash (//) in the source URL is intentional and required. It's part of Terraform and OpenTofu's Git syntax for module sources.)

You then run Terragrunt, and it will download the source code specified in the source URL into a temporary folder, copy your terragrunt.hcl file into that folder, and run your Terraform / OpenTofu command in that folder:

> terragrunt apply
[terragrunt] Reading Terragrunt config file at terragrunt.hcl
[terragrunt] Downloading Tofu configurations from git::[email protected]:gruntwork-io/terragrunt-infrastructure-modules-example.git//path/to/module?ref=<VERSION>
[terragrunt] Copying files from . into .terragrunt-cache
[terragrunt] Running command: tofu apply
[...]

Check out the terragrunt-infrastructure-live-example repo for examples and the Keep your code DRY documentation for more info.

How do you change a module?

Local changes

Here is how to test out changes to a module locally:

  1. git clone this repo.
  2. Update the code as necessary.
  3. Go into the folder where you have the terragrunt.hcl file that uses a module from this repo (preferably for a dev or staging environment!).
  4. Run terragrunt plan --terragrunt-source <LOCAL_PATH>, where LOCAL_PATH is the path to your local checkout of the module code.
  5. If the plan looks good, run terragrunt apply --terragrunt-source <LOCAL_PATH>.

Using the --terragrunt-source parameter (or TERRAGRUNT_SOURCE environment variable) allows you to do rapid, iterative, make-a-change-and-rerun development.

Releasing a new version

When you're done testing the changes locally, here is how you release a new version:

  1. Update the code as necessary.

  2. Commit your changes to Git: git commit -m "commit message".

  3. Add a new Git tag using one of the following options:

    1. Using GitHub: Go to the releases page and click "Draft a new release".
    2. Using Git:
    git tag -a v0.0.2 -m "tag message"
    git push --follow-tags
    
  4. Now you can use the new Git tag (e.g. v0.0.2) in the ref attribute of the source URL in terragrunt.hcl.

  5. Run terragrunt plan.

  6. If the plan looks good, run terragrunt apply.

Folder structure

This repo uses the following "standard" folder structure:

  • modules: Put module code into this folder.
  • examples: Put example code into this folder. These are examples of how to use the modules in the modules folder. This is useful both for manual testing (you can manually run tofu apply on these examples) and automated testing (as per the tests in the test folder, as described next).
  • test: Put test code into this folder. These should be automated tests for the examples in the examples folder.

Monorepo vs. polyrepo

This repo is an example of a monorepo, where you have multiple modules in a single repository. There are benefits and drawbacks to using a monorepo vs. using a polyrepo - one module per repository. Which you choose depends on your tooling, how you build/test Terraform/OpenTofu modules, and so on. Regardless, the live repo will consume the modules in the same way: a reference to a Git release tag in a terragrunt.hcl file.

Advantages of a monorepo for Terraform/OpenTofu modules

  • Easier to make global changes across the entire codebase. For example, applying a critical security fix or upgrading everything to a new version of Terraform/OpenTofu can happen in one logical commit.
  • Easier to search across the entire codebase. You can search through all the module code using a standard text editor or file searching utility just with one repo checked out.
  • Simpler continuous integration across modules. All your code is tested and versioned together. This reduces the chance of late integration issues arising from out-of-date module-dependencies.
  • Single repo and build pipeline to manage. Permissions, pull requests, etc. all happen in one spot. Everything validates and tests together so you can see any failures in one spot.

Disadvantages of a monorepo for Terraform/OpenTofu modules

  • Harder to keep changes isolated. While you're modifying module foo, you also have to think through whether this will affect module bar.
  • Ever increasing testing time. The simple approach is to run all tests after every commit, but as the monorepo grows, this gets slower and slower (and more brittle).
  • No dependency management system. To only run a subset of the tests or otherwise validate only changed modules, you need a way to tell which modules were affected by which commits. Unfortunately, Terraform/OpenTofu has no first-class dependency management system, so there's no way to know that a code change in a file in module foo won't affect module bar. You have to build custom tooling that figures this out based on heuristics (brittle) or try to integrate Terraform with dependency management / monorepo tooling like bazel (lots of work).
  • Doesn't work with the Terraform Private Registry. Private registries (part of Terraform Enterprise and Terraform Cloud) require one module per repo.
  • No feature toggle support. Terraform/OpenTofu doesn't support feature toggles, which are often critical for making large scale changes in a monorepo.
  • Release versions change even if module code didn't change. A new "release" of a monorepo involves tagging the repo with a new version. Even if only one module changed, all the modules effectively get a new version.

Advantages of one-repo-per-module

  • Easier to keep changes isolated. You mostly only have to think about the one module/repo you're changing rather than how it affects other modules.
  • Works with the Terraform Private Registry. Private registries (part of Terraform Enterprise and Terraform Cloud) can list modules in a one-repo-per-module format if you follow their module structure and repository naming conventions.
  • Testing is faster and isolated. When you run tests, it's just tests for this one module, so no extra tooling is necessary to keep tests fast.
  • Easier to detect individual module changes. With only one module in a repo, there's no guessing at which module changed as releases are published.

Disdvantages of one-repo-per-module

  • Harder to make global changes. Changes across repos require lots of checkouts, separate commits and pull requests, and an updated release per module. This may need to be done in a specific order based on depedency graphs. This may take a lot of time in a large organization, which is problematic when dealing with security issues.
  • Harder to search across the codebase. Searches require checking out all the repos or having tooling (e.g., GitHub or Azure DevOps) that allows searching across repositories remotely.
  • No continuous integration across modules. You might make a change in your module and the teams that depend on that module might not consume that change for a long time. When they do, they may find an incompatibility or other issue that could be hard to fix given the amount of time that's passed.
  • Many repos and builds to manage. Permissions, pull requests, build pipelines, test failures, etc. get managed in several places.
  • Potential dependency graph problems. It is possible to run into issues like "diamond dependencies" when using many modules together, though Terraform/OpenTofu can avoid many of these issues since it can run different versions of the same dependency at the same time.
  • Slower initialization. Terraform/OpenTofu downloads each dependency from scratch, so if one repo depends on modules from many other repos โ€” or even the exact same module from the same repo, but used many times in your code โ€” it will download that module every time it's used rather than just once.

terragrunt-infrastructure-modules-example's People

Contributors

brikis98 avatar dependabot[bot] avatar eak12913 avatar gruntwork-ci avatar infraredgirl avatar mpuels avatar rhoboat avatar robmorgan avatar santoshr1016 avatar tillig avatar yorinasub17 avatar zackproser avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

terragrunt-infrastructure-modules-example's Issues

update main.tf for both modules "required_version" to use >=

Ran into an issue while using the latest 12.26 terraform version.
As the comment above stated, any terraform version above 12.x should work fine.
# The latest version of Terragrunt (v0.19.0 and above) requires Terraform 0.12.0 or above.

Error: Unsupported Terraform Core version

This configuration does not support Terraform version 0.12.26. To proceed,
either choose another supported Terraform version or update the root module's
version constraint. Version constraints are normally set for good reason, so
updating the constraint may lead to other errors or unexpected behavior.

Fix or remove CircleCI build

Currently we've got a Github check running the project in CircleCI, but we don't actually have it configured. As part of this issue, we should:

  • make a decision if we want a CircleCI pipeline to run - e.g. some terraform validate steps, or some other relevant steps
    if yes, then add a configuration file & make sure it passes
  • if no, then disable this check for the project from CircleCI here
    Why?
    Currently, some PRs will appear with a broken build because of the missing CircleCI configuration. This is not a big problem, so this task is more a housekeeping type of job.

Terragrunt validate failing

Hi Getting below issue while running terragrunt validate-all. Not getting any clue .
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Duplicate backend configuration

on main.tf line 15, in terraform:
15: backend "s3" {}

A module may have only one backend configuration. The backend was previously
configured at backend.tf:3,3-15.

Error: Duplicate provider configuration

on provider.tf line 2:
2: provider "aws" {

A default (non-aliased) provider configuration for "aws" was already given at
main.tf:9,1-15. If multiple configurations are required, set the "alias"
argument for alternative configurations.

Please add a new release - currently broken with terragrunt-infrastructure-live-example

Hello TerraGrunts,

I noticed while trying to git the live example up, it's not pulling the latest commits and unfortunately causes errors with terragrunt.

There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Error: Duplicate backend configuration

  on main.tf line 15, in terraform:
  15:   backend "s3" {}

A module may have only one backend configuration. The backend was previously
configured at backend.tf:3,3-15.


Error: Duplicate provider configuration

  on provider.tf line 2:
   2: provider "aws" {

A default (non-aliased) provider configuration for "aws" was already given at
main.tf:9,1-15. If multiple configurations are required, set the "alias"
argument for alternative configurations.

[terragrunt] 2020/06/06 01:48:18 Hit multiple errors:
exit status 1

How is automated testing handled in a multi-module repository?

Let's say I have a "live modules" repository like this with multiple individual modules referenced by the "live repository." What's the recommendation for how to handle something like this in a CI/automated testing pipeline?

I can see a few options:

  • Run all the tests and validation for every module in the repo on every push. Depending on how many modules are in the repo, this could take a very, very long time. Maybe that's OK.
  • Create separate CI pipelines for each module using some sort of "path filtering" mechanism built into your CI server. This would be faster, but would also potentially have a lot of pipelines depending on how many modules are in the repo.
  • Only do manual kickoff of a CI build. Require some sort of input parameter to indicate, say, the path of the component that changed.
  • Do some fairly extravagant magic to try to determine what commits have shown up since the last build, which files in those commits changed, and, from that, infer which module(s) need to be tested.

I noticed on this issue it sounded like CI was an area being worked on. Not a huge deal if there's no specific recommendation, but I'm sort of working on this now myself and am struggling to balance number of pipelines with speed of feedback.

Why does module contain Terraform backend declaration?

I wonder what is the reason to put Terraform backend declaration inside the module? It's reasonable to assume user would expect to see backend declaration in the environment area (prod, stage, qa). Module should not care what kind of backend is in use. I understand terraform.tfvars can't hold something except variables, and backend should be specified somewhere. However, main.tf or backend.tf inside the environment folder is a more appropriate place. Consider the next folder layouts:

GOOD

prod
  terraform.tfvars
  backend.tf
stage
  terraform.tfvars
  backend.tf
BETTER

backend.tf # usually we use one backend for all kind of TF states
prod
  terraform.tfvars
  backend.tf # but you may always override the global value in the downstream
stage
  terraform.tfvars

Ideally, Terragrunt could infer main.tf/backend.tf file with the corresponding Terraform code from terragrunt.remote_state.backend config in terraform.tfvars file:

BEST

terraform.tfvars # infer terraform backend code from `terragrunt.remote_state.backend` config  and copy it to `.terragrunt-cache` folder
prod
  terraform.tfvars # you may always override the global `terragrunt.remote_state.backend` value in the downstream
stage
  terraform.tfvars

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.