Can't use this module with S3 bucket in different region about terraform-aws-tfstate-backend HOT 7 CLOSED

Hey all, just wanted to chime in here as this has come up as well in another issue. Just wanted to explain our reasoning for the breaking change.

@antonbabenko first things first =) Your Medium post was very influential for us last year when we first approached this topic. For that reason, we decided at the to not pin providers at the module level and punt on it. However, since then a lot has improved.

Most notably, in 0.11 they improved the way that providers were passed.

Clear Path for Multi-Region Support

By being explicit about expecting providers to be passed we're setting a clear example and providing a clearer path towards multi region. It's also better than accepting a region argument, since as you mention providers accept a lot of arguments and it would be tedious to accept all of them. Passing the provider is a cleaner solution that lets us pass a single argument. It still respects all pinning (we verified).

Explicit is Better than Implicit

In the Zen of Python one of the aphorisms is "Explicit is better than implicit". Terraform has been moving in this direction and also cites to this saying. For example, they used to allow vars to be passed on the command line with -var even if it wasn't explicitly defined as a variable. That is no longer supported.

Official Best Practice

More importantly, HashiCorp suggests passing providers in their documentation as the recommended design pattern. Of course, best practices can be debated, but by being explicit it's more clear what's going on.

it looks pretty ugly

In our examples, we set aws = "aws", which is just the path-of-least-resistance for compatibility (e.g. drop this in your config and it will keep working). In line with the best practices, the provider should actually be aliased (e.g. by region) in the root module. Then the alias passed to the submodules.

I think once this becomes the norm, it will look less ugly because it will be familiar. Today, few modules are pinning providers; I argue this is not because that is the best practice, but rather that the trade hasn't caught up to what's currently supported.

AWS 2.x Incompatibilities

When the terraform-aws-provider was updated to 2.0, many of our modules broke. We were inundated with issues/prs related to this. While pinning the modules doesn't necessarily prevent that breakage, it would have given a hint to terraform so it could negotiate the best version of the provider to use or error precisely that the providers are incompatible with the last known working configuration.

Testing

We're adding real tests to all of our modules, including this one. This is a monumental effort we are undertaking with the help of @aknysh. While the tests are far from comprehensive, they do prove at a minimum that the modules pass terraform plan, terraform apply and terraform destroy, as well as have sane outputs. We also run it through a battery of tests in our test-harness which ensure modules are pinned, providers are pinned, everything is formatted, and a litany of other checks.

This module has tests. They are working (green) and build logs are available here.

With tests in place, we're now able to review/accept/merge PRs much faster than before. Also, since we're pinning providers, we assume there will be more PRs related to updating pins (and perhaps tests!) in the future. We'll also know much sooner when incompatibilities are found.

Module Pinning

Module pinning is not as necessary in a closed ecosystem (e.g. within the walled garden of a corporate git repo) where the organization is in control of how or when changes are introduced. However, in the huge public ecosystem we are working with, the change velocity outside of our control is tremendous. Anywhere we can pin, we want to pin so we can stop playing wack-a-mole.

Precedent Set

We are not the only ones starting to do this. In the latest public modules by Gruntworks, they are pinning providers in non-root modules as well.

The Time is Now

If there ever was a time to introduce breaking changes, it's with the 0.12 conversion of our modules. With terraform 0.12, a lot changed, broke and improved. Supporting 0.12 requires users to update their code to HCL and test that it works. This is why we feel now is the time to introduce these changes as developers are already forced to make wide sweeping changes.

We are going through literally every single one of our modules and updating them for 0.12 support, so we're also taking the step of pinning providers to the last known working configuration.

So with that said, I challenge everyone in our awesome community to help us continuing to push the bar higher. Rising tide floats all boats.

from terraform-aws-tfstate-backend.

@n6g7 The solution is to call this module like this:

module "terraform_state_backend" {
  source = "git::https://github.com/cloudposse/terraform-aws-tfstate-backend.git?ref=tags/0.8.0"

  # ...

  providers = {
    aws = "aws"
  }
}

While it works as expected it looks pretty ugly and I really prefer to not have provider block inside of modules.

from terraform-aws-tfstate-backend.

We're going to update the modules to support required_providers syntax instead.

from terraform-aws-tfstate-backend.

I'm having a similar issue when running terraform validate on my project:

Error: Missing required argument

  on .terraform/modules/terraform_state_backend/versions.tf line 8, in provider "aws":
   8: provider "aws" {

The argument "region" is required, but no definition was found.

Reproducible with a simple script:

provider "aws" {
  region  = var.region
  profile = "myawsprofile"
}

module "terraform_state_backend" {
  source = "git::https://github.com/cloudposse/terraform-aws-tfstate-backend.git?ref=tags/0.8.0"

  region        = var.region
  namespace     = "myproject"
  stage         = "main"
  force_destroy = var.force_destroy
}

@aknysh could you take a look?

from terraform-aws-tfstate-backend.

I got exactly the same error when upgrading to terraform 0.12, Thank you @antonbabenko Your solution may be ugly but it is working.

from terraform-aws-tfstate-backend.

@antonbabenko that does fix it indeed!

from terraform-aws-tfstate-backend.

@antonbabenko (via slack) suggested having this solution would work better than hardcoding providers "..." { version = "..." } configurations:

terraform {
  required_providers {
    aws = ">= 2.7.0"
  }
}

Doing this preserve the inheritance of the default provider, so that the caller is not required to pass it explicitly. While I still think explicit is good, I think a compromise here will serve everyone better since this satisfies our primary concern which is version pinning.

https://www.terraform.io/docs/configuration/terraform.html#specifying-required-provider-versions

from terraform-aws-tfstate-backend.