Azure DevOps Pipeline Extension for Terraform

License: Mozilla Public License 2.0

TypeScript 98.97% HCL 1.03%

azure-pipelines-extension-terraform's Introduction

Azure Pipelines Extension for Terraform

This repository contains the source for an Azure Pipelines extension that provides Tasks to easily install and use Terraform.

This extension provides a TerraformInstaller task to ease in installing specific Terraform versions, as well as a Terraform task to help call Terraform without needing to manage authentication yourself. The Terraform task wraps init, plan, validate, apply, and destroy commands, as well as providing a CLI option.

CLI is used to execute a custom script in a pre-authenticated environment. This can be a great option if you need to use more complex terraform scripts, such as gathering output and setting it to a Piplelines variable (see example below).

Once this task has been added to your Organization from the Azure DevOps Marketplace you can use it in any Azure Pipelines build or release job. It is available in both the GUI pipeline editor as well as yaml templates.

Options

General

Common options available in most configurations

Name	Type	Description
`command`	`pickList`	The Terraform command to run. Options: `init`, `validate`, `plan`, `apply`, `destroy`, `cli`
`provider`	`pickList`	The Cloud provider to authenticate with. Options: `Azure`, `Remote` (`AWS` and `GCP` Coming soon)
`backend`	`pickList`	Where to store the Terraform backend state. Options: `Azure`, `Remote` (`AWS` and `GCP` Coming soon)

Azure

Options which are available when Azure Backend and Providers are selected.

Name	Type	Description
`providerAzureConnectedServiceName`	`serviceConnection`	The Azure Subscription to execute Terraform against
`backendAzureUseProviderConnectedServiceForBackend`	bool	Should the specified provider connection be re-used to talk to the backend storage account?
`backendAzureConnectedServiceName`	`serviceConnection`	The Azure Subscription to be used to talk to the backend storage accoutn
`backendAzureStorageAccountName`	`serviceConnection`	If a separate backend connection is specified: the Storage Account to store the backend state in.
`backendAzureProviderStorageAccountName`	`serviceConnection`	If no separate backend connection is specified: the Storage Account to store the backend state in.
`backendAzureContainerName`	`string`	The Storage Account Container name
`backendAzureStateFileKey`	`string`	The name of the terraform state file

CLI

Options which are available when command is set to CLI

Name	Type	Description
`initialize`	`bool`	Should `terraform init` run before executing the CLI script
`scriptLocation`	`pickList`	How will the CLI script be provided? Options: `Inline script`, `Script path`
`scriptPath`	`filePath`	The path to the CLI script to execute
`script`	`string`	The inline script to execute

Advanced

Advanced options available for all non-CLI commands

Name	Type	Description
`args`	`string`	Additional arguments to pass to the Terraform command being run

YAML Pipeline Examples

Install Terraform

pool:
  vmImage: 'Ubuntu-16.04'

steps:
- task: terraformInstaller@0
  inputs:
    terraformVersion: '0.12.12'
  displayName: Install Terraform

Init, plan, and apply

You can invoke the task in a yaml template using the following syntax:

pool:
  vmImage: 'Ubuntu-16.04'

steps:
- task: terraform@0
  inputs:
    command: 'init'
    providerAzureConnectedServiceName: 'MTC Denver Sandbox'
    backendAzureProviderStorageAccountName: 'mtcdenterraformsandbox'
  displayName: Terraform Init
    
- task: terraform@0
  inputs:
    command: 'plan'
    providerAzureConnectedServiceName: 'MTC Denver Sandbox'
    args: -var=environment=demo -out=tfplan.out
  displayName: Terraform Plan

- task: terraform@0
  inputs:
    command: 'apply'
    providerAzureConnectedServiceName: 'MTC Denver Sandbox'
    args: tfplan.out
  displayName: Terraform Apply

Execute a Terraform-authenticated CLI Script

pool:
  vmImage: 'Ubuntu-16.04'

steps:
- task: terraform@0
  inputs:
    command: 'CLI'
    providerAzureConnectedServiceName: 'MTC Denver Sandbox'
    backendAzureProviderStorageAccountName: 'mtcdenterraformsandbox'
    script: |
      # Validate
      terraform validate

      # Plan
      terraform plan -input=false -out=testplan.tf

      # Get output
      STORAGE_ACCOUNT=`terraform output storage_account`

      # Set storageAccountName variable from terraform output
      echo "##vso[task.setvariable variable=storageAccountName]$STORAGE_ACCOUNT"
    displayName: Execute Terraform CLI Script

azure-pipelines-extension-terraform's People

Contributors

Stargazers

Watchers

azure-pipelines-extension-terraform's Issues

Missing tfstate when using separate service connection for backend

Issue
When configuring a separate service connection for the Azure RM backend than the planning and apply stages, there is no way to get the tfstate to actually be stored in the expected Azure storage account.

The apply successfully deploys stuff, but the tfstate is not stored anywhere (probably still local to the pipeline host)

Our configuration

We have a centralized Azure subscription for our customer environment's tfstate files.
We configure 3 steps in the pipeline
Terraform install version 0.13.2 (also tried 0.12 versions)
Terraform init with subscription A for backend. This succeeds only when the storage account and container is available.
Terraform apply into subscription B, expecting to use subscription A for backend
There is NO backend azurerm configured in .tf files. When this is configured, it must match the backend with access key and everything, which of course is not how this should work.
Our providers.tf:

terraform { required_providers { azurerm = { source = "hashicorp/azurerm" version = "~>2.24.0" } } }

We tried both windows-2019 and ubuntu host
Adding an empty backend azurerm {} block causes the wrong subscription (the one configured as target for the deployment) to be checked for the storage account for the tfstate, which does not exist, giving a 404 error.

A workaround
We have added a Azure CLI step instead of the terraform init step, using list keys and generating a separate backend.tf file, that is only used during the release pipeline, before trigger terraform init in the script:

https://pastebin.com/Ky3QMC76

Join forces to consolidate offerings?

Just curious if there is any interest in collaborating to incorporate the feature set from mine into this one?

https://marketplace.visualstudio.com/items?itemName=charleszipp.azure-pipelines-tasks-terraform

I could help update this repo with the azure-based feature set from mine. That way we can converge on a single offering for the community.

Related to responses from #6

Still Alive?

Hi, as there is no reaction to my pull request for a doc change - is this project still being actively maintained? Greetings, Sascha

cwd can not have a trailing slash

If the cwd parameter is supplied with a trailing slash the terraform executable fails to run. CWD should not be necessary as tasks by default have a workingDirectory property.

Extravagant permission requirements on storage account

The extension, at least when performing terraform init, performs certain operations that it shouldn't need to, requiring quite a lot of permissions on not just the container, but even the whole storage account.

The problematic operation I'm running into is listKeys on the whole storage account. This should not be necessary. We have specified the subscription or (as in my case) service connection, the resource group, the storage account, the container, and the blob key (i.e. name). The extension knows exactly what to look for. Listing anything should be unnecessary.

The error message is this (line breaks mine):

Failed to get existing workspaces: Error retrieving keys for Storage Account "___":
storage.AccountsClient#ListKeys: Failure responding to request:
StatusCode=403 -- Original Error: autorest/azure: Service returned an error.
Status=403 Code="AuthorizationFailed" Message=
"The client '___' with object id '___' does not have authorization to perform action
'Microsoft.Storage/storageAccounts/listKeys/action' over scope
'/subscriptions/___/resourceGroups/___/providers/Microsoft.Storage/storageAccounts/___' or the scope is invalid.
If access was recently granted, please refresh your credentials."

My best guess is that the containers are listed in order to check if the required container exists. I expect that this could instead be achieved by simply accessing the container and handling a response saying it does not exist.

I haven't gotten past this error, but it seems likely that the extension will next perform listKeys on the container itself. This could present the same problem (even though the scope is less extravagant).

Why is this so problematic, you ask - apart from requiring more permissions than strictly necessary?

As we know, Terraform stores sensitive data in the tfstate file. For example, we might pull the database administrator password out of Azure KeyVault (or even generate it on the spot with random_password) and create a database resource with that password. Now the password is not just in our well-protected KeyVault, but also in the tfstate file in the storage account. This makes it essential to use a well-protected storage account - for example, one that is only accessible to the service connection (through a dedicated app registration).

Such a dedicated storage account tends to live in a different resource group, since the main resource group usually grants team members access to its resources, which we want to avoid. In our case, a storage account managed by administrators does not allow listing all the keys in a container, let alone listing all the containers in the storage account (i.e. other teams' Terraform storage containers)!

Could the extension be improved such that it only performs the operations it strictly needs to?

Terraform Cloud Support In Azure-Pipelines Terraform Extension

microsoft/azure-pipelines-extensions#754

Referenced the issue I opened. There seems to be no knowledge on the contents from this blog post and they recommended I reach out to Terraform directly. This seems the right place.

I've been eagerly waiting for these improvements to simplify remote backend with terraform + automation on deploying and creating workspaces/variables from my azure pipelines using the terraform extension. However, I'm not finding anyone knowing anything about the content of the blog post mentioning these soon to be released improvements. Can someone help me understand what the timeline/progress on these improvements is?

Original Issue content from Issue 754 in azure-pipelines-extensions

Terraform Blog announced upcoming improvements with the azure pipelines extensions for Terraform that would support the following types of commands:

Terraform Cloud Backend
Create new tfe_workspace, variables, and more.
Initiating remote run, plan, apply.
I've been searching issues, announcements, community discussion, and even commits, but am having difficulty in finding anything on eta, progress, or issues to watch on this.

I need ways to improve the terraform cloud workflow and am hoping azure pipelines can offer this, so I can automate workspace management.

Can someone provide some linked issues on implementation of Terraform Cloud support with the terraform extension so I can be more aware of the progress of this new feature?

Archive inactive repo

Hi @jlorich @chrismatteson
perhaps this repo could be archived with pointer to the active one?

references:
#17

Can't find in marketplace / link to marketplace?

The documentation states it should be in the marketplace, but I can't find it.

Could you add a link to the documentation stating where in the marketplace this can be found?

Allow a TFE Service Connection to exist with a non-remote backend

If you want to use TFE as a a private module registry right now the only option is to specify a remote backend, however in some circumstances one may want to use a non-remote (e.g. Azure, AWS, GCP) backend while still using a private module registry. An option should be added under advanced to facilitate this.

Documentation is missing

I got a few pointers regarding the documentation;

The documentation how to compile this extention is missing. Although there are some Azure-pipeline.yml present I have a feeling that this can be better described.
The usage of the extention is also missing. Nothing is described how to use the extention.
the YML examples are not working at all for PTFE. I would expect at least a well documented step by step "how to" documentation how to use this extention in regards to PTFE or TF Cloud.
by well documentated I mean a real live example how to setup the extention to actuatly build infrastructure. if you want to get in tough, please contact @LanceHaig (from Hashicorp) to get my personal number so we can work together on this.
The 3 tasks seams not to be working together. I am not sure if I need to install a version of Terraform if I use PTFE? I guess not only if I want to validate my code during build. However, on PTFE the propper version should already be installed.
The queuePlan make no sence to me. If I need to queue a plan I think I should be able to select a workspace? or does the token I need to setup in my service connection in AzureDevOps relate to the corresponding workspace?

AzureRM "Features" blocks are required

When planning an initiatlised configuration, errors out because features{} is missing from the AzureRM provider definition.

* provider.azurerm: version = "~> 2.17"

Starting: Plan Terraform
==============================================================================
Task         : Terraform
Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
Version      : 0.0.142
Author       : Microsoft Corporation
Help         : [Learn more about this task](https://aka.ms/AA5j5pf)
==============================================================================
/opt/hostedtoolcache/terraform/0.12.3/x64/terraform providers
.
└── provider.azurerm

/opt/hostedtoolcache/terraform/0.12.3/x64/terraform plan

Error: Insufficient features blocks

  on  line 0:
  (source code not available)

At least 1 "features" blocks are required.

##[error]Error: The process '/opt/hostedtoolcache/terraform/0.12.3/x64/terraform' failed with exit code 1
Finishing: Plan Terraform

Update tasks to Node 10

I am getting a warning when running TerraformInstaller@0 and TerraformTaskV2@2

##[warning]This task uses Node 6 execution handler, which will be deprecated soon. If you are the developer of the task - please consider the migration guideline to Node 10 handler - https://aka.ms/migrateTaskNode10. If you are the user - feel free to reach out to the owners of this task to proceed on migration.

Why use this one?

Can someone explain why this extension would be chosen in preference to the other terraform ADO pipeline extensions?

https://marketplace.visualstudio.com/search?term=terraform&target=AzureDevOps&category=Azure%20Pipelines&sortBy=Relevance

hashicorp / azure-pipelines-extension-terraform Goto Github PK