Regression in http 2.0.0 data source behavior from 0.13.5 to 0.14.0 when fetching url with content type application/x-x509-ca-cert about terraform-provider-http HOT 7 CLOSED

This issue is fixed in Terraform v0.15.0. We recommend users who are experiencing this issue upgrade to Terraform 0.15.

Explanation

The http data source can be used to make HTTP requests whose response has MIME type "application/x-509-ca-cert", which returns binary data. Terraform internals serialise this data to a UTF-8 string (see hashicorp/terraform#3858 (comment)) to store it in state, which means Terraform cannot guarantee that such data will not be corrupted.

Prior to v2.0.0, terraform-provider-http would return an error if the HTTP response had any Content-Type header not on a whitelist of known text types. In v2.0.0 this error was relaxed to a warning (#50), leaving it up to the user to judge whether the possible state corruption was an acceptable tradeoff when using non-text MIME types.

As noted in the original issue description, Terraform 0.14 introduced a bug in which the warning is logged but the output is not written to the state despite the command exiting successfully. This bug does not occur in Terraform v0.15.0.

Configuration

data "http" "auth0_cert_pem" {
  url = "https://exampleco-enterprises.us.auth0.com/pem"
}

output "pem" {
  value = data.http.auth0_cert_pem.body
}

Steps taken

1. terraform init
2. terraform apply -auto-approve

Expected behaviour

The CLI logs the warning:

Warning: Content-Type is not recognized as a text type, got "application/x-x509-ca-cert"

If the content is binary data, Terraform may not properly handle the contents
of the response.

The output is correctly populated with the body as a string.

Actual behaviour

The following results are observed on terraform apply depending on Terraform version:

Terraform v0.12.30, terraform-provider-http v1.2.0

Note that v1.2.0 is the latest version of terraform-provider-http compatible with Terraform v0.12.30.

$ terraform apply -auto-approve
data.http.auth0_cert_pem: Refreshing state...

Error: Content-Type is not a text type. Got: application/x-x509-ca-cert

  on main.tf line 1, in data "http" "auth0_cert_pem":
   1: data "http" "auth0_cert_pem" {

Exit 1, no output written. Incorrect behaviour.

Terraform v0.13.6, terraform-provider-http v2.1.0

$ terraform apply -auto-approve
data.http.auth0_cert_pem: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

<correct cert output>

Correct behaviour.

Terraform v0.14.4, terraform-provider-http v2.1.0

$ terraform apply -auto-approve

Warning: Content-Type is not recognized as a text type, got "application/x-x509-ca-cert"

If the content is binary data, Terraform may not properly handle the contents
of the response.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes


Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Exit 0, output not written. Incorrect behaviour.

Terraform v0.15.0, terraform-provider-http v2.1.0

$ terraform apply -auto-approve

Warning: Content-Type is not recognized as a text type, got "application/x-x509-ca-cert"

If the content is binary data, Terraform may not properly handle the contents
of the response.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

<correct cert output>

Exit 0. Correct behaviour.

from terraform-provider-http.

So how do you remove this warning?

data "http" "azure-mysql-northeurope-root-cert" {
  url = "https://www.digicert.com/CACerts/BaltimoreCyberTrustRoot.crt.pem"
  request_headers = {
    Accept = "application/x-pem-file"
  }
}

This doesn't help I still get this:

│ Warning: Content-Type is not recognized as a text type, got "application/x-pem-file"
│ 
│   with data.http.azure-mysql-northeurope-root-cert,
│   on k8s-provision.tf line 4, in data "http" "azure-mysql-northeurope-root-cert":
│    4: data "http" "azure-mysql-northeurope-root-cert" {
│ 

from terraform-provider-http.

I am experiencing the same issue.

I found that it occurs when the response Content-Type does not match text/.+. application/json, or application/samemetadata+xml

I dug into it a bit further and found that the act of creating the warning diagnostic seems to be what prevents the result from being output.

Removing the following code resolves the issue.

This is not a solution but I hope it is useful information.

Has something changed in the way terraform or the terraform-plugin-sdk handles warning diagnostics?

from terraform-provider-http.

Unsupported Content-Type was changed from an error to a warning in this commit:

7477f3e#diff-bdc98004d4549f4faf184ce5113b2073f1feb2660b1a66596a38d92b37f87194

At the time this worked as intended but I wonder if that was due to a bug that has since been fixed? If terraform and the terraform-plugin-sdk are working as expected it may be wise to consider reverting this change.

I will note that the tests do not seem to fully cover all of the ContentType cases.

from terraform-provider-http.

I am running two separate runs one from enterprise using 13.5 and one from cloud using 14.6.

Enterprise returns a result even though the cert of the website is self signed however the cloud run fails with ...

Error making request: Get "https://blah.bl:9443/api/Solarwinds/GetNextIPAddress?subnetDisplayName=10.108.0.0/16": x509: certificate signed by unknown authority

Operation failed: failed running terraform plan (exit 1)

could this be related ?

from terraform-provider-http.

Any news on that? This bug makes this provider useless for a lot of usecases, including fetching CA certs from Vault.

from terraform-provider-http.

Hi @holms 👋

Sorry you ran into trouble here.

Could you possiblly open a new issue adding information about the version of Terraform you are using and the version of the http provider that you are using?

from terraform-provider-http.