hashicorp / terraform-provider-awscc Goto Github PK
View Code? Open in Web Editor NEWTerraform AWS Cloud Control provider
Home Page: https://registry.terraform.io/providers/hashicorp/awscc/latest/docs
License: Mozilla Public License 2.0
Terraform AWS Cloud Control provider
Home Page: https://registry.terraform.io/providers/hashicorp/awscc/latest/docs
License: Mozilla Public License 2.0
Relates hashicorp/terraform-plugin-framework#34
As of v0.1.0 terraform-plugin-framework
does not support ForceNew attributes.
Once upstream support is added, enhance this provider.
Given a source directory of CloudFormation Resource Schema files from #15, an initial code generator should be written that creates corresponding Terraform Resource files. These files can use the output of #7 (or an empty/placeholder map) for schema and write empty/placeholder CRUD handler definitions (which will be more filled in via #8).
Dependent on: #17
make gen
and/or go generate ./...
) (re)creates all Terraform Resource files based on available CloudFormation Resource Schema files.We currently use the terraform-plugin-sdk
's resource.StateChangeConf
structure and associated WaitForState
method to poll the CloudAPI GetResourceRequestStatus
method waiting for a resource operation to complete.
The AWS Go v2 SDK has a powerful waiter mechanism but as of today the CloudAPI preview SDKs we have received have not had waiters implemented for GetResourceRequestStatus
.
It looks like we should be able to implement our own waiter for this operation.
Use for example NewTypeRegistrationCompleteWaiter
as a guide.
Relates #34.
Currently we have no support for generating data source schemas.
Eventually we will want to add both singular and plural data sources (as Cloud API supports List operations).
The resource schema code generation in internal/provider/generators/resource/main.go
need to be reworked to capture attribute paths for write-only properties (#42) and to correctly determine computed and force-new attributes.
Go back to a version of the code sketched out in #7 (comment) to generate the Schema
as a single structure.
CloudAPI returns resource information in a stringified JSON. Terraform Plugin Go represents state data in a TBD type which is updated via TBD calls currently. Implement this conversion (including PascalCase to snake_case) and generate for all resources, which effectively enables drift detection for configured attributes.
To ensure the viability of this project, the conversion package can print output representing the built schema structures. This output will closely represent the output necessary for code generation.
Currently this provider is using a custom version of the AWS Go v1 SDK.
This major version of the SDK is on a path towards EOL.
AWS will soon provide us with a custom version of the AWS Go v2 SDK.
Once available, migrate the Cloud API calls.
CloudFormation API client bootstrap and provider configuration will also need to be migrated.
Where possible, share implementation patterns with the similar proposed work in the AWS Provider.
If feasible, use an AWS Go v2 SDK-enabled version of the aws-sdk-go-base
package.
We are currently using the strcase
Go package to convert CloudFormation property names (e.g. Arn
or GlobalReplicationGroupDescription
) to their Terraform attribute name equivalents (arn
and global_replication_group_description
) and vice-versa.
However I have noticed that S3Bucket
is converted to s_3_bucket
, which is not what we want.
CloudAPI requires a stringified JSON desired state as input during creation. Terraform Plugin Go represents prior state (configuration) data in a TBD type which is fetched via TBD calls. Implement this conversion (including snake_case to PascalCase) and generate for all resources, which effectively enables very basic resource support.
Augment the code generation to include placeholder Create, Read, and Delete functions for all resources. In particular, the Create function should call CreateResource operation with a progress waiter and d.SetId() with the resource type and identifier. The Read function should perform GetResource operation with error checking and trigger resource recreation when receiving the appropriate resource not found error. The Delete function can perform a DeleteResource operation with a progress waiter.
Ensure that from day one of alpha the GitHub repository configuration matches HashiCorp standards and as much as possible is entirely managed by Terraform. Where possible these should be consistent with the existing provider assuming that standard makes sense.
Relates #33.
If role_arn
is configured on the provider, pass its value as RoleArn
to CloudAPI operations.
Relates hashicorp/terraform-plugin-framework#33.
As of v0.1.0 terraform-plugin-framework
does not support importing existing resources.
Once upstream support is added, enhance this provider.
Meet with the Kubernetes Alpha provider team, since that provider must deal with dynamic schemas and validation. They likely have valuable input on design considerations and possible functionality that will influence other tasks.
Given an input CloudFormation Resource Schema, generate the corresponding Terraform Resource Schema. All PascalCase naming must be converted to snake_case.
To use the current terraform-plugin-sdk
acceptance test framework a top-level attribute named id
must be declared in the resource schema. This is required for the state shim, otherwise errors are reported:
TF_ACC=1 go test ./internal/aws/logs -v -count 1 -parallel 20 -run=TestAccLogGroup_ -timeout 180m
=== RUN TestAccLogGroup_basic
=== PAUSE TestAccLogGroup_basic
=== CONT TestAccLogGroup_basic
testing_new_config.go:111: no "id" found in attributes
testing_new.go:53: no "id" found in attributes
--- FAIL: TestAccLogGroup_basic (3.70s)
The resource code generation should produce an attribute
"id": {
Description: `The unique resource ID.`,
Type: types.StringType,
Computed: true,
},
and the generic resource handlers should populate this field in state.
Relates: #38.
Relates: hashicorp/terraform-plugin-framework#80.
v0.2.0 of terraform-plugin-framework
is available.
Use it.
CloudFormation properties that are not required but are create-only should be emitted as Optional/Computed/ForceNew
attributes.
For example CloudWatch Logs log group name.
Currently we are missing the Computed
flag on these attributes.
ForceNew
requires #44.
Add a minimal set of build tooling to allow contributors to build, test, lint and format code.
Where possible share workflows with the AWS Provider.
CloudAPI requires a customized RFC 6902 JSON Patch input for updates. Terraform Plugin Go implements state updates in the TBD type which can be queried via TBD. This information must be converted to a JSON prior state, converted into a JSON desired state, and those two JSON documents compared into JSON Patch format.
CloudAPI operations support an optional RoleArn
parameter, which is an IAM Role that is used to do the actual provisioning (otherwise it defaults to the principal calling the operation). We should implement support for an optional string argument at the provider level, which can be passed through to all resource-level CloudAPI operations.
provider "awscloudapi" {
role_arn = "arn:aws:iam::123456789012:role/CloudAPIExecutionRole"
}
role_arn
attribute. If configured, all CloudAPI operations are called with the RoleArn
parameter and this configuration value. Should provide IAM Role ARN format validation (e.g. arn.Parse()
, service equals iam
, etc).Relates hashicorp/terraform-plugin-framework#68.
As of v0.1.0 terraform-plugin-framework
does not support per-resource, per-operation timeouts.
Once upstream support is added, enhance this provider.
In the meantime, ensure that the Cloud API waiter timeout values are high enough to cover all supported resource operations.
Relates hashicorp/terraform-plugin-framework#53.
As of v0.1.0 terraform-plugin-framework
does not support Terraform Sets.
This limits the resource schemas that can be generated.
Once upstream support is added, enhance this provider.
I just noticed that properties listed as required
in the CloudFormation schema aren't being marked as such in the generated Terraform schema code.
Fix that.
This provider implements generic CRUD handlers for all supported resource types.
The single code paths for each operation give us the opportunity to enable metrics consistently.
Consider using https://github.com/armon/go-metrics.
Until #34 is implemented, consider migrating to the latest CloudAPI-enabled Go v1 SDK: https://github.com/hashicorp/aws-sdk-go-private/tree/f-cloudapi-20210715.
Allegedly the new SDK supports standard JSON Patch operations via a new field, PatchDocument
- start to use this new field.
To generate a provider with all CloudAPI-compatible CloudFormation Resource Types, the schemas of each resource will need to be consistently readable from remote source(s) (e.g. CloudFormation API and/or GitHub repositories) and likely saved within the codebase. Once those schemas are available locally, then the generator can do its part.
There are currently a few unknowns with this setup as:
Dependent on: #17
sources = [
{
resource_type = "aws_logs_log_group"
git_source = "github.com/aws-cloudformation/aws-cloudformation-resource-providers-logs//aws-logs-loggroup/aws-logs-loggroup.json"
version = "a0363cb2ff6fd29e2e2312a116470baf4774e219"
},
]
resource_type
.Practitioners using the CloudAPI-based provider should be able to declare the AWS Region for all resources/operations at the provider level. Currently the AWS Region is hardcoded to us-west-2 for the beta period. This configuration should be required as there will be no default after beta.
provider "awscloudapi" {
region = "us-west-2"
}
Otherwise if not provided, reads AWS_DEFAULT_REGION
environment variable.
region
attribute. If configured, CloudAPI operations are called against that region's endpoint.AWS_DEFAULT_REGION
environment variable. If configured, CloudAPI operations are called against that region's endpoint.region
provider configuration and AWS_DEFAULT_REGION
environment variable are provided, configuration takes precedence.Relates hashicorp/terraform-plugin-framework#17.
A CloudFormation resource type schema supports various constraints on properties. In Terraform these correspond to attribute validation.
As of v0.1.0 terraform-plugin-framework
does not support attribute validation.
Once upstream support is added, enhance this provider.
All future tasks are dependent on being able to ingest the JSON CloudFormation Resource Provider Schema files. This includes but is not limited to validating the JSON schema against the CloudFormation Resource Provider meta schema, unmarshalling the JSON into Go.
Relates hashicorp/terraform-plugin-framework#63.
As of v0.1.0 terraform-plugin-framework
does not support terraform-plugin-log
.
Once upstream support is added, enhance this provider and remove calls to log.Printf
.
Currently this provider exposes just the role_arn
and region
configuration options (and doesn't use role_arn
), relying on environment variables to specify authentication arguments via the aws-sdk-go-base
package.
Determine and implement the minimal usable set of configuration options, a subset of the AWS Provider's configuration options.
Wherever possible use the same argument names as for the AWS Provider.
Depends on #34.
Minimal usable set of provider configuration options
In theory we can generate acceptance tests based on the CloudAPI schema definition in order to fully validate the behavior of the generated provider and protect against both local and upstream regression.
The terraform-plugin-framework
does not currently provide testing support, however as the existing SDK relies on binary testing it should be possible to use the terraform-plugin-sdk
to enable these tests to be run.
However, given the generated nature of the provider, and that the generation code is really the subject under test we may choose to rely mainly on unit tests. The detailed per-attribute, full lifecycle tests that we do for the existing provider, may not be appropriate given the vast range of services on offer from AWS and that the provider code is meant to be generic across services.
terraform-plugin-sdk
will allow us to test this provider.0.1
, 0.2
and 1.0
release.Relates #39.
Until the Plugin Framework has a Set
implementation, implement a local, temporary Set type.
This should unblock code generation for all AWS supported resources.
This codebase is based off terraform-provider-scaffolding
which gets us started, but we will need to determine where provider-specific files such as CloudFormation Resource Schemas, generated Terraform Resource files, and various internal packages/tooling should reside.
internal/provider/
|- provider(_test).go
- resource_{RESOURCE_TYPE}_gen.go # Generated Terraform Resource files
internal/service/cloudformation/
|- cfjsonpatch/ # TEMPORARY: RFC 6902 customizations for CloudFormation
|- cfschema/ # CloudFormation Resource Schema Go SDK
|- schemas/ # CloudFormation Resource Schemas cached from update tooling
|- schemas-sync/ # CloudFormation Resource Schema updater
- waiter/ # GetRequestStatus polling
CloudFormation Resource Provider Schemas encode all information about creation-only (ForceNew), write-only (ForceNew), read-only (wholly Computed), and deprecated properties at the root level using a /path/to/property syntax. The conversion package needs to implement lookup functionality to support those resource behaviors.
Generate the resource schemas into per-service packages instead of into a single large package.
Try and follow work in the "main" provider on how resource registration and bootstrapping would work.
Related: #17
The CloudFormation resource type schemas write-only. These are properties, such as passwords, that can be specified on resource creation but aren't returned when reading the current state of the resource.
In this provider's generic Read
handler, the new Terraform state value is constructed from the returned CloudFormation ResourceModel
without regard for any write-only properties.
Write-only properties should be copied from the prior Terraform state value into the new state value.
This will involved capturing the paths to all write-only properties during schema code generation and retaining this information in the generic resource type structure.
The CloudAPI supports operation cancellation:
You can use the CancelResourceRequest API to request the cancellation of an IN_PROGRESS request, passing the
RequestToken
from the CreateResource, UpdateResource, or DeleteResource APIs.
Consider supporting this functionality when context.Context
signals a user cancellation (^C).
To ensure consistency and improve error messaging, standardized warning and error diagnostics can be created.
func ResourceNotFoundAfterCreationDiag(err error) diag.Diagnostic {
return diag.Diagnostic{
Severity: diag.Error,
Summary: "AWS Resource Not Found After Creation",
Detail: fmt.Sprintf("After creating the AWS resource and attempting to read the resource, the API returned a resource not found error. This is typically an error with the Terraform resource implementation. Original Error: %s", err.Error()),
}
}
func ResourceNotFoundWarningDiag(err error) diag.Diagnostic {
return diag.Diagnostic{
Severity: diag.Warning,
Summary: "AWS Resource Not Found During Refresh",
Detail: fmt.Sprintf("Automatically removing from Terraform State instead of returning the error, which may trigger resource recreation. Original Error: %s", err.Error()),
}
}
func ServiceOperationEmptyResultDiag(service string, operation string) diag.Diagnostic {
return diag.Diagnostic{
Severity: diag.Error,
Summary: "AWS SDK Go Service Operation Empty Response",
Detail: fmt.Sprintf("Calling %s service %s operation returned missing contents in the response. This is typically an error with the API implementation.", service, operation),
}
}
func ServiceOperationErrorDiag(service string, operation string, err error) diag.Diagnostic {
return diag.Diagnostic{
Severity: diag.Error,
Summary: "AWS SDK Go Service Operation Unsuccessful",
Detail: fmt.Sprintf("Calling %s service %s operation returned: %s", service, operation, err.Error()),
}
}
func ServiceOperationWaiterErrorDiag(service string, operation string, err error) diag.Diagnostic {
return diag.Diagnostic{
Severity: diag.Error,
Summary: "AWS SDK Go Service Operation Incomplete",
Detail: fmt.Sprintf("Waiting for %s service %s operation completion returned: %s", service, operation, err.Error()),
}
}
Can drop Diag
in function naming if moved into its own package.
To avoid practitioner confusion we don't want the default provider configuration to conflict with the current AWS Provider (aws_
prefix).
Obvious suggestions:
cloudapi_
awscloudapi_
https://github.com/hashicorp/terraform-plugin-framework is the next-generation Terraform Plugin framework.
This provider will target this new framework.
Start the work to target v0.1.0 of terraform-plugin-framework
.
Provider
interfaceResourceType
and Resource
interfacesSchema
and Attribute
typesGiven that the current version of terraform-plugin-docs relies on Terraform CLI output (terraform providers schema -json
), the tools should work out of the box with the generated provider. This task captures validating that this process will work, and whether are not there are any shortcoming or modifications required to the tool or to the generated provider.
The scope that the tool needs to satisfy in this case is minimal:
Relates: #9.
Validate that we can test the existing release process and that it works for this new provider.
Check with Registry Team and Release team what needs to be done to ensure that the first release works, and how it can be testing without making an artifact public.
Decide what the process is for pre-GA builds vs post.
Some notes from Megan:
Builds should be repeatable, and be one click.
Relates #35
The Terraform HCL v2 SDK no longer implements (or enforces) an implicit id
attribute for every resource.
As the Cloud API has the idea of a unique resource instance identifier, generate an id
attribute for every resource type and map it to the Cloud API unique ID.
Related: #28.
The Cloud API (currently) names this as Identifier
. For consistency, generated resources will name this attribute identifier
.
A CloudFormation resource schema may (in the future) have a version
attribute. The CloudAPI may support this field to lock a resource definition to a major version of a resource schema.
Relates: #15.
Documentation generation via terraform-plugin-docs
can begin when Terraform Resources (and at least their schema) are being created via the code generation. This will allow stakeholders to investigate the current coverage and verbiage for appropriateness in Terraform. The CloudFormation schema providers resource level and property level documentation, but does not provide detailed service level documentation. There is also clear inconsistencies between the level of documentation in each service.
It is expected that the generated documentation will be much more minimal than practitioners are used to seeing in the existing provider.
terraform-plugin-docs
to run with other generation command(s).terraform-plugin-docs
has been synchronized on pull requests and the main branch.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.