minamijoyo / tfschema Goto Github PK

View Code? Open in Web Editor NEW

292.0 6.0 15.0 738 KB

A schema inspector for Terraform / OpenTofu providers

License: MIT License

Go 98.14% Makefile 0.51% Dockerfile 1.05% Shell 0.30%

terraform go opentofu

tfschema's People

Contributors

Stargazers

Watchers

Forkers

yamamoto-febc pchalamet avjafrey keiichi-hikita wan92hen sysunix env0 3mard b-rad36 standardgalactic bugcy013 thor-consulting iq-scm chenrui333

tfschema's Issues

Dynamic library issues

The v0.7.6 release does not work on Apline or Ubutnu (or probably Debian, either) due to dynamic linking issues. This is a known consequence of upgrading from Go 1.19 to 1.20 or later. You need to explicitly set CGO_ENABLED=0 in your build.

See golang/go#58550 for some details.

Add RST formatter

Hi minamijoyo,

Nice to e-meet you.
My name is Keiichi Hikita, Custom Provider Developer of our cloud service "Enterprise Cloud 2.0" .
(https://www.ntt.com/business/services/cloud/iaas/enterprise-cloud.html)

(Exactly saying, this is not a "issue" ...)

We were finding good management way of documentation of each resources, especially about Schema, then found this good and suitable tool, tfschema.
(I found it here: http://engineer.crowdworks.jp/entry/2018/03/27/173704)

But in our service, all documentations are managed as "rst(sphinx)".
So I would like to develop "rst formatter" of tfschema.

Would it be okay to propose this?
If you allow me about this, I would like to develop that and would like to send PR.
(From me, of maybe my team's other member)

Anyway, I would like to ask your impression on this point.

Thanks.

Predictable json schema

Hello,

I'm using your tools as mean to extract plugins schemas but since json output depends largely on named attributes and block_types it's harder to parse.

I've reworked the code (can be found here: https://github.com/pchalamet/tfschema) in order to output this json format to be easily machine parseable. Here is an output example for tfschema resource show -format=json azurerm_virtual_network

{
    "attributes": [
        {
            "name": "address_space",
            "type": "List(String)",
            "required": true,
            "optional": false,
            "computed": false,
            "sensitive": false
        },
        {
            "name": "dns_servers",
            "type": "List(String)",
            "required": false,
            "optional": true,
            "computed": false,
            "sensitive": false
        },
        ...
    ],
    "block_types": [
        {
            "name": "subnet",
            "attributes": [
                {
                    "name": "address_prefix",
                    "type": "String",
                    "required": true,
                    "optional": false,
                    "computed": false,
                    "sensitive": false
                },
                {
                    "name": "security_group",
                    "type": "String",
                    "required": false,
                    "optional": true,
                    "computed": false,
                    "sensitive": false
                },
                ...
            ],
            "block_types": [],
            "nesting": 3,
            "min_items": 0,
            "max_items": 0
        }
    ]
}

This means attributes have no more children of unknown keys - it's an array of well defined structure (new field is name). Same for block_types.

Before submitting a PR, I would like to be sure you agree with aforementioned changed. If you agree, I will send you the PR.

Thanks btw for your cool work !

Nested blocks are broken with terraform-provider-aws v2.7.0

This may be caused by a change for Terraform v0.12 support in the provider.

$ terraform version
Terraform v0.11.13
+ provider.aws v2.7.0

$ tfschema --version
0.2.0

$ tfschema resource show aws_security_group
+------------------------+-----------------------------------------------+----------+----------+----------+-----------+
| ATTRIBUTE              | TYPE                                          | REQUIRED | OPTIONAL | COMPUTED | SENSITIVE |
+------------------------+-----------------------------------------------+----------+----------+----------+-----------+
| arn                    | String                                        | false    | false    | true     | false     |
| description            | String                                        | false    | true     | false    | false     |
| egress                 | Set(Object(map[string]Type{"self":Bool,       | false    | true     | true     | false     |
|                        | "description":String,                         |          |          |          |           |
|                        | "from_port":Number,                           |          |          |          |           |
|                        | "prefix_list_ids":List(String),               |          |          |          |           |
|                        | "protocol":String,                            |          |          |          |           |
|                        | "cidr_blocks":List(String),                   |          |          |          |           |
|                        | "to_port":Number,                             |          |          |          |           |
|                        | "ipv6_cidr_blocks":List(String),              |          |          |          |           |
|                        | "security_groups":Set(String)}))              |          |          |          |           |
| id                     | String                                        | false    | true     | true     | false     |
| ingress                | Set(Object(map[string]Type{"protocol":String, | false    | true     | true     | false     |
|                        | "self":Bool, "prefix_list_ids":List(String),  |          |          |          |           |
|                        | "from_port":Number, "to_port":Number,         |          |          |          |           |
|                        | "security_groups":Set(String),                |          |          |          |           |
|                        | "cidr_blocks":List(String),                   |          |          |          |           |
|                        | "description":String,                         |          |          |          |           |
|                        | "ipv6_cidr_blocks":List(String)}))            |          |          |          |           |
| name                   | String                                        | false    | true     | true     | false     |
| name_prefix            | String                                        | false    | true     | false    | false     |
| owner_id               | String                                        | false    | false    | true     | false     |
| revoke_rules_on_delete | Bool                                          | false    | true     | false    | false     |
| tags                   | Map(String)                                   | false    | true     | false    | false     |
| vpc_id                 | String                                        | false    | true     | true     | false     |
+------------------------+-----------------------------------------------+----------+----------+----------+-----------+

block_type: timeouts, nesting: NestingSingle, min_items: 0, max_items: 0
+-----------+--------+----------+----------+----------+-----------+
| ATTRIBUTE | TYPE   | REQUIRED | OPTIONAL | COMPUTED | SENSITIVE |
+-----------+--------+----------+----------+----------+-----------+
| create    | String | false    | true     | false    | false     |
| delete    | String | false    | true     | false    | false     |
+-----------+--------+----------+----------+----------+-----------+

Type.Name() should use HCL2 typeexpr.TypeString

Related to #16,

it seems that HCL2 has a type annotation parser but no writer

Oh, this is not true. I found it !!
https://github.com/hashicorp/hcl2/blob/4b22149b7cef7272799ac85dca150e553d667971/ext/typeexpr/public.go

We should use this.

Could not execute `tfschema resource list aws`

Hi, I just installed tfschema with brew, but failed to execute first tfschema command in Mac Sierra.

% uname -a
Darwin A-M-201711-003 16.7.0 Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64 x86_64

When I try to execute tfschema resource list aws, it says

% tfschema resource list aws
Failed to find plugin: aws. Plugin binary was not found in any of the following directories: [., /Users/ksuzuki/usr/local/bin, terraform.d/plugins/darwin_amd64, .terraform/plugins/darwin_amd64, /Users/ksuzuki/.terraform.d/plugins, /Users/ksuzuki/.terraform.d/plugins/darwin_amd64, /Users/ksuzuki/workspace/go/bin]

However, my aws command already exists under valid directory.

% which aws
/Users/ksuzuki/usr/local/bin/aws

OpenTofu support

https://opentf.org/announcement

We want to understand how tfschema is affected by the OpenTF announcement, and what the plans for support are moving forward.

Failed to find plugin with Terraform v0.13.0-beta1

Terraform v0.13.0-beta1 has been released.
https://github.com/hashicorp/terraform/releases/tag/v0.13.0-beta1

TF v0.13 changed the default location for storing provider's binary to support downloading community providers. This causes an error: Failed to find plugin.

The default paths on macOS are as follows:

v0.12: .terraform/plugins/darwin_amd64/terraform-provider-aws_v2.64.0_x4
v0.13: .terraform/plugins/registry.terraform.io/hashicorp/aws/2.64.0/darwin_amd64/terraform-provider-aws_v2.64.0_x4

We should check the new location.

$ terraform --version
Terraform v0.13.0-beta1
+ provider registry.terraform.io/hashicorp/aws v2.64.0

$ tfschema resource show aws_security_group
Failed to NewClient: Failed to find plugin: aws. Plugin binary was not found in any of the following directories: [., /usr/local/bin, terraform.d/plugins/darwin_amd64, .terraform/plugins/darwin_amd64, /Users/masayuki.morita/.terraform.d/plugins, /Users/masayuki.morita/.terraform.d/plugins/darwin_amd64, /Users/masayuki.morita/bin]

$ ls -la .terraform/plugins/registry.terraform.io/hashicorp/aws/2.64.0/darwin_amd64/terraform-provider-aws_v2.64.0_x4
-rwxr-xr-x  1 masayuki.morita  staff  185615296  6  4 10:01 .terraform/plugins/registry.terraform.io/hashicorp/aws/2.64.0/darwin_amd64/terraform-provider-aws_v2.64.0_x4

Show if arguments require a new resource i.e. `ForceNew: true`

Hi @minamijoyo.

Is it possible to grab the ForceNew arguments for resources and build a new column to show which resource arguments cause a recreation ?

For example: https://github.com/hashicorp/terraform-provider-aws/blob/main/aws/resource_aws_ecs_service.go

✗ tfschema resource show aws_ecs_service
+------------------------------------+-------------+----------+----------+----------+-----------+-----------+
| ATTRIBUTE                          | TYPE        | REQUIRED | OPTIONAL | COMPUTED | SENSITIVE | FORCENEW  |
+------------------------------------+-------------+----------+----------+----------+-----------+-----------+
| cluster                            | string      | false    | true     | true     | false     | true      |

Drop hashicorp/terraform dependency

The Terraform v0.15.4 moved all Go packages to internal.
hashicorp/terraform#28723

This means that we can no longer directly import hashicorp/terraform as a Go library. As a result, it will be difficult to support future Terraform releases.

The current implementation of the tfschema depends on the following packages:

plugin/discovery: Find a plugin
plugin: Start a plugin client for protocol v5
plugin6: Start a plugin client for protocol v6
providers: Call provider's APIs
configs/configschema: Decode API respose types

I investigated some alternatives such as terraform-plugin-go, terraform-plugin-sdk, etc., but these are provider development libraries and do not include the grpc client side implementation. Unfortunately, the grpc proto file and the Go code generated from it are also internal and cannot be imported directly. If we would reimplement the provider client, it would be possible to copy and reuse the proto file under the original MPL license. However the problem is not only for the provider client, but also the implementations of discovering and starting plugin and decoding types. So if we go this direction, it would need to reimplement lots of the Terraform internals.

Another option in my mind is to use Terraform as a CLI and parse the output of the terraform providers schema -json.
https://www.terraform.io/docs/cli/commands/providers/schema.html

Historically, the tfschema has been written before the official providers schema command was added, but now it's a reasonable option. A downside of this option is only that it requires initializing a backend to access a tfstate, which is needed to detected dependencies for working destroy action correctly.

hashicorp/terraform#24261

Typically accessing the backend requires credentials for cloud providers, it is not desirable for the tfschema. It might be avoided by executing terraform init in a temporary directory and caching provider's binary, but invoking a completion probably becomes slow.

Failed to find plugin with Terraform v0.14.0-beta2

Terraform v0.14.0-beta2 has been released.
https://github.com/hashicorp/terraform/releases/tag/v0.14.0-beta2

TF v0.14 changed the default location for storing provider's binary to support locking provider dependencies. This causes an error: Failed to find plugin.

The default paths on macOS are as follows:

v0.13: .terraform/plugins/registry.terraform.io/hashicorp/aws/3.12.0/darwin_amd64/terraform-provider-aws_v3.12.0_x5
v0.14: .terraform/providers/registry.terraform.io/hashicorp/aws/3.12.0/darwin_amd64/terraform-provider-aws_v3.12.0_x5

We should check the new location.

In addition, a hidden selection file was changed to a new lock file format in HCL.

v0.13: .terraform/plugins/selections.json
v0.14: .terraform.lock.hcl

We also need to parse it to build search paths.

$ terraform -v
Terraform v0.14.0-beta2
+ provider registry.terraform.io/hashicorp/aws v3.12.0

$ tfschema -v
0.5.0

$ tfschema resource show aws_security_group
Failed to NewClient: Failed to find plugin: aws. Plugin binary was not found in any of the following directories: [., /usr/local/bin, terraform.d/plugins/darwin_amd64, .terraform/plugins/darwin_amd64, /Users/masayuki.morita/.terraform.d/plugins, /Users/masayuki.morita/.terraform.d/plugins/darwin_amd64, /Users/masayuki.morita/bin]

$ ls -la .terraform/providers/registry.terraform.io/hashicorp/aws/3.12.0/darwin_amd64/terraform-provider-aws_v3.12.0_x5
-rwxr-xr-x  1 masayuki.morita  staff  196627744 10 29 10:16 .terraform/providers/registry.terraform.io/hashicorp/aws/3.12.0/darwin_amd64/terraform-provider-aws_v3.12.0_x5

$ ls -la
total 16
drwxr-xr-x   5 masayuki.morita  staff   160 10 29 10:16 .
drwxr-xr-x  30 masayuki.morita  staff   960 10 29 10:11 ..
drwxr-xr-x   3 masayuki.morita  staff    96 10 29 10:16 .terraform
-rwxrwxrwx   1 masayuki.morita  staff  1031 10 29 10:16 .terraform.lock.hcl
-rw-r--r--   1 masayuki.morita  staff   203 10 29 10:15 main.tf

$ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "3.12.0"
  constraints = "3.12.0"
  hashes = [
    "h1:3gkfYjOVSHc3g/eXnk/JnRuoYtoDRu1oV3YPmBnuVtY=",
    "zh:002f7c17f927978e06c5c6ff7dd3f4600f0ab7ec7e43e62eedfce2b22c70b009",
    "zh:0400f415121efad618d9ddca71522eb4d0958905e705daa6ecd9f7d8c1330253",
    "zh:285e630188f7d0fd3d1e6f715326723904f02a2fa571e40dde36ae1b5cd5542e",
    "zh:402140c0a7d0423ff7c49b6bd03769332b8beb41838bf58eb83b7cfa73f67e09",
    "zh:7506eadb178fa41c02a0127e06ab4a49f2604860f0992f7c4f519993932d2a41",
    "zh:95dffd98bad81a2288953f36250f358ab9560d6bfb249ff5d00bf4f3131cc08b",
    "zh:aa530b43e4a2ff89025d8b4ed2384b67cadedbf84c67ee903bac133376d929b9",
    "zh:b2314b788a6a7091f4d70bff3825077e06c9c994b1efe76ad5f0e4b60944e60b",
    "zh:b68d3df21d7d67641e08476bd456d04bd33d963caba41024a14dddd85b0bde10",
    "zh:b9a8204ce6ab4929145cd2a284ba89790813692c6de9c8e12be9c1d984989180",
  ]
}