cycloidio / terracognita Goto Github PK
View Code? Open in Web Editor NEWReads from existing public and private cloud providers (reverse Terraform) and generates your infrastructure as code on Terraform configuration
License: MIT License
Reads from existing public and private cloud providers (reverse Terraform) and generates your infrastructure as code on Terraform configuration
License: MIT License
I used the command from Docker section: https://github.com/cycloidio/terracognita#docker
the program was working for a minute and then has crashed with the following error:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x13c6b59]
goroutine 1 [running]:
github.com/cycloidio/raws.(*connector).ListBuckets(0xc0003bcdc0, 0x532a300, 0xc000038018, 0x0, 0xc000b4e708, 0xc00043a0d0, 0xc000ca7a40)
/go/pkg/mod/github.com/cycloidio/[email protected]/list_buckets.go:34 +0x539
github.com/cycloidio/terracognita/aws.s3Buckets(0x532a300, 0xc000038018, 0xc000264bc0, 0x4b8f239, 0xd, 0x87e5810, 0x0, 0x0, 0x87e5810, 0x0, ...)
/app/aws/resources.go:401 +0x6e
github.com/cycloidio/terracognita/aws.(*aws).Resources(0xc000264bc0, 0x532a300, 0xc000038018, 0x4b8f239, 0xd, 0xc000410be0, 0x87e5810, 0x0, 0x0, 0x0, ...)
/app/aws/provider.go:66 +0xf6
github.com/cycloidio/terracognita/provider.Import(0x532a300, 0xc000038018, 0x5345740, 0xc000264bc0, 0x52d98c0, 0xc0004bc5c0, 0x0, 0x0, 0xc000410be0, 0xc000264bc0, ...)
/app/provider/import.go:33 +0x2f6
github.com/cycloidio/terracognita/cmd.glob..func1(0x87b9220, 0xc0003d0dc0, 0x0, 0x5, 0x0, 0x0)
/app/cmd/aws.go:66 +0x4d8
github.com/spf13/cobra.(*Command).execute(0x87b9220, 0xc0003d0d70, 0x5, 0x5, 0x87b9220, 0xc0003d0d70)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0x87b96e0, 0xc00043c500, 0xc0002a7f88, 0x40584f)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
/app/main.go:11 +0x32
Running the command the list out all aws_iam_policies and it's only ever returning a max of 100
terracognita aws --hcl policy.tf --tfstate .terraform/terraform.tfstate -i aws_iam_policy --access-key=$AWS_ACCESS_KEY_ID --secret-key=$AWS_SECRET_ACCESS_KEY --region=$AWS_REGION --verbose > output
the head of the verbose log output showing the return call for iam
ts=2019-09-26T22:09:32.809406641Z caller=import.go:47 func=provider.Import filters="\n\tTags: [],\n\tInclude: [aws_iam_policy],\n\tExclude: [],\n"
ts=2019-09-26T22:09:32.809421957Z caller=import.go:57 func=provider.Import resource=aws_iam_policy msg="fetching the list of resources"
ts=2019-09-26T22:09:32.857468923Z caller=import.go:70 func=provider.Import resource=aws_iam_policy id=arn:aws:iam::XXXX:policy/XXXX total=100 current=1 msg="reading from TF"
I also did a spot check against the aws cli command to check how many local/customer managed IAM policies the account is using/attached and it's definitely over 100. Not sure where it's being truncated.
aws list-policies --only-attached --scope Local | grep PolicyName | wc -l
The README is slowly growing, so TOC may be a nice addition
(require sdk upgrade)
- [x] azurerm_policy_assignment-> not supported azure sdk 3.0
iam policies are currently in-line in resources. this feature would be to support generating the aws_iam_policy_document
data source of terraform to be used in resource definitions.
e.g.
data "aws_iam_policy_document" "foo" {
statement {
effect = "Allow"
principals {
type = "AWS"
identifiers = [
"arn:aws:iam::blah"
]
}
actions = "sts:AssumeRole"
}
}
resource "aws_iam_role" "backups" {
name = "backups"
path = "/"
assume_role_policy = data.aws_iam_policy_document.foo.json
}
vs
resource "aws_iam_role" "foo" {
name = "backups"
path = "/"
assume_role_policy = <<POLICY
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::blah"
]
},
"Action": "sts:AssumeRole"
}
]
}
POLICY
}
It would be apt to include in the documentation the difference between this project (Terracognita) and its alternatives - Terraforming and Terraformer. It would be great if you can expand on what you said in this Reddit comment.
Right now we do not iterate over the results from AWS, we just use the first page, we should read all the pages of each resource to have the full list of services for that resource.
Related: #52
Right now the HCL that we generate has the IDs of the resources it needs (security_group, security_group_rule, policy ...).
The idea would be to be able to use the resource attributes to reference between them security_groups = ["${aws_security_group.elb-front.id}"]
this way would be more readable.
Hi,
I had this error while importing aws resources. It failed during retrieving S3:
Error: could not import from AWS: error reading S3 Bucket (console-backup): BucketRegionError: incorrect region, the bucket is not in 'eu-central-1' region at endpoint ''
status code: 301, request id: , host id:
while reading on type "aws_s3_bucket"
github.com/cycloidio/terracognita/provider.(*resource).Read
/Users/jbonhomm/Documents/Developpements/golang/src/github.com/cycloidio/terracognita/provider/resource.go:104
github.com/cycloidio/terracognita/provider.Import
/Users/jbonhomm/Documents/Developpements/golang/src/github.com/cycloidio/terracognita/provider/import.go:34
github.com/cycloidio/terracognita/cmd.glob..func1
/Users/jbonhomm/Documents/Developpements/golang/src/github.com/cycloidio/terracognita/cmd/aws.go:64
github.com/spf13/cobra.(*Command).execute
/Users/jbonhomm/Documents/Developpements/golang/pkg/mod/github.com/spf13/[email protected]/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
/Users/jbonhomm/Documents/Developpements/golang/pkg/mod/github.com/spf13/[email protected]/command.go:852
github.com/spf13/cobra.(*Command).Execute
/Users/jbonhomm/Documents/Developpements/golang/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main
/Users/jbonhomm/Documents/Developpements/golang/src/github.com/cycloidio/terracognita/main.go:11
runtime.main
/usr/local/go/src/runtime/proc.go:201
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1333
could not read resource aws_s3_bucket:
github.com/cycloidio/terracognita/provider.Import
/Users/jbonhomm/Documents/Developpements/golang/src/github.com/cycloidio/terracognita/provider/import.go:40
github.com/cycloidio/terracognita/cmd.glob..func1
/Users/jbonhomm/Documents/Developpements/golang/src/github.com/cycloidio/terracognita/cmd/aws.go:64
github.com/spf13/cobra.(*Command).execute
/Users/jbonhomm/Documents/Developpements/golang/pkg/mod/github.com/spf13/[email protected]/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
/Users/jbonhomm/Documents/Developpements/golang/pkg/mod/github.com/spf13/[email protected]/command.go:852
github.com/spf13/cobra.(*Command).Execute
/Users/jbonhomm/Documents/Developpements/golang/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main
/Users/jbonhomm/Documents/Developpements/golang/src/github.com/cycloidio/terracognita/main.go:11
runtime.main
/usr/local/go/src/runtime/proc.go:201
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1333
Cheers
Be able to always have the logs available with the information of the last run with -v
Always write the logs of the last run to $XDG_CACHE_HOME
. Should also be able to be modifield with --log-file=path
or LOG_FILE=path
.
Just always initialize the log writter to the LOG_FILE and if the -v
is used then use a io.MultiWriter
.
On the aws_instance.user_data
the attribute it's invalid as it's taking the sha1 from AWS and then it's all wrong. We have to change it to read the user_data_base64
which can be read and interpreated correctly.
Using docker to pull the latest version of terracognita I attempted to import a medium sized AWS account. Ran into the ListBuckets issue mentioned in issue #29. Reran with --exclude=aws_s3_bucket
, which seemed to run almost to completion. After showing SES related DEBUG output, I got the following panic()
:
Error: could not import from AWS: error while fmt HCL: In <standard input>: At 9438:5: expected: IDENT | STRING | ASSIGN | LBRACE got: NUMBER
error while Sync Config
github.com/cycloidio/terracognita/provider.Import
/app/provider/import.go:73
github.com/cycloidio/terracognita/cmd.glob..func1
/app/cmd/aws.go:66
github.com/spf13/cobra.(*Command).execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
/go/pkg/mod/github.com/spf13/[email protected]/command.go:852
github.com/spf13/cobra.(*Command).Execute
/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main
/app/main.go:11
runtime.main
/usr/local/go/src/runtime/proc.go:200
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1337
General information:
Describe the bug:
Could not read aws_iam_user_group_membership. User in question has no groups.
Log message
ts=2020-05-28T14:37:40.501085Z caller=import.go:102 func=provider.Import resource=aws_instance resource=aws_vpc resource=aws_vpc_peering_connection resource=aws_key_pair resource=aws_security_group resource=aws_subnet resource=aws_ebs_volume resource=aws_elasticache_cluster resource=aws_elb resource=aws_alb resource=aws_alb_listener resource=aws_alb_listener_rule resource=aws_alb_listener_certificate resource=aws_alb_target_group resource=aws_db_instance resource=aws_db_parameter_group resource=aws_db_subnet_group resource=aws_s3_bucket resource=aws_cloudfront_distribution resource=aws_cloudfront_origin_access_identity resource=aws_cloudfront_public_key resource=aws_cloudwatch_metric_alarm resource=aws_iam_access_key resource=aws_iam_account_alias resource=aws_iam_account_password_policy resource=aws_iam_group resource=aws_iam_group_membership resource=aws_iam_group_policy resource=aws_iam_group_policy_attachment resource=aws_iam_instance_profile resource=aws_iam_openid_connect_provider resource=aws_iam_policy resource=aws_iam_role resource=aws_iam_role_policy resource=aws_iam_role_policy_attachment resource=aws_iam_saml_provider resource=aws_iam_server_certificate resource=aws_iam_user resource=aws_iam_user_group_membership id=Terraformer total=1 current=1 msg="reading from TF"
Error: could not import from AWS: unexpected format of ID ("Terraformer"), expected <user-name>/<group-name1>/...
could not import resource aws_iam_user_group_membership with id Terraformer
github.com/cycloidio/terracognita/provider.(*resource).ImportState
/tmp/build/6b18b934/code/provider/resource.go:194
github.com/cycloidio/terracognita/provider.Import
/tmp/build/6b18b934/code/provider/import.go:103
github.com/cycloidio/terracognita/cmd.glob..func2
/tmp/build/6b18b934/code/cmd/aws.go:86
github.com/spf13/cobra.(*Command).execute
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main
/tmp/build/6b18b934/code/main.go:11
runtime.main
/usr/local/go/src/runtime/proc.go:200
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1337
Additional context
Add any other context about the problem here.
Using the following, demonstrates no groups or groups for specific user.
$ aws --profile management iam list-groups-for-user --user Terraformer
{
"Groups": []
}
$ aws --profile management iam list-groups
{
"Groups": []
}
Make it work like terraform import
which can only import by target (resource+ID) so have a parameter to be able to define an ID (or list) to import.
Could also improve it to be able to do something like --target "aws_instance:1,2,3" --target "aws_iam_user:1,2,3"
so we could import multiple resources with multiple IDs.
I followed the instructions to build my own binary but instead I got an error
๏ผ ~/go/src/github.com/cycloidio/terracognita
โฐโ make install
GO111MODULE=on CGO_ENABLED=0 GOARCH=amd64 go install -ldflags "-X github.com/cycloidio/terracognita/cmd.Version=v0.1.5-16-gae4c64f"
# github.com/cycloidio/terracognita/aws
aws/provider.go:46:32: impossible type assertion:
*"github.com/hashicorp/terraform/helper/schema".Provider does not implement "github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".ResourceProvider (wrong type for Apply method)
have Apply(*"github.com/hashicorp/terraform/terraform".InstanceInfo, *"github.com/hashicorp/terraform/terraform".InstanceState, *"github.com/hashicorp/terraform/terraform".InstanceDiff) (*"github.com/hashicorp/terraform/terraform".InstanceState, error)
want Apply(*"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".InstanceInfo, *"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".InstanceState, *"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".InstanceDiff) (*"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".InstanceState, error)
make: *** [install] Error 2
I am pretty new to Golang, so I am not sure what could be the possible error.
I also downloaded the terraform libraries via
go get -d github.com/terraform-providers/terraform-provider-aws
go get -d github.com/hashicorp/terraform
This is on macOS High Sierra (10.13.6) on go version go1.10.3 darwin/amd64
General information:
Describe the bug:
Security Group import asking for force replacement during terraform plan
Log message
Additional context
Using terracognita, I generated resource file and state files for security groups. Now when I'm running "terraform plan" command. Facing issues like force replacement and resource must be replaced
More Importantly "why terraform plan says:-->> 43 resources to add??
But on the contrary, we are using the same files which are generated from terracognita [PS: Alos, we have not added any resource manually ]
The imported resources need to be "classify" by services (network, compute, iam, etc.).
Currently, the resources are imported in one Terraform file. For a given supported cloud provider, the imported resources should be sorted following the type of the resource.
This could be generated from the Terraform resource name or from an explicit mapping.
General information:
Describe the bug:
Improvement
If the user have no group, empty group_membership is written
resource "aws_iam_user_group_membership" "ucpMM" {
groups = []
user = "myuser"
}
Would be nice to not write the resource if groups = []
. Use ListGroupsForUser
?
Log message
Additional context
The core logic to import is based on the internal code of Terraform before the TF-SDK came out. Now with the TF-SDK beeing more evolved I think we'll be able to use it to move more logic outside of Terracognita into the TF-SDK.
Migrate the core logic to TF-SDKv2 so we are not stuck
The main idea would be to kind of fake a server initialization using directly the Plugins. Normally how TF works is a GRPC communication between TF and the Providers, if what I'm saying is correct we should be able to initialize in code the TFProvider server and call it manually inside code without GRPC directly.
This is more or less what we are doing now but manually without any TF code in between, but with this change we would need to maintain less code as it'll be maintained by the TF-SDK directly.
All this said this is still a theory but we need to migrate to it somehow because we cannot use TF-SDKv2 internally as they have changed the imports path on the Providers to not use TF anymore but TF-SDKv2 and other changes, so we would be blocked.
General information:
On the #71 a test was introduced and Skipped https://github.com/cycloidio/terracognita/blob/master/aws/cmd/main_test.go#L16 because it was hard to make it pass.
The issue with that test is that it uses gomiports
which is not present on the Docker image when running the make test
, to add it just add a -v
with the path to the bin and the internal path to the bin (that one is easy), but then the real issue is that it does not add any of the imports as it does not know where they are because they are not present on the Docker image, so ideally another -v
of the entire $GOPATH/src
would solve that but it feels to extreme (did not try this last one).
Another idea would be to create an specific Docker image for TC with all the needed make generate
binaries and more things.
We've noticed that some default variables from Terraform or not directly imported via their importer method - some are already referenced as bug from the CLI importer -:
This issue is used to describe the current Roadmap of Terracognita. It will evolve over the time !
feature definition | issue |
---|---|
Sort an imported resource following its cloud service | #117 |
Be able to merge states | #41 |
Upgrade to terraform-plugin-sdk/v2 | #151 |
Import as Module | #141 |
To make the integration of new AWS resources and beeing able to solve some issues #52, we'll move https://github.com/cycloidio/raws/ inside of aws/
as the google/
provider has right now.
So basically not using it anymore.
We have 2 methods ListBuckets
and ListObjects
that should use Get*
instead of List*
as all the other methods use the Get*
.
Received the error:
could not import from AWS: Error retrieving EC2 tags for CloudFront Distribution [omitted] Throttling: Rate exceeded
and got a stack trace, with no output going to the output file.
There should be some way to see all of the complete resources that have been fetched so that even with throttling all resources can be fetched over time.
Changing the '--hcl' to be a directory instead of a file will let us modify the output of the HCL to be separated in different files like: ec2_instance.tf
, security_groups.tf
and so on.
The idea is to write a CONTRIBUTING.md file explaining what to do when we want to contribute. This would make it easier for external developers.
examples:
https://github.com/weaveworks/weave/blob/master/CONTRIBUTING.md
https://github.com/weaveworks/weave/wiki/WorkingOnWeave
we should switch our imports from
import "github.com/hashicorp/terraform...
to
import "github.com/hashicorp/terraform-plugin-sdk/terraform..."
This change should not bring breaking changes. Then we should be able to use the latest version of our supported providers.
This way we can not the version some one it's using.
It would be nice if a pre-built binary for Terracognita can be provided as part of the release. This will help newcomers to try out the tool themselves without having to resort to building it themselves with the GoLang tools.
Numerous projects including Terraform have been able to auto-build this as part their CI/CD pipeline, so I am sure this is doable for Terracognita project as well.
General information:
macOS 10.15.6
v0.5.1
binary release downloaded from github
Describe the bug:
I was trying to generate the config (especially for SES) but it aborts.
Could this be a permission issue for the given credentials?
Either way - it should not give a stack trace like this.
Log message
$ ./terracognita-darwin-amd64 aws --access-key ".." --secret-key ".." --region eu-central-1
Starting Terracognita with version v0.5.1
Importing with filters:
Tags: [],
Include: [],
Exclude: [],
Targets: [],
Importing aws_vpc [1/1] Done!
Importing aws_key_pair [1/1] Done!
Importing aws_security_group [1/1] Done!
Importing aws_subnet [3/3] Done!
Importing aws_alb_target_group [1/1] Done!
Importing aws_iam_access_key [2/2] Done!
Importing aws_iam_account_alias [1/1] Done!
Importing aws_iam_account_password_policy [1/1] Done!
Importing aws_iam_group [2/2] Done!
Importing aws_iam_group_membership [2/2] Done!
Importing aws_iam_group_policy_attachment [5/5] Done!
Importing aws_iam_policy [1/1] Done!
Importing aws_iam_role [5/5] Done!
Importing aws_iam_role_policy [3/3] Done!
Importing aws_iam_role_policy_attachment [4/4] Done!
Importing aws_iam_user [2/2] Done!
Importing aws_iam_user_policy_attachment [1/1] Done!
Importing aws_route53_resolver_rule_association [1/1] Done!
Error: could not import from AWS: InvalidAction: Unavailable Operation
status code: 400, request id: e1edbc82-c70d-4184-9852-c1843066ec7a
error while reading from resource "aws_ses_active_receipt_rule_set"
github.com/cycloidio/terracognita/aws.(*aws).Resources
/tmp/build/6b18b934/code/aws/provider.go:74
github.com/cycloidio/terracognita/provider.Import
/tmp/build/6b18b934/code/provider/import.go:91
github.com/cycloidio/terracognita/cmd.glob..func2
/tmp/build/6b18b934/code/cmd/aws.go:86
github.com/spf13/cobra.(*Command).execute
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main
/tmp/build/6b18b934/code/main.go:11
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1373
github.com/cycloidio/terracognita/provider.Import
/tmp/build/6b18b934/code/provider/import.go:93
github.com/cycloidio/terracognita/cmd.glob..func2
/tmp/build/6b18b934/code/cmd/aws.go:86
github.com/spf13/cobra.(*Command).execute
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/tmp/build/6b18b934/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main
/tmp/build/6b18b934/code/main.go:11
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1373
Additional context
At the beginning we started using the stdlib errors
to deal with errors, then at some point we started using golang.org/pkg/errors/
to enrich a bit more the errors, would be good to move all the code to one format and follow it.
During this time GO 1.13 also made appearance and the new API for errors with it, introducing the Is
and As
(https://golang.org/pkg/errors/) with a lib that offers the "full" implementation (https://godoc.org/golang.org/x/xerrors) as described on https://github.com/golang/go/wiki/ErrorValueFAQ and all the proposals.
So we have to decide on a format to move all the code to that one and stay with it.
General information:
Describe the bug:
It's more to keep a trace here, but when you try to import two resources with the same name (but different IDs), it fails with a duplicate key error.
Log message
could not import from AWS: the key already exists
with key "aws_instance.cycloidio-website-front-prod"
github.com/cycloidio/terracognita/state.(*Writer).Write
/home/mathieu/go/src/github.com/cycloidio/terracognita/state/writer.go:46
github.com/cycloidio/terracognita/provider.(*resource).State
/home/mathieu/go/src/github.com/cycloidio/terracognita/provider/resource.go:398
github.com/cycloidio/terracognita/provider.Import
/home/mathieu/go/src/github.com/cycloidio/terracognita/provider/import.go:134
github.com/cycloidio/terracognita/cmd.glob..func2
/home/mathieu/go/src/github.com/cycloidio/terracognita/cmd/aws.go:86
github.com/spf13/cobra.(*Command).execute
/home/mathieu/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/home/mathieu/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/home/mathieu/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main
/home/mathieu/go/src/github.com/cycloidio/terracognita/main.go:11
runtime.main
/usr/lib/go/src/runtime/proc.go:203
runtime.goexit
/usr/lib/go/src/runtime/asm_amd64.s:1373
error while calculating the satate of resource "aws_instance"
github.com/cycloidio/terracognita/provider.Import
/home/mathieu/go/src/github.com/cycloidio/terracognita/provider/import.go:136
github.com/cycloidio/terracognita/cmd.glob..func2
/home/mathieu/go/src/github.com/cycloidio/terracognita/cmd/aws.go:86
github.com/spf13/cobra.(*Command).execute
/home/mathieu/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/home/mathieu/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/home/mathieu/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main
/home/mathieu/go/src/github.com/cycloidio/terracognita/main.go:11
runtime.main
/usr/lib/go/src/runtime/proc.go:203
runtime.goexit
/usr/lib/go/src/runtime/asm_amd64.s:1373
Additional context
Add any other context about the problem here.
TC is currently using HCL v1:
Line 14 in 5709caa
To generate a TF12 compliant format we need to upgrade to HCL v2.x. APIs have been heavily modified including some breaking changes like:
printer
packagefmtcmd
packageresources:
From a provided .tfstate
, we should be able to merge the generated state in order to create a new updated state (result of the combination)
Using this feature, the user will be able to update a "partial" working .tfstate
with a legacy infra not declared as-code
.
from the documentation, it's asked to create a provider.tf
to use the generated HCL / tfstate files. Terracognita should be able to generate it by itself.
terraform {
backend "local" {
path = "./$TFSTATE_PATH"
}
}
provider "aws" {
access_key = var.access_key
secret_key = var.secret_key
region = var.region
}
variable "access_key" {}
variable "secret_key" {}
variable "region" {}
terraform {
backend "local" {
path = "./$TFSTATE_PATH"
}
}
provider "google" {
credentials = file("${var.credentials}")
project = var.project
region = var.region
}
variable "region" {}
variable "project" {}
variable "credentials" {}
TFSTATE_PATH
and region
could be filled from the CLI flags. The credentials could be filled to but it would required to have read / write permissions (TC only requires read permission)
I am trying to build the project following instructions using latest go 1.12.6
but i get the following error:
user@host:~# go get -u github.com/cycloidio/terracognita
# github.com/cycloidio/terracognita/aws
programmi/go/src/github.com/cycloidio/terracognita/aws/provider.go:46:32: impossible type assertion:
*"github.com/hashicorp/terraform/helper/schema".Provider does not implement "github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".ResourceProvider (wrong type for Apply method)
have Apply(*"github.com/hashicorp/terraform/terraform".InstanceInfo, *"github.com/hashicorp/terraform/terraform".InstanceState, *"github.com/hashicorp/terraform/terraform".InstanceDiff) (*"github.com/hashicorp/terraform/terraform".InstanceState, error)
want Apply(*"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".InstanceInfo, *"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".InstanceState, *"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".InstanceDiff) (*"github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/terraform".InstanceState, error)
Hey!
Currently your Dockerfile is not building.
System: MacOS 10.14
Docker: 19.03.4
Output:
$> make dbuild
Sending build context to Docker daemon 737.8kB
Step 1/12 : FROM golang:1.12 as builder
1.12: Pulling from library/golang
c7b7d16361e0: Pull complete
b7a128769df1: Pull complete
1128949d0793: Pull complete
667692510b70: Pull complete
c70d80036479: Pull complete
07b6f8daecbf: Pull complete
f41babb1524b: Pull complete
Digest: sha256:53b838583589af3bddf26c5217343acac84313751d8655e666cc1309279134e2
Status: Downloaded newer image for golang:1.12
---> bc0268f5ce47
Step 2/12 : ENV GO111MODULE=on
---> Running in 63ad8a97e600
Removing intermediate container 63ad8a97e600
---> ad8d81fccdac
Step 3/12 : WORKDIR /app
---> Running in 2e15fca0f375
Removing intermediate container 2e15fca0f375
---> d03a6909c448
Step 4/12 : COPY go.mod .
---> 61ead281f36c
Step 5/12 : COPY go.sum .
---> b6e91c4ecadb
Step 6/12 : RUN go mod download
---> Running in 9bd29596a937
go: finding github.com/terraform-providers/terraform-provider-aws v1.60.1-0.20191003145700-f8707a46c6ec
go: finding github.com/zclconf/go-cty v1.0.1-0.20190708163926-19588f92a98f
[...]
go: finding golang.org/x/tools v0.0.0-20190719005602-e377ae9d6386
Removing intermediate container 9bd29596a937
---> fc91384405da
Step 7/12 : COPY . .
---> b216462ac446
Step 8/12 : RUN GIT_TAG=$(git describe --tags --always) && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags "-X github.com/cycloidio/terracognita/cmd.Version=$GIT_TAG"
---> Running in 0f988e5f6556
/usr/local/go/pkg/tool/linux_amd64/link: signal: killed
The command '/bin/sh -c GIT_TAG=$(git describe --tags --always) && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags "-X github.com/cycloidio/terracognita/cmd.Version=$GIT_TAG"' returned a non-zero code: 1
make: *** [dbuild] Error 1
General information:
Describe the bug:
When there is no AWS password policy, this is usual 3 retries to find that there is no policy.
Right now the output it's basically the logs from terrafrom
, we should show those when we add the --verbose
flag, which does not exists ATM.
The desired output would be, the one without the --verbose
, this one:
Jun 19 12:17:01 INFO: Importing security_group [35/35] Done!
Jun 19 12:17:01 INFO: Importing aws_instance [10/35] ...
We still have some things TBD:
--verbose
? Or we want something custom?EDIT:
We'll have 2 flags -v
and -d
:
Jun 19 12:17:01 INFO: Importing security_group [35/35] Done!
Jun 19 12:17:01 INFO: Importing aws_instance [10/35] ...
-v
we'll have the logs we had before with more custom logs (Debug) like thisJun 19 12:17:01 DEBUG: Starting terracognita for region R - resources: X, Y, Z
Jun 19 12:17:01 INFO: Importing security_group [35/35] Done!
...
Jun 19 12:17:01 INFO: Importing aws_instance [9/35] ...
Jun 19 12:17:01 DEBUG: Importing aws_instance i-123lasdjk2
Jun 19 12:17:01 DEBUG: Writing HCL aws_instance i-123lasdjk2
Jun 19 12:17:01 DEBUG: Writing State aws_instance i-123lasdjk2
Jun 19 12:17:01 DEBUG: Finished importing aws_instance i-123lasdjk2
Jun 19 12:17:01 INFO: Importing aws_instance [10/35] ...
-d
we'll have the logs we had before with the TF ones:2019/06/20 12:13:06 [INFO] Building AWS auth structure
2019/06/20 12:13:06 [INFO] Setting AWS metadata API timeout to 100ms
2019/06/20 12:13:07 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2019/06/20 12:13:07 [INFO] AWS Auth provider used: "StaticProvider"
2019/06/20 12:13:07 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2019/06/20 12:13:07 [DEBUG] Trying to get account information via sts:GetCallerIdentity
Jun 19 12:17:01 DEBUG: Starting terracognita for region R - resources: X, Y, Z
Jun 19 12:17:01 INFO: Importing security_group [35/35] Done!
...
Jun 19 12:17:01 INFO: Importing aws_instance [9/35] ...
Jun 19 12:17:01 DEBUG: Importing aws_instance i-123lasdjk2
Jun 19 12:17:01 DEBUG: Writing HCL aws_instance i-123lasdjk2
Jun 19 12:17:01 DEBUG: Writing State aws_instance i-123lasdjk2
Jun 19 12:17:01 DEBUG: Finished importing aws_instance i-123lasdjk2
Jun 19 12:17:01 INFO: Importing aws_instance [10/35] ...
TODO aws_iam_group_membership
resource is not importable yes.
We Defined our own ResourceImporter
It should be removed when terraform will support it hashicorp/terraform-provider-aws#13795
Same for:
General information:
Describe the bug:
When trying to import IAM roles and IAM users it only imports configurations for first 100 resources.
Agenda: We want to import 200+ IAM roles
the user should be able to gauge the usage of TC for its own need by reading the documentation
Some feedback about Terracognita is the lack of examples / how-to / getting started. We should have a way to display use-case / examples. A wiki could be a good place in order to not have a huge README
N/A
General information:
Describe the bug:
terracognita google --credentials ./creds.json -i google_compute_instance -i google_compute_firewall --region europe-west2 --project my-project --hcl resources.tf --tfstate resources.tfstate
The import works but Terraform plan is failing.
Log message
$ terraform plan
Error: Unsupported argument
on resources.tf line 147, in resource "google_compute_instance" "dhHWx":
147: initialize_params = [{ image = "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-10-buster-v20200910", size = 10, type = "pd-standard" }]
An argument named "initialize_params" is not expected here. Did you mean to
define a block of type "initialize_params"?
Error: Unsupported argument
on resources.tf line 153, in resource "google_compute_instance" "dhHWx":
153: access_config = [{ nat_ip = "4.5.6.7", network_tier = "PREMIUM" }]
An argument named "access_config" is not expected here. Did you mean to define
a block of type "access_config"?
Error: Unsupported argument
on resources.tf line 187, in resource "google_compute_instance" "xoLPY":
187: access_config = [{ nat_ip = "1.2.3.4", network_tier = "PREMIUM" }]
An argument named "access_config" is not expected here. Did you mean to define
a block of type "access_config"?
Error: Unsupported argument
on resources.tf line 203, in resource "google_compute_instance" "xoLPY":
203: initialize_params = [{ image = "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-9-stretch-v20200521", size = 10, type = "pd-standard" }]
An argument named "initialize_params" is not expected here. Did you mean to
define a block of type "initialize_params"?
Additional context
It's a side effect of HCL2 migration. This blocks should be:
initialize_params {
image = "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-9-stretch-v20200521"
size = 10
type = "pd-standard"
}
General information:
Some resources like lb_listener_certificate
are not importable via terraform (https://www.terraform.io/docs/providers/aws/r/lb_listener_certificate.html) but we are able to get the information needed to write the terraform files.
We should find a way to write it.
Describe the bug:
resource like lb_listener_certificate
should be writable.
Log message
Additional context
General information:
Describe the bug:
When running against aws ...
TF_LOG=trace terracognita aws --access-key=AKI**** --secret-key=***** --region=eu-west-1 --debug --verbose
Log message
Trimmed content.
ts=2020-05-28T14:55:34.910696Z caller=import.go:102 func=provider.Import resource=aws_instance id=i-0a4fa244257965a67 total=22 current=3 msg="reading from TF"
2020/05/28 15:55:34 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeInstances Details:
2020/05/28 15:55:35 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeInstances Details:
2020/05/28 15:55:35 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
ts=2020-05-28T14:55:35.105867Z caller=retry.go:37 func=utils.Retry msg="waiting for Throttling error" times-left=2
2020/05/28 15:56:05 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeInstances Details:
2020/05/28 15:56:05 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeInstances Details:
2020/05/28 15:56:05 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
ts=2020-05-28T14:56:05.477216Z caller=retry.go:37 func=utils.Retry msg="waiting for Throttling error" times-left=1
2020/05/28 15:56:35 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeInstances Details:
2020/05/28 15:56:35 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeInstances Details:
2020/05/28 15:56:35 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
ts=2020-05-28T14:56:35.671009Z caller=import.go:119 func=provider.Import resource=aws_instance id=i-0a4fa244257965a67 total=22 current=3 error="the resource did not return an ID"
Additional context
So, I had 12 EC2 instances, 10 of which where terminated.
terracognita attempted to look at all the instances, then the reservations.
Not sure what else is going to be processed in the same way.
The command started at 2020-05-28 15:54:12 and it is finished at 2020-05-28 16:11:02.
General information:
Describe the bug:
Running:
$ curl -L https://github.com/cycloidio/terracognita/releases/download/v0.4.0/terracognita-darwin-amd64.tar.gz -o terracognita-darwin-amd64.tar.gz && tar -xf terracognita-darwin-amd64.tar.gz && chmod +x terracognita-darwin-amd64 && ./terracognita-darwin-amd64 version
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 645 100 645 0 0 1514 0 --:--:-- --:--:-- --:--:-- 1514
100 101M 100 101M 0 0 9.9M 0 0:00:10 0:00:10 --:--:-- 11.1M
The current version is: v0.3.0-65-g72cf96b
Don't insist on an AWS key/secret-key, ideally use profiles as the standard AWS SDK supports them.
Accepting a session-token parameter might be a vaguely acceptable mid-way step for your code...
Would be good to have an import directly as a Terraform Module so we are able to directly use the imported infrastructure as a module.
As the import is directly filled there is no way in which we can define which attributes of a Resource need to be defined as variables so they can be modified when it's used as a Module.
For this we could have an specific configuration for each Provider in which we would define from each resource which attributes are the most "common" to have variables so it can be changed from the Module.
The implementation I have in mind is to add a new flag of --module-name
that will generate the HCL as a Module which means to have the HCL inside a ./module-{module-name}/hcl.tf
and have also a ./{module-name}.tf
.
The {module-name}.tf
will source the ./module-{module-name}/
and have all the variables that are defined on the hcl.tf
with the filled data that it had when importing, so it can be generated as it was imported.
The config that was mentioned that we'll have for each Provider will be something like:
{
"aws_instance": [
"count",
],
}
It'll be able to be overrided by a --module-config
with the same format, in which case that one would be used instead of the default one.
Which means that any aws_instance.count
will be defined as a variable (different for each aws_instance
, something like var.aws_instance_{instance-name}_count
And then the module would be like:
module "{module-name}" {
source = "./module-{module-name}"
# If it has a value
aws_instance_{instance-name}_count = 4
# If it has no value inside of the module
# so the module usage can have them defined
# aws_instance_{instance-name}_count
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.