Hello Team,
We are more than happy to have MySQL (Beta) by Aiven :)
Would it be possible to add the support inside Terraform provider ?
Thanks,
Alex

Summary

When moving an existing Aiven Kafka cluster from the Public Network to a newly created VPC, the operation does not wait for the VPC to be ready and fails.

Morevover, the resulting Terraform state then seems to be left inconsistent: the Kafka cluster is still in the public cloud, but executing terraform apply reports that no modification is required.

Versions

Platform: OSX 10.14.2

> terraform --version
Terraform v0.11.13

# terraform provider 1.0.9: 
> cksum terraform-provider-aiven
3376569139 17751024 terraform-provider-aiven

How to reproduce

First create a Kafka cluster in the public network:

# Kafka service
resource "aiven_service" "kafka-dev" {

  project = "${aiven_project.visitor-count.project}"
  plan = "startup-2"
  service_name = "kafka-cluster"
  service_type = "kafka"

  cloud_name = "aws-eu-central-1"
 
  kafka_user_config {
    // connect, schema registry and rest are not available in startup plan:
    kafka_connect = false
    kafka_rest = false
    schema_registry = false

    kafka_version = "2.2"

    kafka {
      group_max_session_timeout_ms = 70000
      log_retention_bytes = 1000000000
    }
  }
}

Run terraform apply

then update the terraform script by adding a vpc:

resource "aiven_project_vpc" "bac-aiven-kafka-dev" {
  project = "${aiven_project.visitor-count.project}"
  cloud_name = "aws-eu-central-1"
  network_cidr = "10.67.0.0/24"
}

update the kafka service to now be deployed in that VPC:

  project_vpc_id = "${aiven_project_vpc.bac-aiven-kafka-dev.id}"

Run terraform apply again

Expected behaviour

Both executions of terraform apply should succeed. After the second, the cluster should be inside the specified VPC.

actual behaviour

The second execution of terraform apply fails as follows:

409: {"errors":[{"message":"Project VPC 843a0a53-5aa7-4a44-bd7f-ef2695ba3a4b is in state APPROVED, not ACTIVE","status":409}],"message":"Project VPC 843a0a53-5aa7-4a44-bd7f-ef2695ba3a4b is in state APPROVED, not
 ACTIVE"} -

workaround

Creating the VPC first and then moving the Kafka cluster to it when ready seems to work.

More info

I understand this PR is related to the issue. I confirm I am using release 1.0.9, which includes this PR.

#53

Summary

When creating an aiven_service with Terraform and not specifying values for maintenance_window_time nor maintenance_window_dow always, Terraform always considers the service to require an update

Versions

Platform: OSX 10.14.2

> terraform --version
Terraform v0.11.13

# terraform provider 1.0.9: 
> cksum terraform-provider-aiven
3376569139 17751024 terraform-provider-aiven

How to reproduce

Using this Terraform snippet:

# Kafka service
resource "aiven_service" "kafka-dev" {


  project = "${aiven_project.visitor-count.project}"
  cloud_name = "aws-eu-central-1"
  plan = "startup-2"
  service_name = "kafka-cluster"
  service_type = "kafka"

  kafka_user_config {
    kafka_connect = false
    kafka_rest = false
    schema_registry = false

    kafka_version = "2.2"

    kafka {
      group_max_session_timeout_ms = 70000
      log_retention_bytes = 1000000000
    }
  }
}

Run terraform apply then right after terraform plan

Expected behaviour

terraform plan should report that no change needs to be applied to the target environement

actual behaviour

terraform plan reports the following:

Terraform will perform the following actions:

  ~ aiven_service.kafka-dev
      maintenance_window_dow:  "friday" => ""
      maintenance_window_time: "07:39:02" => ""

workaround

Specifying the following two ignore statements at the beginning of the resource makes the behaviour disappear, although it defeats a bit the point of those two fields.

  lifecycle {
    ignore_changes = [
      "maintenance_window_dow",
      "maintenance_window_time"]
  }

Add support for crossregion VPC peering. Currently the provider only support single-region VPC peering.

Can you implement a way to stop and start Aiven resource like we can do it using WebUI now?

Perhaps add additional attribute to existing TF resource types to control power state.

# Kafka service resource "aiven_service" "samplekafka" { project = "${aiven_project.sample.project}" cloud_name = "google-europe-west1" powerstate = "${var.powerstate}" ... }

Hello,

We tried to create a topic using the cleanup policy "delete,compact", but we encountered the following error:

Invalid input for cleanup_policy: 'delete,compact' is not one of ['compact', 'delete']"

I checked the Kafka documentation and there is mentioned that the cleanup policy option supports either one of them or both of them.
https://kafka.apache.org/documentation/#topicconfigs

In the Aiven REST API documentation is mentioned that it's supports only one of them.
https://api.aiven.io/doc/#api-Service__Kafka-ServiceKafkaTopicCreate

Do you have any plans to support also the "delete, compact" option in your apps? The delete and compact options have different behaviours. More about the compaction: https://kafka.apache.org/documentation/#compaction

Thank you!
Laura

When creating a user in Aiven using terraform, I need to be able to retrieve the user's access certificate and access key. This information is required to be able to connect to the specified service.

When trying to create a Kafka cluster with the following configuration, the provider keeps waiting for more than 20 minutes, and eventually crashes, even though the cluster is up and perfectly running:

  + module.aiven-kafka.aiven_service.kafka
      id:                                  <computed>
      cassandra.#:                         <computed>
      cloud_name:                          "google-europe-west3"
      elasticsearch.#:                     <computed>
      grafana.#:                           <computed>
      influxdb.#:                          <computed>
      kafka.#:                             <computed>
      kafka_user_config.#:                 "1"
      kafka_user_config.0.custom_domain:   "<<value not set>>"
      kafka_user_config.0.kafka_connect:   "true"
      kafka_user_config.0.kafka_rest:      "false"
      kafka_user_config.0.kafka_version:   "2.2"
      kafka_user_config.0.schema_registry: "false"
      maintenance_window_dow:              "tuesday"
      maintenance_window_time:             "00:00:00"
      mysql.#:                             <computed>
      pg.#:                                <computed>
      plan:                                "business-4"
      project:                             "frlm-a-svc"
      redis.#:                             <computed>
      service_host:                        <computed>
      service_name:                        "svcs-pltf-kafka-cluster-preprod"
      service_password:                    <computed>
      service_port:                        <computed>
      service_type:                        "kafka"
      service_uri:                         <computed>
      service_username:                    <computed>
      state:                               <computed>
      termination_protection:              "true"

Hi,

Would you be interested in upgrading this project to use the Terraform Plugin SDK rather than having the full Terraform dependency?

If you are then I will happily submit a PR to do this - it would require the project to make sure it is using Terraform 0.12.7 as a minimum though

Thoughts?

Paul

It would be great to publish binary releases of the terraform provider on github (for supported platforms, such as darwin_amd64, linux_amd64 and windows_amd64).

Installing Go, dependencies and compiling is quite time consuming.
Downloading a binary and coping to ~/.terraform.d/plugins/ would be much easier.

Used the plugin from ... for provisioning a kafka 2.0 service with users and topics, which worked fine. I'm using S3 as a backend for the terraform configuration.

Switched to your plugin with version 1.0.1 and fixed the syntax changes in the terraform script.

Now, I get the following crash when I run terraform plan

panic: runtime error: index out of range
2018-10-19T09:19:52.718Z [DEBUG] plugin.terraform-provider-aiven: 
2018/10/19 09:19:52 [ERROR] root: eval: *terraform.EvalRefresh, err: aiven_service.my-kafka: unexpected EOF
2018-10-19T09:19:52.719Z [DEBUG] plugin.terraform-provider-aiven: goroutine 41 [running]:
2018/10/19 09:19:52 [ERROR] root: eval: *terraform.EvalSequence, err: aiven_service.my-kafka: unexpected EOF
2018/10/19 09:19:52 [TRACE] [walkRefresh] Exiting eval tree: aiven_service.my-kafka
2018/10/19 09:19:52 [TRACE] root: eval: *terraform.EvalIf
2018/10/19 09:19:52 [TRACE] root: eval: *terraform.EvalCountFixZeroOneBoundary
2018/10/19 09:19:52 [TRACE] Graph after step *terraform.ReferenceTransformer:

Can you help?

➜  nonprod git:(master) ✗ terraform import module.aiven.aiven_kafka_topic.boost_account_migration_v1 ovo-uat/kafka-uat/boost_account_migration_v1
module.aiven.aiven_kafka_topic.boost_account_migration_v1: Importing from ID "ovo-uat/kafka-uat/boost_account_migration_v1"...

Error: module.aiven.aiven_kafka_topic.boost_account_migration_v1 (import id: ovo-uat/kafka-uat/boost_account_migration_v1): import module.aiven.aiven_kafka_topic.boost_account_migration_v1 (id: ovo-uat/kafka-uat/boost_account_migration_v1): 404: {"errors":[{"message":"Topic 'boost_account_migration_v1' does not exist","status":404}],"message":"Topic 'boost_account_migration_v1' does not exist"} -

the project and service exists, the topic is there I have double checked the console.

I did import many resources a month ago. Everything was perfect but today it started to fall apart.
I have updated the plugin to use the latest 1.0.5 which was released couple of hours ago.

Other issue I found is that when I run terraform plan it shows the already imported topics as they were new topics (they appear in the addition section) despite the fact that they are there in the terraform state when I run terraform state list.
I can imagine that the two issues are related.
Could you pls take a look into this issue.
I can give you more details if you need it.
thx

The database resource can be used to create Influx databases. The documentation shows there to be 3 properties. The code shows that there are a further 2 (lc_collate and lc_ctype) that are optional.

The documentation states that:

None of the database properties can currently be changed after creation. Doing so will result in the old database getting dropped and a new database created.

Running, then re-running the following, db1 and db2 will be destroyed. db3 will not be destroyed when re-run.

resource "aiven_database" "db1" {
    project       = "<my-project>"
    service_name  = "<influx-service-name>"
    database_name = "db1"
}

resource "aiven_database" "db2" {
    project       = "<my-project>"
    service_name  = "<influx-service-name>"
    database_name = "db2"
    lc_collate    = "some-value"
    lc_ctype      = "some-other-value"
}

resource "aiven_database" "db3" {
    project       = "<my-project>"
    service_name  = "<influx-service-name>"
    database_name = "db3"
    lc_collate    = ""
    lc_ctype      = ""
}

Based on the documentation (and expected behaviour of default values), none of these should be destroyed when re-run.

Hello,

I wanted to create a new aiven Project today and I could no add dashes (-) to it.

On the website I am able to do this without issues.

There is currently no way to create a Timescale type service. I can create a service with type pg but to create a timescale service the provider needs to support variant in the pg_user_config.

Hello team,
I don't find any way to set up a maintenance updates in terraform configuration ?
Is it not yet implemented ? Or could you tell how to do it ?
Thanks,
Alex

Currently, the definition of the service_uri attribute on a aiven_service resource doesn't state it as sensitive. This while the service_uri usually contains the user+password combination of the default/admin user to connect to the service.

I think it's best to always mark the service_uri as sensitive because of this, just like service_password. This prevents passwords popping up in Terraform state output in terminals.

Hello team,

Sometimes I got some errors when i create:
1 aiven_project_vpc
1 aiven_vpc_peering_connection
1 aiven_service

I think it tries to create everything in the same time and the VPC interconnexion is not yet created and the apply failed.

I got this error:
{"errors":[{"message":"Project VPC xxxxx is in state APPROVED, not ACTIVE","status":409}],"message":"Project VPC xxx is in state APPROVED, not ACTIVE"}

I retry 30s after and it works ? Maybe you should add some dependancy between trying creating a service if the aiven_project_vpc/aiven_vpc_peering_connection is not yet finished.

Thanks,
Alex

Hello,

We encountered the following error after calling the terraform validate command. We are using the master branch, we tried also the latest released version(1.0.16), terraform version(0.12.8), go version(1.12.8):

Failed to instantiate provider "aiven" to obtain schema: Unrecognized remote plugin message: [ENVY] unable to load env file(s) on init: open .env: no such file or directory

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

We are doing the steps from here: https://help.aiven.io/en/articles/2370350-aiven-terraform-integration.

Also, I would like to mention that during debug, I discovered that the git.apache.org is down(hashicorp/terraform#22664) and tried also with you branch: mte-replace-git-apache-org, but the same error still appears when I run only the plugin binary:

/go/src/github.com/aiven/terraform-provider-aiven # ./terraform-provider-aiven
[ENVY] unable to load env file(s) on init: open .env: no such file or directory
This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

Thank you!

Currently the documentation of this provider and its resources is not embedded in the main Terraform documentation.

As this is the official Aiven provider for Terraform, it would be good - including for publicity - if it would be available in there.

After importing an already existing PG service database, next apply forces a new resource.

resource "aiven_database" "pg-db" {
  project      = "my-project"
  service_name = "my-pg"

  database_name = "my-database"
}

> terraform import aiven_database.pg-db my-project/my-pg/my-database
aiven_database.pg-db: Importing from ID "my-project/my-pg/my-database"...
aiven_database.pg-db: Import complete!
  Imported aiven_database (ID: my-project/my-pg/my-database)
aiven_database.pg-db: Refreshing state... (ID: my-project/my-pg/my-database)

> terraform apply
[...]
-/+ aiven_database.pg-db (new resource required)
      id:                          "my-project/my-pg/my-database" => <computed> (forces new resource)
      database_name:               "my-database" => "my-database"
      lc_collate:                  "en_US.UTF-8" => "en_US.UTF-8"
      lc_ctype:                    "en_US.UTF-8" => "en_US.UTF-8"
      project:                     "" => "my-project" (forces new resource)
      service_name:                "" => "my-pg" (forces new resource)

When creating a postgres service, I would like to obtain the port number of the created database and pass it to other resources that require it.

Here is my configuration:

resource aiven_project "xyz" {
  project = "${var.aiven_project_name}"
  card_id = "${var.aiven_card_id}"
}

resource aiven_service "xyz" {
  project                 = "${aiven_project.xyz.project}"
  cloud_name              = "aws-${var.region}"
  plan                    = "${var.aiven_service_plan}"
  service_name            = "${var.aiven_service_name}"
  service_type            = "pg"
  project_vpc_id          = "${aiven_project_vpc.xyz.id}"
  maintenance_window_dow  = "${var.maintenance_window_dow}"
  maintenance_window_time = "${var.maintenance_window_time}"
  termination_protection  = "${var.aiven_service_termination_protection}"

  pg_user_config {
    pg_version = "${var.postgres_version}"
  }
}

resource aiven_database "xyz" {
  project       = "${aiven_project.xyz.project}"
  service_name  = "${aiven_service.xyz.service_name}"
  database_name = "${var.db_name}"
}

I can then access the port number using "${aiven_service.xyz.pg.0.port}".

On the initial terraform apply, the port is always "0".

Running terraform plan and terraform apply again causes the port to be updated to the correct value.

Need to add force new on project and service_name else it will trigger a 404 not found when trying to update the service.

When attempting to use the provider version 1.0.14, terraform crashes. This is using terraform 0.12.6 and the darwin binary from the releases page.

main.tf:

provider "aiven" {}

plan:

$ terraform plan
Error: Failed to instantiate provider "aiven" to obtain schema: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.


panic: Failed to read service_user_config_schema.json: stat /srv/jenkins/workspace/terraform-provider-aiven/gopath/src/github.com/aiven/terraform-provider-aiven/aiven/templates/service_user_config_schema.json: no such file or directory
2019-08-17T15:02:05.403+0100 [DEBUG] plugin.terraform-provider-aiven_v1.0.14:
2019-08-17T15:02:05.403+0100 [DEBUG] plugin.terraform-provider-aiven_v1.0.14: goroutine 1 [running]:
2019-08-17T15:02:05.403+0100 [DEBUG] plugin.terraform-provider-aiven_v1.0.14: github.com/aiven/terraform-provider-aiven/aiven.readUserConfigJSONSchema(0x1c52b64, 0x1f, 0x1c3a9a0)
2019-08-17T15:02:05.403+0100 [DEBUG] plugin.terraform-provider-aiven_v1.0.14: 	/srv/jenkins/workspace/terraform-provider-aiven/gopath/src/github.com/aiven/terraform-provider-aiven/aiven/user_config.go:17 +0x376
2019-08-17T15:02:05.403+0100 [DEBUG] plugin.terraform-provider-aiven_v1.0.14: github.com/aiven/terraform-provider-aiven/aiven.GetUserConfigSchema(0x1c3a9a0, 0x7, 0x1c42256)
2019-08-17T15:02:05.403+0100 [DEBUG] plugin.terraform-provider-aiven_v1.0.14: 	/srv/jenkins/workspace/terraform-provider-aiven/gopath/src/github.com/aiven/terraform-provider-aiven/aiven/user_config.go:41 +0x1a0
2019-08-17T15:02:05.403+0100 [DEBUG] plugin.terraform-provider-aiven_v1.0.14: github.com/aiven/terraform-provider-aiven/aiven.init.ializers()
2019-08-17T15:02:05.403+0100 [DEBUG] plugin.terraform-provider-aiven_v1.0.14: 	/srv/jenkins/workspace/terraform-provider-aiven/gopath/src/github.com/aiven/terraform-provider-aiven/aiven/resource_service.go:139 +0x1f14
2019-08-17T15:02:05.403+0100 [DEBUG] plugin: plugin process exited: path=/Users/danielflook/.terraform.d/plugins/terraform-provider-aiven_v1.0.14 pid=78700 error="exit status 2"
2019/08/17 15:02:05 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2019/08/17 15:02:05 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
2019-08-17T15:02:05.404+0100 [WARN]  plugin: plugin failed to exit gracefully



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Using binary release https://github.com/aiven/terraform-provider-aiven/releases/download/v1.0.1/terraform-provider-aiven-linux_amd64

I get this error:

panic: Failed to read integration_endpoints_user_config_schema.json: stat /home/rikonen/go/src/github.com/aiven/terraform-provider-aiven/templates/integration_endpoints_user_config_schema.json: no such file or directory
2018-10-08T12:03:10.265Z [DEBUG] plugin.terraform-provider-aiven: 
2018-10-08T12:03:10.265Z [DEBUG] plugin.terraform-provider-aiven: goroutine 1 [running]:
2018-10-08T12:03:10.265Z [DEBUG] plugin.terraform-provider-aiven: main.readUserConfigJSONSchema(0xab5825, 0x2d, 0xc420075f78)
2018-10-08T12:03:10.265Z [DEBUG] plugin.terraform-provider-aiven: 	/home/rikonen/go/src/github.com/aiven/terraform-provider-aiven/user_config.go:17 +0x3db
2018-10-08T12:03:10.266Z [DEBUG] plugin: plugin process exited: path=/root/.terraform.d/plugins/terraform-provider-aiven
2018/10/08 12:03:10 [ERROR] root: eval: *terraform.EvalInitProvider, err: plugin exited before we could connect
2018/10/08 12:03:10 [ERROR] root: eval: *terraform.EvalSequence, err: plugin exited before we could connect

The binary tries to read config files from directory /home/rikonen/go/src/github.com/aiven/terraform-provider-aiven/

Hi I cant seem to set:

kafka_authentication_methods.sasl = true

using this terraform provider for aiven_service kafka.

Should be something like:

resource "aiven_service" "kafka" {
  service_type           = "kafka"
....

  kafka_user_config {
    kafka_version   = "2.1"

    kafka {
      auto_create_topics_enable    = true
      compression_type             = "producer"
    }

    kafka_authentication_methods {
      sasl = true
    }
  }
}

Hi guys, I was wondering if it is currently possible to automatically setup VPC peering.

I was planning on using a vpc_peering_connection_accepter to automatically accept the VPC connection request on our end. It requires the AWS vpc_peering_connection_id (e.g. "pcx-0123456").

Would be possible to expose the vpc_peering_connection_id from the aiven_vpc_peering_connection resource? That way we could use a vpc_peering_connection_accepter and also use the id for adding the necessary route in our route table(s).

If there is a better, recommended approach for handling this then please let me know. Thanks!

We started running into this issue this morning on version 1.0.12

* aiven_kafka_topic.topic: error waiting for Aiven Kafka topic to be ACTIVE: 501: {"errors":[{"message":"Failed to fetch Kafka topic: events_topic","status":501}],"message":"Failed to fetch Kafka topic: events_topic"} -

The topic is being created in Aiven in the active state, but this request is failing for some reason. Is there something we can configure on our side to increase this timeout?

I have a question about importing existing Kafka service into terraform state. I was able to successfully import such resource, but when I am running terraform plan afterwards, I can see that kafka_user_config section (marked with + sign below) wasn't imported. Is there something we can do about it?

I am using latest (v1.0.11) version of aiven provider.

~ resource "aiven_service" "kafka" {
        cloud_name              = "aws-eu-central-1"
        ...
        maintenance_window_dow  = "monday"
        maintenance_window_time = "02:00:00"
        plan                    = "business-4"
        ...
        service_port            = 14738
        service_type            = "kafka"
        service_uri             = (sensitive value)
        state                   = "RUNNING"
        termination_protection  = true

        kafka {
            access_cert         = (sensitive value)
            access_key          = (sensitive value)
            connect_uri         = (sensitive value)
            rest_uri            = (sensitive value)
            schema_registry_uri = (sensitive value)
        }

      + kafka_user_config {
          + custom_domain   = "<<value not set>>"
          + kafka_connect   = false
          + kafka_rest      = true
          + kafka_version   = "2.1"
          + schema_registry = true

          + kafka {
              + auto_create_topics_enable                  = false
              + compression_type                           = "<<value not set>>"
              + default_replication_factor                 = -1
              + group_max_session_timeout_ms               = 300000
              + group_min_session_timeout_ms               = 6000
              + log_cleaner_min_compaction_lag_ms          = -1
              + log_message_timestamp_difference_max_ms    = -1
              + log_message_timestamp_type                 = "<<value not set>>"
              + log_retention_bytes                        = -1
              + log_retention_hours                        = 720
              + log_segment_bytes                          = -1
              + max_connections_per_ip                     = -1
              + message_max_bytes                          = 1000012
              + num_partitions                             = -1
              + offsets_retention_minutes                  = 1440
              + producer_purgatory_purge_interval_requests = -1
              + replica_fetch_max_bytes                    = -1
              + replica_fetch_response_max_bytes           = -1
              + socket_request_max_bytes                   = -1
            }

          + kafka_rest_config {
              + consumer_enable_auto_commit  = true
              + consumer_request_max_bytes   = 67108864
              + consumer_request_timeout_ms  = 1000
              + producer_acks                = "1"
              + producer_linger_ms           = 0
              + simpleconsumer_pool_size_max = 25
            }
        }
    }

consumer_max_poll_interval_mx is very useful and it will be great if this is configurable via terraform. It's not showing up in this list now.

From the doc: https://kafka.apache.org/documentation/

max_poll_interval_mx: The maximum delay between invocations of poll() 
when using consumer group management. This places an upper bound
on the amount of time that the consumer can be idle before fetching 
more records. If poll() is not called before expiration of this timeout, 
then the consumer is considered failed and the group will rebalance in 
order to reassign the partitions to another member.

At the moment, terraform data sources are not supported.

Adding it would be interesting for us in the following cases:

One TF script for infra and one per streaming app

I have the following setup in mind:

[1] one Terrraform script for deploying the Kafka service, the VPC, the peering with our account and the rest of the infrastructure

[2] one Terraform script for each set of stream applications, responsible for deploying the apps as well as for creating the necessary kafka topics

In order for [2] to work it needs to refer to the following data source created by [1], s.t. the aiven_kafka_topic resource can work:

data "aiven_project" "my-project" {...}
data "aiven_service" "my-kafka-service" {...}

This is currently not possible, although a workaround is simply to depend on the project names and service as string (or variables) in [2], which are immutable anyway (thanks @rikonen for the pointer).

Scaling a consumer app up to the partitioning level of a topic

I have the following setup in mind:

[3] one Terrraform script for deploying a streaming app together with the topic it writes to

[4] one Terraform script for another streaming app reading the output topic of [3], deployed as a K8s deployment.

One desirable feature would be to set the maximum number of replicas of [4] equal to the number of partitions of the output topic of [3].

Here again, duplicating that number in both scripts provides a workaround, although availability of TF aiven_kafka_topic data source would reduce coupling.

When creating an aiven_service_integration_endpoint we can't get client port directly.

Using this definition:

resource "aiven_service_integration_endpoint" "prometheus" {
  project = "myproject"
  endpoint_name = "prometheus"
  endpoint_type = "prometheus"

  prometheus_user_config {
    basic_auth_username = "promuser"
    basic_auth_password = "prompassword"
  }
}

output "aiven-prometheus-endpoint-port" {
  value = "${aiven_service_integration_endpoint.prometheus.endpoint_config.client_port}"
}

Wtih either terraform plan or terraform apply, I get:

* output.aiven-prometheus-endpoint-port: Resource 'aiven_service_integration_endpoint.prometheus' does not have attribute 'endpoint_config.client_port' for variable 'aiven_service_integration_endpoint.qs-prometheus.endpoint_config.client_port'

I must first comment output aiven-prometheus-endpoint-port, run terraform apply and then
un-comment ouput.

Furthermore, it is possible to "register" Aiven provider computed attributes ?
In order to have auto-completion and validation working.

➜ terraform version
Terraform v0.11.11
+ provider.aiven v1.0.9

Hello,

I've created a fork of the project to add support for Kafka topics there, but I am not able to compile it. What I did is:

1. Cloned the repository locally
2. Ran glide install
3. Ran make

The end result is:

robertof@robfas-lin:~/go/src/terraform-provider-aiven (master)$ make
go build -o terraform-provider-aiven .
# terraform-provider-aiven
./provider.go:39:11: undefined: aiven.NewMFAUserClient
./resource_service_user.go:79:27: user.AccessCert undefined (type *aiven.ServiceUser has no field or method AccessCert)
./resource_service_user.go:80:26: user.AccessKey undefined (type *aiven.ServiceUser has no field or method AccessKey)
./resource_service_user.go:102:29: user.AccessCert undefined (type *aiven.ServiceUser has no field or method AccessCert)
./resource_service_user.go:103:28: user.AccessKey undefined (type *aiven.ServiceUser has no field or method AccessKey)
Makefile:2: recipe for target 'plugin' failed
make: *** [plugin] Error 2

I believe the issue is that https://github.com/jelmersnoeck/terraform-provider-aiven/pull/5 and https://github.com/jelmersnoeck/terraform-provider-aiven/pull/9 require changes to the aiven client but those haven't been merged back to the repo referenced in the glide file. So @jelmersnoeck @FranOis can that be done?

aiven_resource_service should wait for service to report being RUNNING before declaring a successful creation or update.

Hello,
Is it possible to add support for Kafka Connect? or at least an output from the "aiven_service" resource of the Kafka Connect URL?

Thank you,
Amit.

I compiled the provider in version 1.0.17,
with go v1.12.7 and v1.12.6
with terraform v0.11.14

The compilation is successful, however, when using the provider in another environment, terraform crashes the same way as in this issue : #92

Error: Error asking for user input: 1 error occurred:
	* module.aiven_services.provider.aiven: plugin exited before we could connect




panic: Failed to read service_user_config_schema.json: stat /home/vagrant/test_aiven/templates/service_user_config_schema.json: no such file or directory
2019-11-27T15:13:54.216Z [DEBUG] plugin.terraform-provider-aiven_v1.0.17: 
2019-11-27T15:13:54.216Z [DEBUG] plugin.terraform-provider-aiven_v1.0.17: goroutine 1 [running]:
2019-11-27T15:13:54.216Z [DEBUG] plugin.terraform-provider-aiven_v1.0.17: github.com/aiven/terraform-provider-aiven/aiven.readUserConfigJSONSchema(0x11472d3, 0x1f, 0x112cb2f)
2019-11-27T15:13:54.216Z [DEBUG] plugin.terraform-provider-aiven_v1.0.17: 	/home/jenkins/agent/workspace/terraform_CICD_master/src/github.com/aiven/terraform-provider-aiven/aiven/user_config.go:17 +0x317
2019-11-27T15:13:54.216Z [DEBUG] plugin.terraform-provider-aiven_v1.0.17: github.com/aiven/terraform-provider-aiven/aiven.GetUserConfigSchema(0x112cb2f, 0x7, 0x1134e0c)
2019-11-27T15:13:54.216Z [DEBUG] plugin.terraform-provider-aiven_v1.0.17: 	/home/jenkins/agent/workspace/terraform_CICD_master/src/github.com/aiven/terraform-provider-aiven/aiven/user_config.go:41 +0x1a0
2019-11-27T15:13:54.216Z [DEBUG] plugin.terraform-provider-aiven_v1.0.17: github.com/aiven/terraform-provider-aiven/aiven.init.ializers()
2019-11-27T15:13:54.216Z [DEBUG] plugin.terraform-provider-aiven_v1.0.17: 	/home/jenkins/agent/workspace/terraform_CICD_master/src/github.com/aiven/terraform-provider-aiven/aiven/resource_service.go:139 +0x1f14

While trying to execute the provider in another environment that the one where it was compiled, it fails with following message:

./terraform-provider-aiven_v1.0.17 
panic: Failed to read service_user_config_schema.json: stat /home/vagrant/.terraform.d/plugins/temp/templates/service_user_config_schema.json: no such file or directory

goroutine 1 [running]:
github.com/aiven/terraform-provider-aiven/aiven.readUserConfigJSONSchema(0xf771d6, 0x1f, 0xf5f2af)
	/home/jenkins/agent/workspace/terraform_CICD_master/src/github.com/aiven/terraform-provider-aiven/aiven/user_config.go:17 +0x317
github.com/aiven/terraform-provider-aiven/aiven.GetUserConfigSchema(0xf5f2af, 0x7, 0xf67124)
	/home/jenkins/agent/workspace/terraform_CICD_master/src/github.com/aiven/terraform-provider-aiven/aiven/user_config.go:41 +0x1a0
github.com/aiven/terraform-provider-aiven/aiven.init.ializers()
	/home/jenkins/agent/workspace/terraform_CICD_master/src/github.com/aiven/terraform-provider-aiven/aiven/resource_service.go:139 +0x1f14

What

We tainted a service so that it'd get recreated, but noticed that the state for the users and ACLs under that service weren't invalidated resulting in them not getting recreated.

I ended up working around this by tainting a single aiven_service_user and terraform picked up the missing users/ACLs.

Terraform Versions

$ terraform version
Terraform v0.11.13
+ provider.aiven v1.0.8

When you have Kafka Connect, Schema Registry or Kafka REST enabled and set an IP filter for a service or move the service to a VPC, making changes to the service stops working properly unless Terraform provider is being called from a host that is within the VPC / IP filter range. This is because the update call has extra check that tries to connect to those auxiliary services to verify they're running before marking the operation complete. That logic must be changed so that it does not rely on actual service calls since those cannot be made reliably.

First thing - thanks a lot for this provider, it's really useful!

There are a couple of merged pull requests on master I'd love to use

#80
#79

I can build the plugin from master but I'd prefer to use an official release.

Would it possible to use milestones or anything similar to show when a new release will be created ?

This would help me deciding whether to wait for the release or to just build it from master.

Thanks again for your work!

I've installed this provider and can get init/plan to work fine. However, I am trying to use this module to create Timescale Cloud services. Here is the aiven_service I am using:

resource "aiven_service" "tsdb" {
	project = "${var.aiven_project_name}"
	cloud_name = "aws-us-east-2"
	plan = "timescale-basic-512-io-optimized"
	service_name = "timescalebd"
	service_type = "pg"
	maintenance_window_dow = "sunday"
	maintenance_window_time = "12:00:00"
	pg_user_config {
		pg {
			idle_in_transaction_session_timeout = 900
		}
		pg_version = "11"
	}
}

When I run terraform plan, I get the following output:

1 error occurred:
	* aiven_service.tsdb: 1 error occurred:
	* aiven_service.tsdb: 400: {"errors":[{"message":"Invalid plan 'timescale-basic-512-io-optimized' for service type 'pg'","status":400}],"message":"Invalid plan 'timescale-basic-512-io-optimized' for service type 'pg'"} - 

However, using the Aiven REST API to create a new service, providing "service_type" of "pg" and "plan" of "timescale-basic-512-io-optimized" creates the service as expected.

Any ideas why this is not working for the TF provider?

What

We had an existing Kafka Service and we made a change to our terraform configs to create a VPC network and add the service to the VPC network. It worked fine in one environment, but we hit an error in another environment.

* aiven_service.ev_kafka: 409: {"errors":[{"message":"Project VPC <VPC_ID> is in state APPROVED, not ACTIVE","status":409}],"message":"Project VPC <VPC_ID> is in state APPROVED, not ACTIVE"} -

When I tried to apply the config again, terraform came back that there were no changes to make.

If I inspect the state with terraform state show for the aiven_service, I see that the project_vpc_id is entered.

Terraform Versions

$ terraform version
Terraform v0.11.13
+ provider.aiven v1.0.8

This is more up for a general design discussion. While I like the flexibility of having a very generic service resource that allows this provider to support any new service that aiven could add to their offering without having to write more code. The downside to this is that you have to manage all service type in the same file or do some kind of logic switch between type plus managing a giant map of possible configuration that depends on the service type. This causes issues with clarity of use and makes it hard to understand exactly what's possible to do and what is not possible to do on a service basis.

My proposal is as follows.

1. Keep the very generic service resource for backward compatibility and for future work by aiven and to support any service that are not yet implemented in this provider.
2. Implement a service type specific resource for each offering of aiven.

For example:
You would still have the service resource as it exists now, but we would also have one for Kafke, Postgres, ElasticSearch etc.

Hi!
Does the provider support Terraform 0.12?

➜ terraform 0.12checklist
After analyzing this configuration and working directory, we have identified some necessary steps that we recommend you take before upgrading to Terraform v0.12:

- [ ] Provider "aiven" may need to be upgraded to a newer version that supports Terraform 0.12. (Supported version information is not available for this provider.)

Hi all,

I experienced an error on my laptop as i was running my terraform plan with ressource aiven-service-integration-endpoint :

I tried with a fresh install (on a vagrant VM) and had the same issue.
Steps to reproduce :

• Install Terraform (Terraform v0.12.3)

• (Optionnal : login to GCP as my backend is hosted on a bucket)

• Download aiven plugin in ~/.terraform.d/plugins/linux_amd64

    vagrant@ubuntu-bionic:~/.terraform.d/plugins/linux_amd64$ curl -L -o terraform-provider-aiven https://github.com/aiven/terraform-provider-aiven/releases/download/v1.0.10/terraform-provider-aiven-linux_amd64
  
    vagrant@ubuntu-bionic:~/.terraform.d/plugins/linux_amd64$ chmod +x terraform-provider-aiven
  

Terraform init -> Turns OK

Terraform Plan -> Error.

    Error: Unsupported argument

    on project.tf line 99, in resource "aiven_service_integration_endpoint" "prometheus_endpoint":
    99:   prometheus_user_config = {

    An argument named "prometheus_user_config" is not expected here. Did you mean
    to define a block of type "prometheus_user_config"?

Extract of my terraform file :

    resource "aiven_service_integration_endpoint" "prometheus_endpoint" {
    project       = "${var.aiven_project_name}"
    endpoint_name = "prometheus"
    endpoint_type = "prometheus"

    prometheus_user_config = {
        basic_auth_username = "${data.vault_generic_secret.prometheus.data["username"]}"
        basic_auth_password = "${data.vault_generic_secret.prometheus.data["password"]}"
    }
    }

I had to terraform an Aiven Instance, so i asked my colleage to run this code on his laptop, with an older version of the plugin / terraform and it runned just fine.

Any idea of what happened?

Regards,

Terraform provider should allow us to use a MFA secured account to interact with aiven.

Check that it builds and works OK on Windows

I am using 1.0.11 version of aiven provider and when I am running plan for Kafka service, sensitive data are revealed, as show on example below.

resource "aiven_service" "kafka" {
        cloud_name              = "aws-eu-central-1"
        id                      = "project-name/service-name"
        maintenance_window_dow  = "monday"
        maintenance_window_time = "02:00:00"
        plan                    = "startup-2"
        ...
        service_type            = "kafka"
        service_uri             = (sensitive value)
        state                   = "RUNNING"
        termination_protection  = true

        kafka {
            access_cert         = (sensitive value)
            access_key          = (sensitive value)
            connect_uri         = "https://avnadmin:[email protected]:14000"
            rest_uri            = "https://avnadmin:[email protected]:14001"
            schema_registry_uri = "https://avnadmin:[email protected]:14002"
        }
       ...
}

Expected behaviour would be:

resource "aiven_service" "kafka" {
        cloud_name              = "aws-eu-central-1"
        id                      = "project-name/service-name"
        maintenance_window_dow  = "monday"
        maintenance_window_time = "02:00:00"
        plan                    = "startup-2"
        ...
        service_type            = "kafka"
        service_uri             = (sensitive value)
        state                   = "RUNNING"
        termination_protection  = true

        kafka {
            access_cert         = (sensitive value)
            access_key          = (sensitive value)
            connect_uri         = (sensitive value)
            rest_uri            = (sensitive value)
            schema_registry_uri = (sensitive value)
        }
       ...
}

The official docker images from terraform are built upon Alpine which does not support binaries compiled with CGO support without explicitly installing libc6-compat.

This issue also occurred with the terraform-provider-helm, see hashicorp/terraform-provider-helm#59.

This can be fixed by simply adding the CGO_ENABLED=0 for Linux builds.

Steps to reproduce:

1. docker run -it --entrypoint /bin/sh hashicorp/terraform:0.11.13
2. wget -O terraform-provider-aiven_v1.0.8_x4 https://github.com/aiven/terraform-provider-aiven/releases/download/v1.0.8/terraform-provider-aiven-linux_amd64
3. chmod +x terraform-provider-aiven_v1.0.8_x4
4. ./terraform-provider-aiven_v1.0.8_x4

The result would be terraform-provider-aiven_v1.0.8_x4: No such file or directory

A workaround would be executing apk add libc6-compat before trying to executing terraform, but this is a very undesirable workaround.

I'm willing to make a pull request.

Hey,

I just started using the terraform aiven provider. I deployed a new project with it now in the next terraform plan run it wants to remove the billing details.

variable "aiven_api_token" {}
variable "aiven_card_id" {}

# Initialize provider. No other config options than api_token
provider "aiven" {
  api_token = "${var.aiven_api_token}"
}

resource "aiven_project" "project" {
  project           = "${terraform.workspace}"
  copy_from_project = "karl-3727"
}

Terraform Plan:

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aiven_project.project
      billing_address:             "**********************************" => ""
      billing_emails.#:            "1" => "0"
      billing_emails.1238831172:   "**********************************" => ""
      card_id:                     "**********************************" => ""
      country_code:                "DE" => ""
      technical_emails.#:          "1" => "0"
      technical_emails.1238831172: "**********************************" => ""


Plan: 0 to add, 1 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: planfile

To perform exactly these actions, run the following command to apply:
    terraform apply "planfile"

Hello Team,
Sometimes when I try to create service, I got sometime out.
Ref : Error waiting for Aiven service to be RUNNING: 408:

408 Request Time-out

Is there any way to increase the timeout ?
The issue is after I have some difference between terraform and Aiven (DB is created but Terraform think it is not !)
Alex