Question about updating existing cluster vs creating new one in new vcn about terraform-kubernetes-installer HOT 8 OPEN

In general I've been seeing so many issues that it's hard for me to pin down. I assume that I should be able to update the number of workers in ad's all day long (add or reduce the number) and run terraform and everything should work but that's not the case. For example, after 4 successful runs in a row where I changed the number of worker nodes in each AD between runs, I just went from 1 worker in each of three ad's to 0 in one of them and I got this error.

My intent is to have 200 different tfvars files w different settings and copy each in turn, run terraform and not see a single error. Is this reasonable? I won't feel comfortable using oracle cloud/terraform in production until I can run terraform over and over w. different configs w/o a single error.

from terraform-kubernetes-installer.

Hi @jferr,

Firstly, sorry if you've had any slow responses.

Terraform stores all state about what it's created in a terraform.tfstate file. Generally this needs to be persisted and used for each run with the same resources so that Terraform can keep track of what it has created (https://www.terraform.io/docs/state/). We notice that this isn't being persisted between runs which at a first guess is likely the cause of many of these issues.

Manually deleting and editing things outside of Terraform will likely causes problems. The un-deletable subnet issue happens when a subnet is referenced by another resource and so it is technically unsafe to delete it. This is most likely caused by the manual deleting and editing of things outside of Terraform's control. In short, if the state file and the state of the world is inconsistent for consecutive runs then it's likely Terraform will get confused. This is generally more a property of Terraform itself than the OCI specific implementation.

The Terraform refresh command is used to reconcile the state Terraform knows about (via its state file) so the state file is needed for Terraform to do the right thing.

Let us know if that helps and please ask if you have any further questions on this.

from terraform-kubernetes-installer.

Thanks @owainlewis

My thought was that refreshing w. each run is what will allow us to run this via a docker container and should also allow terraform to see an accurate view of infrastructure as-is (e.g. even if something is modified via the OCI console terraform will accurately see "as is" and will reconcile). Is this not true? In our case we are a small group and we can manage a single terraform run at a time by only running terraform via a Jenkins job.

from terraform-kubernetes-installer.

@owainlewis shouldn't this work? I started w. using an s3 backend for terraform but had a number of issues plus we've had issues where at some point terraform starts failing and we need to delete everything from the oracle console (compute/lb/vcn) and start over. I figure refreshing each time should be the most stable though less performant. Stability is the important thing here for us.

from terraform-kubernetes-installer.

Hi @jferr

I think for stability you'll want to

1. Ensure that resources created by Terraform are (as much as possible) only managed by terraform (i.e manually deleting things might cause problems)
2. Persist the terraform.tfstate throughout the lifecycle of the cluster you are managing. This could be by mounting the statefile on the host somewhere if running this in Docker.

from terraform-kubernetes-installer.

Thanks @owainlewis I will try this out. I started that way...with a persistant tfstate file...I tried both locally and amazon s3 backed...but I still had lots of issues and failures which required me to manually delete resources via the console.

It seems to me that the most stable should be having terraform read the state w. every run w/o persisting the tfstate between runs...though for some organizations this might not be practical. Can you explain why this wouldn't be the most stable way to go.

from terraform-kubernetes-installer.

@owainlewis in my testing I often saw that when terraform was refreshing right after I deleted resources via the console (because of the inability to get a successful followup terraform run) I would see references to entities (e.g. subnets) that had already been deleted. Is that the reason why? Is there some bug on Oracle's side where perhaps console changes aren't reflected immediately in the API?

We are very uncomfortable w. Oracle Kubernetes Terraform stability at the moment (we are not live yet) and are escalating this via other channels so any info that you can give us is appreciated.

from terraform-kubernetes-installer.

Hi @jferr

To clarify the "un-deletable" resources issue when you destroy a cluster, this is because the resource cannot be safely deleted (i.e someone/something else is using or referencing the resource). This is a deliberate feature.

I would see references to entities (e.g. subnets) that had already been deleted

The docs are helpful here when discussing why we need to persist the terraform.tfstate file.

It is often asked if it is possible for Terraform to work without state, or for Terraform to not use state and just inspect cloud resources on every run. This page will help explain why Terraform state is required.

https://www.terraform.io/docs/state/purpose.html

from terraform-kubernetes-installer.