When replacing a volume attached to a machine, the delete fails with an error like "cannot delete volume, still in use". The volume property on the machine should probably be marked as "forces replacement".

The tunnel code can be found in internal/provider/wg/tunnel.go
A example of the tunnel and it's problems can be found in tools/tools.go. To run example: go run tools/tools.go.

I think the tunnel actually does get open but I think it breaks in DNS resolution? But to be honest I'm not sure.

Hey @DAlperin! Thanks a ton for creating and maintaining this provider. I'm very excited to increase usage of it. I've been having some non-starters, but after upgrading to 0.15 I got my first successful app launches.

Now, when trying to add services to these existing apps I'm running into some consistency issues. I've tried to reduce the problem as much as possible and include all available debug info. Let me know if there's anything else that would be helpful!

It's worth noting that entirely recreating the app does not cause this same issue.

terraform apply
fly_app.nginx: Refreshing state... [id=jg-nginx]
fly_machine.nginx: Refreshing state... [id=e784343a275778]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # fly_machine.nginx will be updated in-place
  ~ resource "fly_machine" "nginx" {
      ~ env      = {} -> (known after apply)
      ~ id       = "e784343a275778" -> (known after apply)
        name     = "jg-nginx"
      + services = [
          + {
              + internal_port = 80
              + ports         = [
                  + {
                      + handlers = [
                          + "http",
                        ]
                      + port     = 80
                    },
                ]
              + protocol      = "tcp"
            },
        ]
        # (6 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

fly_machine.nginx: Modifying... [id=e784343a275778]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to fly_machine.nginx, provider "provider[\"registry.terraform.io/fly-apps/fly\"]" produced an unexpected new value: .services: was
│ cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"internal_port":cty.NumberIntVal(80),
│ "ports":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"handlers":cty.ListVal([]cty.Value{cty.StringVal("http")}), "port":cty.NumberIntVal(80)})}), "protocol":cty.StringVal("tcp")})}), but
│ now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

DEBUG=1 TF_LOG=debug terraform apply

terraform {
  required_providers {
    fly = {
      source  = "fly-apps/fly"
      version = "0.0.15"
    }
  }
}

provider "fly" {}

resource "fly_app" "nginx" {
  name = "jg-nginx"
  org  = "personal"
}

resource "fly_machine" "nginx" {
  name = "jg-nginx"

  app   = fly_app.nginx.name
  image = "nginx"

  region   = "sea"
  cpus     = 1
  memorymb = 1024
  cputype  = "shared"

  # Newly Added 
  services = [
    {
      internal_port = 80
      ports = [
        {
          handlers = [
            "http",
          ]
          port = 80
        },
      ]
      protocol = "tcp"
    }
  ]
}

When creating a fly_app resource, if I specify an org other than "personal", I get an error when I run terraform apply:

│ Error: Could not resolve organization
│
│   with fly_app.main_app,
│   on main.tf line 51, in resource "fly_app" "main_app":
│   51: resource "fly_app" "main_app" {
│
│ input:3: organization Could not find

The terraform resource block is very simple:

resource "fly_app" "main_app" {
  name = local.fly_app_name
  org  = var.fly_organization
}

Though I am using a local variable and a variable for these values respectively, I have ruled out an issues with variable references by just changing the value of var.fly_organization. When the value is "personal", everything works fine. When it is set to the name of an organization that I have created in the Fly.io dashboard, it fails.

I'm using terraform taint to replace a fly machine that needs to use an updated image. It takes a bit for the fly api to respond that the old machine is destroyed and terraform throws 422 errors during that time.

It's not a showstopper, just a nice to have.

When I add cputype, cpus, or memorymb to a machine's configuration, it errors with this:

12:16:07.564
fly_machine.machine[0]: Creating...
12:16:12.173
fly_machine.machine[0]: Creation errored after 6s
12:16:12.230
Error: Provider produced inconsistent result after apply
12:16:12.286
When applying changes to fly_machine.machine[0], provider "provider[\"registry.terraform.io/fly-apps/fly\"]" produced an unexpected new value: .cpus: was cty.NumberIntVal(8), but now cty.NumberIntVal(1). This is a bug in the provider, which should be reported in the provider's own issue tracker.
12:16:12.348
Error: Provider produced inconsistent result after apply
12:16:12.405
When applying changes to fly_machine.machine[0], provider "provider[\"registry.terraform.io/fly-apps/fly\"]" produced an unexpected new value: .memorymb: was cty.NumberIntVal(8192), but now cty.NumberIntVal(256). This is a bug in the provider, which should be reported in the provider's own issue tracker.

Here's my Terraform configuration: https://github.com/kotx/coder/blob/master/fly/main.tf#L113-L139

resource "fly_machine" "machine" {
  count = data.coder_workspace.me.start_count

  app = var.fly_app

  cpus = 8
  cputype = "shared"
  memorymb = 8192

  region = var.fly_region
  image = var.docker_image
  name = "${data.coder_workspace.me.owner}_${data.coder_workspace.me.name}_${random_id.machine_id.hex}"
  env = {
    CODER_AGENT_TOKEN = coder_agent.main.token
  }

  cmd = [
    "sh", "-c", coder_agent.main.init_script]

  mounts = [
    {
      path = "/home"
      volume = fly_volume.home_volume.id
    }
  ]
  depends_on = [fly_volume.home_volume]
}

Is it possible at the moment (or will it be possible at some point) to set application-level secrets? It would be cool to have a terraform-based alternative to running

flyctl secrets set foo=bar -a XXXXXXX

https://github.com/hashicorp/terraform-plugin-docs

Changing a machine name should trigger a recreate.

You can't migrate regions using update, so the region attribute changing should taint the resource.

I changed some services on a machine and now I get this error.

Error: Provider produced inconsistent result after apply

When applying changes to fly_machine.zA87GG1KO, provider "provider[\"registry.terraform.io/fly-apps/fly\"]" produced an unexpected new value: .services: was null, but now
cty.ListValEmpty(cty.Object(map[string]cty.Type{"internal_port":cty.Number, "ports":cty.List(cty.Object(map[string]cty.Type{"handlers":cty.List(cty.String), "port":cty.Number})),
"protocol":cty.String})).

This is a bug in the provider, which should be reported in the provider's own issue tracker.

The README currently states that Fly apps will be deprecated soon. Is that hidden knowledge of the Fly platform or a choice to deprecate them from this provider?

I ask as Fly Machines appear to have some increased complexity for those of us that want simple application management (particularly around auto-migration on hardware failure):
https://fly.io/blog/fly-machines/#how-fly-machines-will-frustrate-you-the-emotional-cost-of-simplicity

Happy to ask on the Fly forum instead, but this is the only mention of app deprecation I've been able to find. If it is just in the provider, is there a reason for it?

Thanks a ton for your work on this. Just hoping to get some increased clarity :)

I got this inscrutable error when creating a machine:

Create request failed: 404 Not Found, &{ID: Name: State: Region: InstanceID: PrivateIP: Config:{Env:map[] Init:{Exec:[] Entrypoint:[] Cmd:[]} Image: Metadata:<nil> Restart:{Policy:} Services:[]
Mounts:[] Guest:{CPUKind: Cpus:0 MemoryMb:0}} ImageRef:{Registry: Repository: Tag: Digest: Labels:{}} CreatedAt:0001-01-01 00:00:00 +0000 UTC}

This was resolved by setting the app.

It would be great if

1. app is made to be required, to avoid making this mistake in the first place
2. The error were clarified to indicate exactly what issue occurred

Getting this error:

╷
│ Error: Failed to create machine
│ 
│   with fly_machine.server,
│   on main.tf line 197, in resource "fly_machine" "server":
│  197: resource "fly_machine" "server" {
│ 
│ Create request failed: 422 Unprocessable Entity, {"error":"Apps for machines must be created via the Machines API. See https://fly.io/docs/reference/machines/#create-a-fly-application."}
│ 
╵

I'm guessing this is related to this news: https://community.fly.io/t/fly-apps-on-machine-prerelease/10219/26

This began happening with the flyctl version update from 0.0.473 -> v0.0.474.

The App resource org argument only accepts org IDs. It should accept a slug, the ones visible in fly orgs list.

I see that the exec field for machines is commented out here:

This seems to assume an open WG tunnel during provisioning, I can't think of a way this could work in TF Cloud but might be missing something? Thanks!

I am using v0.0.9 of the provider and attempting to allocate a v4 IP address for my app. Here is an example of what the TForm looks like

resource "fly_app" "production" {
  name = "site-production"
}

resource "fly_ip" "exampleIp" {
  app  = "site-production"
  type = "v4"
}

When running terraform apply I was consistently hitting the following error:

input:3: mutation AllocateIpAddress.allocateIpAddress.input.type Nullability mismatch on variable $addrType and argument type (IPAddressType /
│ IPAddressType!)

I dug into the source and it seems like the GraphQL schema has changed since the last time the provider was released. I put together a PR that includes an update to the GraphQL client, the changed code does seem to fix the issue #34.

I am a newcomer to go and GraphQL in general. It looks like my PR includes some potentially unwanted changes. Can you help document the correct way to update the client?

Thanks!

It seems that services property cannot be set to null and going around with setting to empty list will fail when data are received back as the response does not contain the services property.

+ resource "fly_machine" "machine" {
      + ...
      + services = [
        ]
    }

Plan: 1 to add, 0 to change, 0 to destroy.

fly_machine.machine["fra"]: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to fly_machine.machine["fra"], provider
│ "provider[\"registry.terraform.io/fly-apps/fly\"]" produced an unexpected new value: .services: was
│ cty.ListValEmpty(cty.Object(map[string]cty.Type{"internal_port":cty.Number,
│ "ports":cty.List(cty.Object(map[string]cty.Type{"handlers":cty.List(cty.String), "port":cty.Number})),
│ "protocol":cty.String})), but now null.

Plan above actually creates new resource, but cannot store it into the tf state.

I can create machine without services trough API.

I think services property on fly_machine resource could be optional.

We launch some external machines that would like to participate in the Fly mesh, is it possible to manage WG peers via this Terraform provider? Not sure if this is something currently supported by the undoc'd internal API, but would be awesome.

Graphql accepts the Fly-Force-Trace header. The REST API may support this later.

Please see following:

Volume:

resource "fly_volume" "volume" {
  app        = var.config["proejct_name"]
  name       = replace(format("%s-volume", var.config["proejct_name"]), "-", "_")
  region     = var.config["region"]
  size       = tonumber(var.config["disk_size"])
  depends_on = [fly_app.app]
}

Machine:

resource "fly_machine" "machine" {
...
  mounts = [{
    encrypted = true
    path      = "/data"
    volume    = fly_volume.volume.name
  }]
...
}

Resulted in:

=> fly volumes list 
ID          STATE   NAME            SIZE    REGION  ZONE    ENCRYPTED       ATTACHED VM     CREATED AT    
<vol id>    created <volume name>   5GB     sjc     91af    false           <vm id>         9 minutes ago

It seems like the encrypted option shouldn't be in the mounts block inside fly_machine resource, it should be a part of fly_volume resource.

Hey @DAlperin, the services block doesn't support the concurrency attribute. Is it possible to extend the provider to be able to pass in concurrency as well?

https://fly.io/docs/reference/configuration/#services-concurrency

Hey all. Thought I'd point out that we needed to clarify what cputype actually means. I spent an hour trying "shared-1x" "shared-cpu-1x" and what not till I figured out that the options were "standard" and "performance"

Please clarify this in the docs. Thanks!

I'm running into an interesting error. I have some workspaces that haven't been exercised in awhile (0.0.10 ~ 0.0.19) and am running up against:

│ Error: Value Conversion Error
│
│ with module.base.data.fly_app.existing[0],
│ An unexpected error was encountered trying to convert tftypes.Value into
│ provider.appDataSourceOutput. This is always an error in the provider. Please
│ report the following to the provider developer:
│
│ mismatch between struct and object: Struct defines fields not found in
│ object: secrets.
╵

There does not appear to be anything additional in the debug logs that would be helpful, however if y'all need anything just ask. It occurs 100% of the time.

I'm running it on MacOS 12.6 against Terraform v1.2.4.

Hi there,

Great work on creating this terraform provider: it makes it easier to manage fly.io resources by code, which is great when working in teams.

A challenge however, is to keep up this terraform provider with the fast evolving fly.io resources and services. For example, shared IPs are currently not (yet) supported.
Therefore, I suggest to use the go implementation of flyctl for the fly.io API. This eases the adoption of new features and changes, and as Fly.io organisation you have only one implementation to maintain.

If this is something you agree on, I am happy to give it a shot rewriting the provider to the flyctl implementation.

I encountered this error after updating a machine image:

│ When applying changes to fly_machine.MYID, provider "provider[\"registry.terraform.io/fly-apps/fly\"]" produced an unexpected new value: .services: was null, but now
│ cty.ListValEmpty(cty.Object(map[string]cty.Type{"internal_port":cty.Number, "ports":cty.List(cty.Object(map[string]cty.Type{"handlers":cty.List(cty.String), "port":cty.Number})),
│ "protocol":cty.String})).

I guess Read and Create set services = nil but this is missing from Update? At

	tfservices := ServicesToTfServices(updatedMachine.Config.Services)

Currently the provider requires an open flyctl proxy. I have a POC builtin wireguard tunnel but it has a problem with DNS resolution. This probably needs eyes from someone who understands wireguard better than I do.

I would like to try creating a pulumi provider using pulumi-terraform-bridge but it seems this module is still not published

If you go to dov.dev/fly/fly-provider it gives 404

Why do I need to reference an existing app when creating a volume even if I just want to attach a machine to it?

edit: nvm I saw that the machine itself also references an app. So I guess an app is always grouping these entities?

I'm facing the following error when trying to run terraform apply

Error: fly wireguard tunnel must be open
│ 
│   with fly_machine.machine,
│   on main.tf line 31, in resource "fly_machine" "machine":
│   31: resource "fly_machine" "machine" {

Does anyone already get it ?

Say you change the name of a fly_app, which forces a replacement of the IP, machine, everything - they'll race to delete and anything the fly_app beats won't be able to delete, as it will now get a 404. (Without checking, I assume the API is something like /apps/:id/machines/:id).

If it already doesn't exist (either app or machine), delete should just succeed, remove from state. There's no need to alert that the app doesn't exist, it must be correct, since it was checked on create/import when we added it to the state.

Following this blog post I end up with following error:

fly_machine.mcServer: Creating...
╷
│ Error: Failed to create machine
│
│   with fly_machine.mcServer,
│   on main.tf line 33, in resource "fly_machine" "mcServer":
│   33: resource "fly_machine" "mcServer" {
│
│ Create request failed: 422 Unprocessable Entity, &{ID: Name: State: Region: InstanceID: PrivateIP: Config:{Env:map[] Init:{Exec:[] Entrypoint:[] Cmd:[]} Image: Metadata:<nil> Restart:{Policy:} Services:[]
│ Mounts:[] Guest:{CPUKind: Cpus:0 MemoryMb:0}} ImageRef:{Registry: Repository: Tag: Digest: Labels:{}} CreatedAt:0001-01-01 00:00:00 +0000 UTC}

I updated to

terraform {
  required_providers {
    fly = {
      source = "fly-apps/fly"
      version = "0.0.20"
    }
  }
}

which didn't help.

The logged struct doesn't seem to have the correct data (assuming that that is what is getting logged here).

This would be useful to fetch the current list of regions for automating something like 'one instance per region'.

I'd like to have the ability to modify and query the organizations I'm associated to if I have such a privilege. My want for this is to simplify onboarding for other teammates by making it a code change versus needing to pop into the dashboard (which is really well made, btw!).

From what I can tell, there's no API endpoints to do this programmatically so would that make this a moot request?

(Originally published at: https://jacky.wtf/2022/11/iAYM)

I have a single machine resource in my config with the optional name field excluded. On initial deploy, this results in an automatic name assignment, which is what I expected.

Expected behavior: On running terraform apply with a new image value, I expected an in place update that would only modify the image. Roughly equivalent to running fly deploy --image=[new image tag].
Actual behavior: The command errors due to the machine name

Terraform will perform the following actions:

  # fly_machine.server will be updated in-place
  ~ resource "fly_machine" "server" {
      ~ cpus     = 1 -> (known after apply)
      ~ cputype  = "shared" -> (known after apply)
      ~ id       = "..." -> (known after apply)
      ~ image    = "[old image]" -> "[new image]"
      ~ memorymb = 256 -> (known after apply)
      ~ mounts   = [
          ~ {
              ~ encrypted = false -> (known after apply)
              ~ size_gb   = 0 -> (known after apply)
                # (2 unchanged attributes hidden)
            },
        ]
      ~ name     = "crimson-waterfall-6616" -> (known after apply)
        # (4 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

fly_machine.server: Modifying... [id=...]
╷
│ Error: Can't mutate name of existing machine
│
│   with fly_machine.server,
│   on main.tf line 35, in resource "fly_machine" "server":
│   35: resource "fly_machine" "server" {
│
│ Can't swith name crimson-waterfall-6616 to

This may just be a bad expectation/usage on my part.

EDIT: The image update was applied, despite the error. Changing the env map exhibits the same behavior - the machine is properly updated, but a machine name mutation error is displayed.

In particular, sizeGb and encrypted are no-ops and should be marked as optional. Eventually they would allow creating a volume at machine launch time. For now, only a volume ID/name and a path are required.

Most providers include terraform import information for each resource. Example: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route53_record#import

I propose that we do the same thing here.

https://fly.io/docs/machines/guides-examples/machines-app-using-flyctl/ recommends using shared ipv4 addresses, but I don't see any way to create one in https://registry.terraform.io/providers/fly-apps/fly/latest/docs/resources/ip.

I was wondering if there is a likelihood that the postresql equivalent called cockroachdb might be supported ?

https://github.com/cockroachdb/terraform-provider-cockroach

my uses case is that I was using postresql but need to change to cockroachdb so I can use their change feeds.

I know that with the changeover to Fly Machines there is much flux.

I just wanted to know what blockers thee are so that I could know the extend of the effort .

Similar to the 404, I'm getting an inscrutable error when creating a machine (this is a different machine):

Create request failed: 422 Unprocessable Entity, &{ID: Name: State: Region: InstanceID: PrivateIP: Config:{Env:map[] Init:{Exec:[] Entrypoint:[] Cmd:[]} Image: Metadata:<nil> Restart:{Policy:}
Services:[] Mounts:[] Guest:{CPUKind: Cpus:0 MemoryMb:0}} ImageRef:{Registry: Repository: Tag: Digest: Labels:{}} CreatedAt:0001-01-01 00:00:00 +0000 UTC}

I have no idea what I did wrong, at the moment it looks like I'll need to resolve this via trial and error.

Improving the error message to indicate

1. what was sent
2. what was expected
3. why this was a problem
4. potential remedies

would be wonderful.

Repro:

1. Use the fly terraform provider to create a certificate via resource "fly_cert".
2. Manually remove the certificate from fly.
3. terraform plan
4. Observe the following error.

Error: Could not find AppCertificate
app.certificate

I was able to work around this by using terraform state rm.

Is there already a way in this TF provider v0.0.6 to trigger the build host to (re-)build images from Dockerfiles (and from Buildpacks as the other option), similar to what flyctl launch/deploy do, which can thereafter be started as machines (from the private registry)? Thanks.

It's in the Fly.io docs, but not in this provider.

Most of my apps do not expose any public ports and are accessed through caddy as a proxy. It looks like I can't create a machine that doesn't expose any public ports though.

The ports key and then the port key are both required. Is there a way to bypass this by any chance? I do need the internal_port and have that set along with protocol.

I'm hoping to be able to do something like this:

services = [
    {
      internal_port =  9100
      protocol = "tcp"
    },
    {
      internal_port = 9101
      protocol = "tcp"
    }
]

The API will generate a unique machine name if none is provided, so the name argument should be optional.