ability to create an index about terraform-provider-elasticsearch HOT 5 CLOSED

It would be great if the elasticsearch_index resource feature is merged to the master , index creation via TF CI pipelines is a requirement for most ES use cases

from terraform-provider-elasticsearch.

Hi @nelg are you using this provider to configure the index lifecyle? If so you can post the terraform config you're using now?

from terraform-provider-elasticsearch.

yes, we use the provider to configure the index lifecycle, as per:

resource "elasticsearch_index_lifecycle_policy" "beat_policy" {
  name = "${var.beat_name}-${var.env}-${var.beat_version}"
  body = <<EOF
{
"policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_docs": 500000000
          },
          "set_priority": {
            "priority": 500
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "number_of_replicas": 0,
            "include": {},
            "exclude": {},
            "require": {
              "data": "warm"
            }
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age": "14d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
EOF
provisioner "local-exec" {
     when = destroy
     command = <<EOF
curl -XDELETE "${var.elasticsearch_url}/%3C${var.beat_name}-${var.env}-${var.beat_version}-%7Bnow%2Fy%7Byyyy%7D%7D-000001%3E" ${var.creds} -H 'Content-Type: application/json';
EOF
}
}


# And the template
resource "elasticsearch_index_template" "beat_template" {
  name = "${var.beat_name}-${var.env}-${var.beat_version}"
  body = <<EOF
{
  "order": 2,
  "index_patterns": [
    "${var.beat_name}-${var.env}-${var.beat_version}-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "${elasticsearch_index_lifecycle_policy.beat_policy.name}",
        "rollover_alias": "${var.beat_name}-${var.env}-${var.beat_version}"
      }
    }
  }
}
EOF
}

This bit works fine, but in order for an ILM policy to apply to an index, and to know when to rotate it, what to call the index, it needs an initial index creates, that has the write alias.

The below case, uses URL date math encoding, to create an index, for example, for filebeat it would create: filebeat-prod-7.5.0-2019-000001. The trailing 000001 is the pattern that ILM will use to rotate it, so it will create a filebeat-prod-7.5.0-2019-000002 index, when this is rotated. If the year changes, then it will create say: filebeat-prod-7.5.0-2020-000003

My current work around for the provider not having this is to use a null resource:

 triggers = {
   template =  elasticsearch_index_template.beat_template.name
 }
  provisioner "local-exec" {
    command = <<EOF
curl -XPUT "${var.elasticsearch_url}/%3C${var.beat_name}-${var.env}-${var.beat_version}-%7Bnow%2Fy%7Byyyy%7D%7D-000001%3E" ${var.creds} -s -H 'Content-Type: application/json' -d'{  "aliases": {    "${var.beat_name}-${var.env}-${var.beat_version}": {      "is_write_index": true    }  }}'
EOF
  }
  depends_on = [elasticsearch_index_lifecycle_policy.beat_policy]
}

However, this is not nice, because I have to use curl, pass the authentication, etc. It also needs a separate curl for destroy when deleting the ILM policy. The curl is crude. I better approach would be to refuse to delete the index, if it has records in it, unless a force flag is set, similar to an s3 bucket delete when the bucket has objects in it.

from terraform-provider-elasticsearch.

There's a branch with a elasticsearch_index resource: f77349b, it was done on a fork and I haven't had time to review it. Can you take a look at that and see if it would meet your needs?

from terraform-provider-elasticsearch.

It looks very close. From reading the source code, it looks like they are being specific as to which settings can be set for an index, in which case, it would need support to set aliases added to it.

IE, line 178 would need a new line below it:
body["aliases"] = aliases

plus all the other appropriate additions, almost identical to settings, to add support for aliases.

So, I guess my needs should be a feature request on to the elasticsearch_index resource.

from terraform-provider-elasticsearch.