hashicorp / hcl Goto Github PK

1.02 is parsed as 1.2.

I've noticed that you don't need to terminate a curly brace on a structure. Is this expected?

Here's a failing test case that demonstrates the problem:

diff --git a/decoder_test.go b/decoder_test.go
index c02bb57..f92776b 100644
--- a/decoder_test.go
+++ b/decoder_test.go
@@ -225,6 +225,12 @@ func TestDecode_interface(t *testing.T) {
                },

                {
+                       "structure_unterminated_brace.hcl",
+                       true,
+                       nil,
+               },
+
+               {
                        "nested_block_comment.hcl",
                        false,
                        map[string]interface{}{
diff --git a/test-fixtures/structure_unterminated_brace.hcl b/test-fixtures/structure_unterminated_brace.hcl
new file mode 100644
index 0000000..091e9e6
--- /dev/null
+++ b/test-fixtures/structure_unterminated_brace.hcl
@@ -0,0 +1,3 @@
+foo "baz" {
+       key = 7
+       foo = "bar"

Probably best solved with documentation or better error messages, but the rules to escape quotes inside a quoted string literal has changed in the new HCL parser:

Old parser:

resource "template_file" "kafka" {
  filename = "/dev/null"
  vars {
    broker_list = "${join(\",\", formatlist(\"%s:%d\", aws_route53_record.kafka.*.name, 9092))}"
    zookeeper_list = "${join(\",\", formatlist(\"%s:%d\", aws_route53_record.kafka.*.name, 2181))}"
  }
}

New parser:

resource "template_file" "kafka" {
  filename = "/dev/null"
  vars {
    broker_list = "${join(",", formatlist("%s:%d", aws_route53_record.kafka.*.name, 9092))}"
    zookeeper_list = "${join(",", formatlist("%s:%d", aws_route53_record.kafka.*.name, 2181))}"
  }
}

Relevant code in HCL scanner: https://github.com/hashicorp/hcl/blob/master/hcl/scanner/scanner.go#L455

This could be an issue with unsuspecting Terraform users upgrading to 0.6.7 and suddenly seeing syntax errors.

As a side-effect, the change also brakes syntax highlighting in Sublime Text but not in vim.

The new syntax is easier to write and read, but it is a breaking change so I wanted to make sure it was addressed before downstream releases.

It feels like a strange omission not to include write-support for HCL. This decision was probably made with very good reasons but I have yet to discover them.

Making HCL read-only makes it an all-or-nothing thing: either a human writes the entire configuration file in HCL or the server writes it entirely in JSON (which renders it uneditable, or at least very clumsy, for a human).

The typical scenario I have in mind is a server that has a big configuration file with lots of bells and whistles. Some of these options are editable in the UI, while others are considered advanced and aren't exposed in the UI (because only 1% of the users would need them).

I don't see a way to make some variables machine-editable while maintaining an easy configuration language (for humans) for the more advanced options. One example of a configuration format that does support writing (and is reasonably readable) is the git configuration format.

Advice and recommendations are highly welcome. Let me know if none of this makes any sense, I'll happily clarify.

Perhaps what I'm doing just isn't part of the scope of HCL. I'd be cool with that too, I'll look for an alternative.

I have some hcl like so:

a = "test"
b = "test2"
c {
   d = "test3"
}

buf := ... // Assuming hcl is in buffer

root, err := hcl.Parse(buf.String()) // Parse

list, _ := root.Node.(*ast.ObjectList) // Turn it to an ObjectList

fmt.Println(len(list.Filter("a").Items)) // 1
fmt.Println(len(list.Filter("a", "b").Items)) // 0
fmt.Println(len(list.Filter("a", "b", "c").Items)) // 0

As can be seen above, if I filter for 1 key, it finds it, however, if I attempt to filter for multiple keys, it always finds nothing.

There seems to be support for JSON <--> HCL; a CLI that makes it easy to convert JSON into HCL would make it easy to integrate HCL into other tool chains?

hcl sounds almost exactly the same as HOCON, which I've contributed to in the past. They both have very JSON-like file formats and there's quite a bit of overlap. They both support types like 1kb even! It'd be cool if maybe the two could work together to define a common spec.

Right now HOCON is used by a lot of Scala & Java projects since Typesafe authored the original client. Puppet has also created a Ruby version that they're using.

Hocon supports some features that hcl doesn't such as importing files and having types for durations, which I think could be added to hcl fairly easily I think. I'm not super fluent in yacc, but it looks like hocon and hcl may handle variables slightly differently, so it's pretty similar as both support syntax like ${aws_instance.web.id}. The Terraform docs seem to have more details on this than the hcl README, so that might be a good place to look.

Not sure if this is worth the effort, but thought it was worth making you guys aware of each other's projects in case there is anything that does make sense.

CC @havocp - maintainer of hocon

I might be completely missing something, but for my application I'm missing a way to have something like the Unmarshaler interface from encoding/json (without having to go through the "raw" object).

Just using Decode() does 90% of the work for me correctly, but I have a few fields where I want to convert the input before it goes into my data structure (for example "1d" to int(86400)).

Hi,

Since HCL compiles to JSON, using JSON schema to validate the configuration is natural. For this I suggest a feature of the sort which would look for a magic comment in the file of the form:

# json_schema_web = "https://schema-domain.com/path/to/schema-file.json"

or

# json_schema_path = "../config/service-name-config-schema.json"

Oh and if magic comments are too much magic (I feel so personally), an alternative approach could be used to specify the schema file path.

The validation can be powered by https://github.com/xeipuuv/gojsonschema

When clubbed with #24 I see a pass-through which applications in other languages can invoke so that the HCL format can be used to write config, validate it and translate it to general-purpose JSON input. This means existing apps using JSON can benefit from simpler config and added validation.

PS: This looks like a duplicate of #3 which is closed by what looks like an unrelated commit.

Hi,

I'm looking at implementing a variant of @fatih's hclfmt with the differences that it would:

• be implemented as sub-commands/verbs in the utilities that use HCL e.g. terraform fmt. So that it would be easily available to users of those utilities without building/installing an additional binary
• follow the semantics of go fmt and gofmt a bit more closely, in that terraform fmt would rewrite all config files in your current PWD, but you could also provide list/write/diff/stdin arguments to change the behaviour
• configurable file extensions, e.g. .hcl or .tf

The intention is that most of the code would be implemented in the HCL project but wouldn't be specific to CLI. Then other utilities, such as Terraform and Nomad, could implement the CLI flags and pass over the necessary file extensions and options. The HCL project could also provide an hclfmt binary that works on .hcl files.

Does this sound sensible and useful? Would you like me to finish writing tests and submit a PR for this project and Terraform?

I have a rather long command as a string in hcl.

provisioner "local-exec" {
  command = "knife solo bootstrap -N ${self.name} -j '{\"mysql\": {\"server_root_password\": \"${var.mysql_root_password}\"}}' --no-berkshelf --no-host-key-verify -i ${var.pvt_key} -r ${var.run_list.db} ${var.remote_user}@${self.ipv4_address} | tee -a ${self.name}.log
}

Is there a way to break it and still keep it decoded as a single string?
I can't find anything in test fixtures regarding this.

It'd be nice with support for anonymous struct fields similar to what encoding/json does.

Hi. This is not an issue, just some questions around the HCL spec. According to the README:

Numbers are assumed to be base 10. If you prefix a number with 0x, it is treated as a hexadecimal. If it is prefixed with 0, it is treated as an octal. Numbers can be in scientific notation: "1e10".

I have couple of questions about this (where NUMBER and FLOAT are HCL tokens).

• Can we assume that NUMBER in HCL do have the same set of spec as integer literals in Go: https://golang.org/ref/spec#Integer_literals ?
• Can we assume that a FLOAT token is also the same as a Go float: https://golang.org/ref/spec#Floating-point_literals ?
• Shouldn't be 1e10 a FLOAT? The description above makes it like it's a NUMBER.

As README.md says:

Arrays can be made by wrapping it in []. Example: ["foo", "bar", 42]. Arrays can contain primitives and other arrays, but cannot contain objects. Objects must use the block syntax shown below.

At the same time in hcl/parse.y

listitem ::= number | STRING

Note list is missing here.

Please update either documentation or grammar

Also note there's difference in naming: array vs list

Hi, after trying to implement printer behaviour in IntelliJ-HCL plugin, I've some questions related to hcl format:
In comment_aligned.input and comment_aligned.output some properties (1-3) have equals sign aligned, but 5,6 are not aligned with each other, so I'd expect something like

    #deneme arslan
    fatih       = ["fatih"]  # yoo4
    #fatih arslan
    fatiharslan = ["arslan"] // yoo5

instead of

    #deneme arslan
    fatih = ["fatih"] # yoo4

    #fatih arslan
    fatiharslan = ["arslan"] // yoo5

As I understand one-line (with probably leading or same-line comment) properties grouped and have equals sign and same-line comment (if any) aligned. But that does not for properties with both leading and same-line comments. Looks like bug.

Also it would be nice to see some document which describes printer/formatter behaviour.

PS: Is there some communication channel to ask question about HCL/HIL instead of creating issues here?
//cc @fatih @mitchellh

Heredoc support seems to be gone with #45, hcl/scanner/scanner.go does not tokenize < like the old lexer did: 26239b8#diff-3ea40c09007c8c8608ccf619755a4adfR97

Created issue to track @jen20's work on https://github.com/hashicorp/hcl/tree/b-heredoc-decoding

@jen20 hashicorp/terraform#3840 just bit me on master in a dev build, I thought an open issue would be nice in hcl to see what's going on.

Unfortunately due to a lack of pinning, I don't know exactly when this changed, but decoding a top level array of objects no longer works. Any suggestions on how to fix or get around this?

Example:

type Template struct {
    Source string
    Destination string
    Command string
}

templates := struct {
    Templates []*Template `hcl:"template"`
}

err := hcl.Decode(&templates, `
template {
    source = "blah"
    destination = "foo"
    command = "baz"
}

template {
    source = "blahblah"
    destination = "foofoo"
    command = "bazbaz"
}`)

Currently it fails with: Error: root.template[0]: not an object type for struct (*ast.LiteralType)

First off, I'm unsure if this is the expected behaviour or not.

When decoding maps it's currently not possible to detect duplicates, as HCL will overwrite (last one wins) the map key.

The HCL fix should be pretty easy, i.e. stick something like the following into decodeMap() right before
setting the value on the map:

dupKey := resultMap.MapIndex(key)
if dupKey.IsValid() {
        return fmt.Errorf(
                "%s: duplicate map key %v", name, key)
}

See for example: hashicorp/consul#1055 or generally any HCL file for that matter:

variable "foo" {
        description = "foo variable"
        default = "bar"
}

...

variable "foo" {
        description = "foo variable"
        default = "crux"
}

Could lead to interesting gotchas when depending on the defaults.

I have a patch + tests for HCL, but this probably will bubble up to HCL users that will need munging since this changes behaviour somewhat and I do not know how to best (or even if) handle that.

Is it the case that decoding non-pointer struct fields of a container struct is not supported? Given the following input:

test {
    a = "hello"
    b {
        c = 1
        d = 2
    }
}

Decoded into:

map[string]*A

With types:

type B struct {
    c int     `json:"c"`
    d int     `json:"d"`
}

type A struct {
    a string  `json:"a"`
    b B       `json:"b"` // not a pointer
}

The *A is decoded partially correctly, its field a, a string is correct, but the non-pointer struct field b has a zero value. I get the equivalent of:

test {
    a = "hello"
    b {
        c = 0 // wrong
        d = 0 // wrong
    }
}

Decoding that same input into:

map[string]interface{}

Yields exactly the result I would expect, all the nested fields are present and have the correct keys.

Is this simply not supported or is it a bug? I'll note that the standard library JSON unmarshaler has no problem with unmarshaling into the same structure.

Being able to take a .tf file that uses HCL constructs and emit the equivalent JSON would help people writing tool that drive Terraform, as well as make it easier to provide both JSON and HCL equivalent examples on the website.

Support heredocs for strings.

With TextUnmarshaller own data types can be defined, such as a wrapper for time.Duration, etc.

I don't see why there's need for commas. Make them optional for backward-compatibility.

Currently, hclfmt doesn't seem to allow the following formatting:

variable "foobar" {}

Instead, this will be changed to:

variable "foobar" {
}

This seems odd to me and I'm not sure if this is intentional.

It would be nice if decodePtr() would operate on the thing being pointed to rather than decoding into a new concrete value and assigning its address to the pointer.

I've found this behaviour handy using encoding/json, for example. Say I have a struct prepopulated with defaults. The struct corresponds to options nested under a key in a config file. I can create an instance of an anonymous struct with a field whose name refers to that key and value is a pointer to my original struct. I can then decode into the anonymous struct instance, and only fields present in the config file data will be updated, leaving the default fields intact.

Just updated to the latest hcl code and will no longer expand hcl object into slice of structs

var deploy struct {
    Services []service.Config `hcl:"service,expand"`
}

This used to work.

It would be nice to have an emacs code that can keep indentation right, etc. Anyone use emacs? Is some pre-existing mode a good enough match?

That's what you get from ioutil.ReadFile / ioutil.ReadAll.

Could hcl work without slurping the whole input into memory at once? Just a *bufio.Reader oughta be enough.

We had this in TF, which did not generate issues

provisioner "remote-exec" {
inline = ["foo",
"bar",
]
}

Please, add links to HCL compliant parsers etc.
Vim plugin would be great too 👍

Thanks!

Linking to :
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/terraform-tool/fT7VtRzsVhw/PmgPR4lqAZ4J

Decode is being used to decode expected JSON file, which is:

{
    "name": "terraform-test-app",
    "config_vars": {
        "FOO": "bar"
    }
}

This JSON file should create structure like this (native JSON decoder from "encoding/json"):

map[string]interface {}{
  "name": "terraform-test-app",
  "config_vars": map[string]interface {}{
    "FOO":"bar"
  }
}

But, in fact, we have:

map[string]interface {}{
  "name": "terraform-test-app",
  "config_vars": []map[string]interface {}{
    map[string]interface {}{
      "FOO":"bar"
    }
  }
}

In case there is interest in cross-language support, I've implemented a python parser for HCL that uses the same test fixtures as this repository.

https://github.com/virtuald/pyhcl

For my use case this works by simply adding && fieldKind != reflect.Ptr to

Hi,
I want to contribute to this project by creating a site like http://sassmeister.com. An online site, that helps people learn HCL more easily.
But I don't really know where to start an with tools to use, which architecture to put in place to do that :)
I opened this issue to let people throw in their ideas.
Regards
Lionel

After reviewing #6, I attempted to implement heredoc for user_data in terraform. I had added the following:

    user_data = <<-USER_DATA

    #cloud-config

    coreos:
      etcd:
        # multi-region and multi-cloud deployments need to use $public_ipv4
        addr: $private_ipv4:4001
        peer-addr: $private_ipv4:7001
      units:
        - name: etcd.service
          command: start
        - name: fleet.service
          command: start

    USER_DATA

This throws the following error:
Error loading config: Error parsing /my_file.tf: Line 76, column 0: syntax error

It appears hcl is not honoring the - as outlined in http://en.wikipedia.org/wiki/Here_document:

Appending a minus sign to the << has the effect that leading tabs are ignored. This allows indenting here documents in shell scripts without changing their value.

If I unindent everything and remote the - it works OK, but IMO the file becomes much harder to read since all the heredoc data needs to be completely left-aligned.

change to be able to use environmental variables.

Is it possible to interpolate environment variables in HCL config?
An example would be very helpful.

The JSON parser fails on the values true, false, and null.

This can be tested by adding these values to json/test-fixtures/types.json and running go test in json/.

{
    "foo": "bar",
    "bar": 7,
    "baz": [1,2,3],
    "foo": -12,
    "bar": 3.14159,
    "foo": true
}

$ go test
--- FAIL: TestParse (0.00 seconds)
    parse_test.go:40: Input: types.json

        Error: Line 7, column 9: syntax error
FAIL
exit status 1
FAIL    github.com/hashicorp/hcl/json   0.011s

Unless I'm reading the parser incorrectly, despite that the README says hcl supports other types of boolean values (yes, no, on, off), I don't think it actually does. Either the README or the lexer should be changed.

The python implementation does support these alternative boolean values.

I'm very curious if you plan to enable expression and functions and to what extent you plan to build out the configuration language.

Given input:

foo = [
 # foo
          1, # bar
         # baz
]

Expected output:

foo = [
    # foo
    1, # bar
    # baz
]

Actual output:

foo = [
        1, # bar
]

# foo

# baz

// cc @fatih

hcl/json fails to parse this otherwise valid JSON file:

{
    "a": 1e-10,
    "b": 1e+10,
    "c": 1e10
}

encoding/json does support this:
http://play.golang.org/p/qqWDzlwvLX

hcl/hcl should also probably support scientific notation of floats.

The following HCL causes an "index out of range" panic when you try to decode it:

section {
        subsection {
        }
}
section "subsection" {
}

Test program:

package main

import (
        "github.com/hashicorp/hcl"
)

var hclText = `
section {
        subsection {
        }
}
section "subsection" {
}
`

func main () {
        var shadow interface{}
        hcl.Decode(&shadow, hclText)
}

Backtrace:

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/hashicorp/hcl.(*decoder).decodeMap(0xc820087e60, 0xc82000ab40, 0xf, 0x7f36c08a11c0, 0xc82000e8e0, 0x4ebfe0, 0xc820036078, 0xd5, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:301 +0x1081
github.com/hashicorp/hcl.(*decoder).decode(0xc820087e60, 0xc82000ab40, 0xf, 0x7f36c08a12f0, 0xc820020360, 0x4ebfe0, 0xc820036078, 0xd5, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:83 +0x69c
github.com/hashicorp/hcl.(*decoder).decodeSlice(0xc820087e60, 0xc82000aae0, 0xc, 0x7f36c08a11c0, 0xc82000e8a0, 0x4e44c0, 0xc82000e840, 0x57, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:395 +0x5c6
github.com/hashicorp/hcl.(*decoder).decode(0xc820087e60, 0xc82000aae0, 0xc, 0x7f36c08a11c0, 0xc82000e8a0, 0x4eba20, 0xc82000ab00, 0xd4, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:87 +0x774
github.com/hashicorp/hcl.(*decoder).decodeMap(0xc820087e60, 0x5405e0, 0x4, 0x7f36c08a11c0, 0xc82000e5c0, 0x4ebfe0, 0xc820010510, 0x15, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:330 +0xf1d
github.com/hashicorp/hcl.(*decoder).decode(0xc820087e60, 0x5405e0, 0x4, 0x7f36c08a11c0, 0xc82000e5c0, 0x4eba20, 0xc82000a4b0, 0xd4, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:83 +0x69c
github.com/hashicorp/hcl.(*decoder).decodeInterface(0xc820087e60, 0x5405e0, 0x4, 0x7f36c08a11c0, 0xc82000e5c0, 0x4eba20, 0xc82000a4b0, 0xd4, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:249 +0x384
github.com/hashicorp/hcl.(*decoder).decode(0xc820087e60, 0x5405e0, 0x4, 0x7f36c08a11c0, 0xc82000e5c0, 0x4eba20, 0xc82000a4b0, 0xd4, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:81 +0x628
github.com/hashicorp/hcl.DecodeObject(0x4df000, 0xc82000a4b0, 0x7f36c08a11c0, 0xc82000e5c0, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:43 +0x25f
github.com/hashicorp/hcl.Decode(0x4df000, 0xc82000a4b0, 0x56ae60, 0x37, 0x0, 0x0)
    gopath/src/github.com/hashicorp/hcl/decoder.go:26 +0xb9
main.main()
    crash_hcl.go:18 +0x5f

Embarrassing bug discovery. Because hcl.Decode() takes a string I accidentally passed an absolute file path into HCL at which point it just seems to hang here: https://github.com/hashicorp/hcl/blob/master/hcl/scanner/scanner.go#L123. An example that should replicate this would be:

bar := &bar{}
err := hcl.Decode(bar, "/go/src/github.com/hashicorp/hcl/scanner/scanner.go")

I had to go all the way down the rabbit hole to find out I was doing this because nothing was bubbling back up the trace. Not sure how to fix it, but wasted the better part of a day trying to find this as sad as it may seem.

Currently the scanner will output errors including token position (line/col) format which is great for creating linters for text editors. I am trying to create a linter for Sublime Text, but noticed that some of the errors don't reference token position in the parser.

Hey there--

So as part of building Terraframe I'm finding myself needing to occasionally need to inject a human-readable comment into the JSON output (largely for debugging). Having to strip my comments back out before feeding it into Terraform is a drag.

Maybe we could standardize on, say, any field beginning with "//" being ignored by the HCL reader? Example:

{
  "//builder": "Built by Terraframe 0.1.0.",
  "resource": {
    "aws_instance": {
      "test": {
        "//source": "test_instance.terraframe:25",
      }
    }
  }
}

It'd help with building good tooling on top of the platform, and would make my life a lot easier. :)

Thanks!

The mapping from HCL to JSON (and vice versa) isn't immediately clear. I haven't been able to find concrete documentation for this yet. The fixtures help, but there seems to be a mix of valid/invalid examples.