kubernetes-sigs / yaml Goto Github PK

View Code? Open in Web Editor NEW

This project forked from ghodss/yaml

230.0 13.0 70.0 203 KB

A better way to marshal and unmarshal YAML in Golang

License: Other

Go 100.00%

k8s-sig-api-machinery

yaml's Introduction

YAML marshaling and unmarshaling support for Go

kubernetes-sigs/yaml is a permanent fork of ghodss/yaml.

Introduction

A wrapper around go-yaml designed to enable a better way of handling YAML when marshaling to and from structs.

In short, this library first converts YAML to JSON using go-yaml and then uses json.Marshal and json.Unmarshal to convert to or from the struct. This means that it effectively reuses the JSON struct tags as well as the custom JSON methods MarshalJSON and UnmarshalJSON unlike go-yaml. For a detailed overview of the rationale behind this method, see this blog post.

Compatibility

This package uses go-yaml and therefore supports everything go-yaml supports.

Caveats

Caveat #1: When using yaml.Marshal and yaml.Unmarshal, binary data should NOT be preceded with the !!binary YAML tag. If you do, go-yaml will convert the binary data from base64 to native binary data, which is not compatible with JSON. You can still use binary in your YAML files though - just store them without the !!binary tag and decode the base64 in your code (e.g. in the custom JSON methods MarshalJSON and UnmarshalJSON). This also has the benefit that your YAML and your JSON binary data will be decoded exactly the same way. As an example:

BAD:
	exampleKey: !!binary gIGC

GOOD:
	exampleKey: gIGC
... and decode the base64 data in your code.

Caveat #2: When using YAMLToJSON directly, maps with keys that are maps will result in an error since this is not supported by JSON. This error will occur in Unmarshal as well since you can't unmarshal map keys anyways since struct fields can't be keys.

Installation and usage

To install, run:

$ go get sigs.k8s.io/yaml

And import using:

import "sigs.k8s.io/yaml"

Usage is very similar to the JSON library:

package main

import (
	"fmt"

	"sigs.k8s.io/yaml"
)

type Person struct {
	Name string `json:"name"` // Affects YAML field names too.
	Age  int    `json:"age"`
}

func main() {
	// Marshal a Person struct to YAML.
	p := Person{"John", 30}
	y, err := yaml.Marshal(p)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	age: 30
	name: John
	*/

	// Unmarshal the YAML back into a Person struct.
	var p2 Person
	err = yaml.Unmarshal(y, &p2)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(p2)
	/* Output:
	{John 30}
	*/
}

yaml.YAMLToJSON and yaml.JSONToYAML methods are also available:

package main

import (
	"fmt"

	"sigs.k8s.io/yaml"
)

func main() {
	j := []byte(`{"name": "John", "age": 30}`)
	y, err := yaml.JSONToYAML(j)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	age: 30
	name: John
	*/
	j2, err := yaml.YAMLToJSON(y)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(j2))
	/* Output:
	{"age":30,"name":"John"}
	*/
}

yaml's People

Contributors

Stargazers

Watchers

yaml's Issues

annotations key start with ？character after `yaml.Marshal`

the raw deployment:

when I run the yaml.Marshal(deployment), the annotations key start with ？character.

I find there are ? if the length of key is bigger than 129

func TestMarshal(t *testing.T) {
	m := map[string]string{
		"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xx":   "xxxxx",
	"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx":  "xxxxx",
		"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xx": "xxxxx",
	}
	raw, err := yaml.Marshal(m)
	fmt.Println(err)
	fmt.Println(string(raw))
}

panic: runtime error: invalid memory address or nil pointer dereference

Hi.

Got such an issue:

~/operator-sdk$ make build
go build -i -v -x -gcflags "all=-trimpath=/home/oceanfish81" -asmflags "all=-trimpath=/home/oceanfish81" -ldflags " -X 'github.com/operator-framework/operator-sdk/internal/version.Version=v1.1.0+git' -X 'github.com/operator-framework/operator-sdk/internal/version.GitVersion=v1.1.0-16-g33e37efc-dirty' -X 'github.com/operator-framework/operator-sdk/internal/version.GitCommit=33e37efc921bab5cd6b5c84eb68a604d733e06ff' -X 'github.com/operator-framework/operator-sdk/internal/version.KubernetesVersion=v1.18.8' " -o build ./cmd/{operator-sdk,ansible-operator,helm-operator}
WORK=/tmp/go-build342156229
go: downloading sigs.k8s.io/controller-runtime v0.6.3
# get https://proxy.golang.org/sigs.k8s.io/controller-runtime/@v/v0.6.3.zip
go: downloading k8s.io/apimachinery v0.18.8
# get https://proxy.golang.org/k8s.io/apimachinery/@v/v0.18.8.zip
go: downloading github.com/operator-framework/operator-lib v0.1.0
# get https://proxy.golang.org/github.com/operator-framework/operator-lib/@v/v0.1.0.zip
go: downloading k8s.io/client-go v0.18.8
# get https://proxy.golang.org/k8s.io/client-go/@v/v0.18.8.zip
go: downloading sigs.k8s.io/yaml v1.2.0
# get https://proxy.golang.org/sigs.k8s.io/yaml/@v/v1.2.0.zip
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7fbb1c2f86b6]

goroutine 50 [running]:
x509.loadSystemRoots
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/x509/root_unix.go:81
x509.initSystemRoots
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/x509/root.go:23
sync.Once.doSlow
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/sync/once.go:66
sync.Once.Do
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/sync/once.go:57
x509.systemRootsPool
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/x509/root.go:18
crypto..z2fx509.Certificate.Verify
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/x509/verify.go:776
crypto..z2ftls.Conn.verifyServerCertificate
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/tls/handshake_client.go:846
crypto..z2ftls.clientHandshakeStateTLS13.readServerCertificate
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/tls/handshake_client_tls13.go:455
crypto..z2ftls.clientHandshakeStateTLS13.handshake
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/tls/handshake_client_tls13.go:85
crypto..z2ftls.Conn.clientHandshake
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/tls/handshake_client.go:209
crypto..z2ftls.Conn.Handshake
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/crypto/tls/conn.go:1362
http.func2
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/net/http/transport.go:1509
created by net..z2fhttp.persistConn.addTLS
/home/oceanfish81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/net/http/transport.go:1505 +0x290
make: *** [Makefile:60: build] Error 2

Here is my environment:

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/oceanfish81/.cache/go-build"
GOENV="/home/oceanfish81/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/oceanfish81/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/oceanfish81/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/oceanfish81/gollvm_dist"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/oceanfish81/gollvm_dist/tools"
GCCGO="/home/oceanfish81/gollvm_dist/bin/llvm-goc"
AR="ar"
CC="/usr/bin/clang"
CXX="/usr/bin/clang++"
CGO_ENABLED="1"
GOMOD="/home/oceanfish81/operator-sdk/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build573184114=/tmp/go-build -gno-record-gcc-switches -funwind-tables"

Here is what I am using:

$ go version
go version go1.15.2 gollvm LLVM 12.0.0git linux/amd64

Could be obtained here

CC @ianlancetaylor @cherrymui @thanm

JSONToYAML doesn't preserve map order

package main

import (
        "encoding/json"
        "fmt"

        "sigs.k8s.io/yaml"
)

type A struct {
        B string
        A string
}

func main() {
        a := A{"B", "A"}
        y, _ := yaml.Marshal(a)
        fmt.Printf("yaml.Marshal\n%s\n", y)
        j, _ := json.MarshalIndent(a, "", " ")
        fmt.Printf("json.Marshal\n%s\n\n", j)
        y, _ = yaml.JSONToYAML(j)
        fmt.Printf("yaml.JSONToYAML\n%s\n", y)
}

yaml.Marshal
A: A
B: B

json.Marshal
{
 "B": "B",
 "A": "A"
}

yaml.JSONToYAML
A: A
B: B

Request to create an official fork of go-yaml

Currently, the go-yaml library is used heavily by kubernetes and many subprojects. In kustomize, we use both go-yaml.v2 and go-yaml.v3 as well as this library for all of our yaml processing.

An update to go-yaml v3 resulted in what would have been a breaking change in kustomize. We talked to the go-yaml maintainer, who agreed to make the fixes, but could not complete them in time for our deadline. As a result, we created a temporary fork of go-yaml v3 with the intent to remove the fork once the changes were made upstream. More details can be found in this issue: kubernetes-sigs/kustomize#4033

Unfortunately, shortly after we created the fork, the maintainer stopped responding to our issues and did not review the PRs we made to upstream the fix. As I understand it, there is only one go-yaml maintainer and they are very busy. The last response we received was in June 2021: go-yaml/yaml#720 (comment)

Maintaining the fork within kustomize is undesirable not only because of the overhead to the kustomize maintainers, but also because kustomize is built into kubectl, so our fork is now also vendored in k/k. It was accepted into kubectl with the understanding that it would be a temporary fork, but now I'm not so sure we will ever be able to remove it. Besides the fixes that we (kustomize) need to have upstream, it seems that we are not the only ones with this problem; there are lots of open issues and PRs in the go-yaml library with no response.

We would like to propose an official, SIG-sponsored fork of the go-yaml library. I had a very brief exchange in slack about this and I think it's time to push this forward. In that slack exchange I was advised that if we do have this official fork, we would probably want to fold it into this repo, hence why I am creating the issue here.

I would appreciate some advice on next steps and some indication about whether or not a permanent fork would be acceptable.

cc @liggitt @apelisse @kevindelgado @KnVerey @mengqiy

Support `yaml` tags

yaml/README.md

Line 11 in c3772b5

 In short, this library first converts YAML to JSON using go-yaml and then uses `json.Marshal` and `json.Unmarshal` to convert to or from the struct. This means that it effectively reuses the JSON struct tags as well as the custom JSON methods `MarshalJSON` and `UnmarshalJSON` unlike go-yaml. For a detailed overview of the rationale behind this method, [see this blog post](http://web.archive.org/web/20190603050330/http://ghodss.com/2014/the-right-way-to-handle-yaml-in-golang/). 

I got bitten by the fact that this is a YAML library but it needs json tags. That's not intuitive to me.

Would it be possible to also support yaml tags?

Strict unmarshalling should be case-sensitive

Using yaml.UnmarshalStrict(...) compares keys in a case insensitive way.

I would expect it to do case sensitive comparisons of key names so that if I define a key in my schema called somethingGreat, a key in a yaml file called somethingGREat wouldn't match, and the unmarshal would fail with the error unknown field "somethingGREat".

support for "flow" in struct tag

using "gopkg.in/yaml.v2" we can mark field tags option with "flow" like this

type Test struct {
	list []string `yaml:"list,flow"`
}

which will marshal a struct like this:
&Test{list: []string{"a", "b", "c"} into the following yaml string:

list: [a, b, c]

test on playground

This package doesn't seem to handle such option in struct tag.
It doesn't read the flow option in json: tag as it's not supported by the json library. Adding a second tag with yaml:"list, flow" doesn't wok either.

Did i missed something ? Is there a way to achieve this result with this package or can we expect a way to do it in future revision ?

README example slightly off

In the README there is a bit of code that says:

	j := []byte(`{"name": "John", "age": 30}`)
	y, err := yaml.JSONToYAML(j)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	name: John
	age: 30
	*/

But really the first output is age then name. Usually this would be no big deal, but it gives the false sense that this library preserves field order like json.Marshal does with structs.

Struct embedding loses type information and causes failed parsing.

package main

import (
	"fmt"

	"sigs.k8s.io/yaml"
)

type Outer struct {
	A string `json:"a" yaml:"a"`
	B string `json:"b" yaml:"b"`
}

type Inner struct {
	A string `json:"a" yaml:"a"`
}

type WithInline struct {
	Inner Inner  `json:",inline" yaml:",inline"`
	B     string `json:"b" yaml:"b"`
}

type WithEmbed struct {
	Inner `json:",inline" yaml:",inline"`
	B     string `json:"b" yaml:"b"`
}

func main() {
	o := Outer{}
	err := yaml.Unmarshal([]byte("a: 10\nb: 11\n"), &o)
	extra := ""
	if err != nil {
		extra = fmt.Sprintf(" [err=%v]", err)
	}
	fmt.Printf("o=%#v%s\n", o, extra)

	wi := WithInline{}
	err = yaml.Unmarshal([]byte("a: 10\nb: 11\n"), &wi)
	extra = ""
	if err != nil {
		extra = fmt.Sprintf(" [err=%v]", err)
	}
	fmt.Printf("wi=%#v%s\n", wi, extra)

	we := WithEmbed{}
	err = yaml.Unmarshal([]byte("a: 10\nb: 11\n"), &we)
	extra = ""
	if err != nil {
		extra = fmt.Sprintf(" [err=%v]", err)
	}
	fmt.Printf("we=%#v%s\n", we, extra)
}

yields

o=main.Outer{A:"10", B:"11"}
wi=main.WithInline{Inner:main.Inner{A:""}, B:"11"}
we=main.WithEmbed{Inner:main.Inner{A:""}, B:"11"} [err=error unmarshaling JSON: while decoding JSON: json: cannot unmarshal number into Go struct field WithEmbed.a of type string]

yamlUnmarshal cannot unmarshal time.Duration

Looking at the source code, the Unmarshal function uses jsonUnmarshal to unmarshal yaml after conversion. As jsonUnmarshal does not support time.Duration, Unmarshal would also not support it.

// Unmarshal converts YAML to JSON then uses JSON to unmarshal into an object,
// optionally configuring the behavior of the JSON unmarshal.
func Unmarshal(y []byte, o interface{}, opts ...JSONOpt) error {
	return yamlUnmarshal(y, o, false, opts...)
}

// yamlUnmarshal unmarshals the given YAML byte stream into the given interface,
// optionally performing the unmarshalling strictly
func yamlUnmarshal(y []byte, o interface{}, strict bool, opts ...JSONOpt) error {
	vo := reflect.ValueOf(o)
	unmarshalFn := yaml.Unmarshal
	if strict {
		unmarshalFn = yaml.UnmarshalStrict
	}
	j, err := yamlToJSON(y, &vo, unmarshalFn)
	if err != nil {
		return fmt.Errorf("error converting YAML to JSON: %v", err)
	}

	err = jsonUnmarshal(bytes.NewReader(j), o, opts...)
	if err != nil {
		return fmt.Errorf("error unmarshaling JSON: %v", err)
	}

	return nil
}

Can yamlUnmarshal be updated to use gopkg.in/yaml.v2 Unmarshal, or was there a specific reason json Unmarshal was chosen?

Marshal ignoring `omitempty` for nested structs

Related to #38

If I have a nested struct with omitempty tag, and nothing declared for that struct, I would expect the whole struct to be omitted, not just the child fields.

Example:
https://go.dev/play/p/3rSbQB9euZT
I would expect the output to be simply

b: foo


Program exited.

Int overflow on 32 bits arches

Golang 1.12.6 on i686 and armv7:

Testing    in: /builddir/build/BUILD/yaml-1.1.0/_build/src
         PATH: /builddir/build/BUILD/yaml-1.1.0/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
       GOPATH: /builddir/build/BUILD/yaml-1.1.0/_build:/usr/share/gocode
  GO111MODULE: off
      command: go test -buildmode pie -compiler gc -ldflags "-X sigs.k8s.io/yaml/version=1.1.0 -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '"
      testing: sigs.k8s.io/yaml
sigs.k8s.io/yaml
FAIL	sigs.k8s.io/yaml [build failed]
BUILDSTDERR: # sigs.k8s.io/yaml [sigs.k8s.io/yaml.test]
BUILDSTDERR: ./yaml_test.go:22:26: constant 9223372036854775807 overflows int

CVE-2022-28948: gopkg.in/yaml.v2

🐛 Bug Report

An issue in the Unmarshal function in Go-Yaml v3 causes the program to crash when attempting to deserialize invalid input.

We should bump the usages on this repo

The problem is that some assumptions of v2 do not hold on v3, like MapSlices and UnmarshallStrict, how to proceed?

Can't unmarshal multiple documents

I have a yaml output that has multiple documents that I want to unmarshal.

When I try and unmarshal into a list I get this error

JSON: while decoding JSON: json: cannot unmarshal object into Go value of type []v1beta1.CustomResourceDefinition"}
github.com/go-logr/zapr.(*zapLogger).Error
        /home/rbelgrave/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
main.main
        /home/rbelgrave/projects/github.com/rmb938/hostport-allocator/tools/crd-helm/main.go:32
runtime.main
        /home/linuxbrew/.linuxbrew/Cellar/go/1.15/libexec/src/runtime/proc.go:204
exit status 1

Related Vulnerabilities fixes

Hi all, while testing package vulnerabilities I saw below issues:

=== Module dependency that show where its coming ===

sabuhigr:home/sabuhigr$ go mod graph | grep sigs
github.com/swaggo/[email protected] sigs.k8s.io/[email protected]

Vulnerabilitiy 1:
sigs.k8s.io/yaml:1.3.0 | Reference: CVE-2022-3064 | CVSS Score: 7.5 | Category: CWE-400 | Parsing malicious or large YAML documents can consume excessive amounts of CPU or memory.

Vulnerability 2:
Reference: CVE-2023-2251 | CVSS Score: 7.5 | Category: CWE-248 | Uncaught Exception in GitHub repository eemeli/yaml prior to 2.0.0-5.

Should this library use json-iterator?

Kubernetes uses github.com/json-iterator/go instead of encoding/json, should this library switch also?

package has no decoder function to read from streams

Hi, I am wondering why this package doesn't provide a decoder to read from streams. Would it be feasible to implement this?

Marshal doesn't respect struct tags?

Hi, when reading the docs I thought that this library is a drop-in replacement for the go-yaml with the benefit that we don't need to maintain multiple tags and marshalers. I faced with the following issues when using yaml.Marshal

Property keys with the name "on" are parsed as boolean even if a struct tag json:"on" exists.
Empty values aren't omitted when using json:"omitempty"
The order of my fields in the resulting YAML is broken.

From my understanding, all information gets lost here. How do you deal with that?

Broken Link Issue

Bug Report

Problem
This page has one broken link of contributor cheat sheet which needs to fix.

solution
need to locate the correct path of README.

/assign
/kind bug
/kind documentation

Difference with gopkg.in/yaml.v2 when value is passed instead of a reference

It seems that when you pass a struct by value to Unmarshal, it silently fails to decode the input data. That seems to be a different and undocumented behavior from gopkg.in/yaml.v2, which works both when you pass by value and by reference.

This might be confusing for users, when one changes the import from gopkg.in/yaml.v2 to sigs.k8s.io/yaml and expects that basic things will work the same way.

Reproduce case: https://play.golang.org/p/ucPJWh04AGH

CC @mauriciovasquezbernal

yaml.UnmarshalStrict() does not respect `json: alias` fields

this libarary's yaml.UnmarshalStrict() for a YAML file that contains a KubeletConfig fails for this field:
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubelet/config/v1beta1/types.go#L473

if the YAML document uses resolverConfig it works, but if it uses resolvConf it fails and the JSON alias is not respected.

/kind bug
/priority critical-urgent

go.mod module name is wrong, should be: sigs.k8s.io/yaml

go: error loading module requirements
go: sigs.k8s.io/yaml@v0.0.0-20180919145318-4e761d0940e599a9569b441f02025fc5981ec2a5: parsing go.mod: unexpected module path "github.com/kubernetes-sigs/yaml"

Keys with underscores are not parsed correctly.

This library does not parse yaml keys with underscores correctly. gopkg.in/yaml.v2 and v3 parse the keys correctly.

package main

import (
    "log"
    "fmt"
    k8s "sigs.k8s.io/yaml"
    yamlv3 "gopkg.in/yaml.v3"
    yamlv2 "gopkg.in/yaml.v2"
)

type Config struct {
    Normal             string `yaml:"normal"`
    OneUnderscore      string `yaml:"one_underscore"`
    TwoUnderscores     string `yaml:"two_under_scores"`
    OneUnderscoreInt   int    `yaml:"one_underscoreint"`
    TwoUnderscoresInt  int    `yaml:"two_under_scoresint"`
}

func main() {
    yamlData := `
---
normal: test
one_underscore: test1
two_under_scores: test2
one_underscoreint: 1
two_under_scoresint: 2
`
    var config Config
    err := k8s.Unmarshal([]byte(yamlData), &config)
    if err != nil {
        log.Fatalf("error: %v", err)
    }
    fmt.Printf("k8s - normal: %s\n", config.Normal)
    fmt.Printf("k8s - one_underscore: %s\n", config.OneUnderscore)
    fmt.Printf("k8s - two_under_scores: %s\n", config.TwoUnderscores)
    fmt.Printf("k8s - one_underscoreint: %d\n", config.OneUnderscoreInt)
    fmt.Printf("k8s - two_under_scoresint: %d\n", config.TwoUnderscoresInt)


    var config2 Config
    err = yamlv2.Unmarshal([]byte(yamlData), &config2)
    if err != nil {
        log.Fatalf("error: %v", err)
    }
    fmt.Printf("yamlv2 - normal: %s\n", config2.Normal)
    fmt.Printf("yamlv2 - one_underscore: %s\n", config2.OneUnderscore)
    fmt.Printf("yamlv2 - two_under_scores: %s\n", config2.TwoUnderscores)
    fmt.Printf("yamlv2 - one_underscoreint: %d\n", config2.OneUnderscoreInt)
    fmt.Printf("yamlv2 - two_under_scoresint: %d\n", config2.TwoUnderscoresInt)

    var config3 Config
    err = yamlv3.Unmarshal([]byte(yamlData), &config3)
    if err != nil {
        log.Fatalf("error: %v", err)
    }
    fmt.Printf("yamlv3 - normal: %s\n", config3.Normal)
    fmt.Printf("yamlv3 - one_underscore: %s\n", config3.OneUnderscore)
    fmt.Printf("yamlv3 - two_under_scores: %s\n", config3.TwoUnderscores)
    fmt.Printf("yamlv3 - one_underscoreint: %d\n", config3.OneUnderscoreInt)
    fmt.Printf("yamlv3 - two_under_scoresint: %d\n", config3.TwoUnderscoresInt)
}

Output:

k8s - normal: test
k8s - one_underscore:
k8s - two_under_scores:
k8s - one_underscoreint: 0
k8s - two_under_scoresint: 0
yamlv2 - normal: test
yamlv2 - one_underscore: test1
yamlv2 - two_under_scores: test2
yamlv2 - one_underscoreint: 1
yamlv2 - two_under_scoresint: 2
yamlv3 - normal: test
yamlv3 - one_underscore: test1
yamlv3 - two_under_scores: test2
yamlv3 - one_underscoreint: 1
yamlv3 - two_under_scoresint: 2

Roundtrip unmarshal/marshal with omitempty failed to restore original yaml

As illustrated in #55:

With struct defined as

type RoundTripTest struct {
	A []string `json:"a,omitempty"`
}

and provided yaml:

a: []

Unmarshal got:

RoundTripTest{A: []string{}}

So far things went expected.

But when marshal the unmarshalled struct and got:

{}

instead of a: []

And of course unmarshal again and got:

RoundTripTest{A: nil}

Feel like a bug but not sure. Is it intended behavior for omitempty?

Question: How to parse multiple yaml resource in a single file

Hi team,

I have a single yaml file let us suppose config.yml and it has multiple yaml configuration separated by --- (three dashes). I'm using YAMLToJSON function to convert it to json but it is not returning 2nd resource config.

yaml/yaml.go

Line 111 in 9535b3b

func YAMLToJSON(y []byte) ([]byte, error) {

Does anyone how to do it?
Thanks

yaml marshal sequences indent style

Hello, when I use this package to marshal an object, the format of sequences is:

containers:
- name: hello
  image: alpine

But seems the suggested format in yaml spec site is:

containers:
  - name: hello
    image: alpine

I know both of them are fine, but I think it is better to use a widely-used style.

https://yaml.org/spec/1.2/#id2797382

yaml.Unmarshal HTML escapes input when decoding to json.RawMessage

I'm using this package via apimachinery's yaml.NewYAMLOrJSONDecoder, and I'm noticing that the encoding of the input is not preserved.

Specifically, when YAML is unmarshaled, it is converted to JSON with json.Marshal, which hardcodes use of HTML escaping.

It is easy to isolate this behavior when unmarshaling to json.RawMessage, which is what I'm using.

Here is a reproducer:

package main

import (
	"encoding/json"
	"fmt"
	"os"

	"sigs.k8s.io/yaml"
)

func main() {
	msg := json.RawMessage(`"A < B"`)
	enc := json.NewEncoder(os.Stdout)
	enc.SetEscapeHTML(false)

	fmt.Print("Using unescaped message: ")
	if err := enc.Encode(msg); err != nil {
		panic(err)
	}

	var viaYAML json.RawMessage
	if err := yaml.Unmarshal(msg, &viaYAML); err != nil {
		panic(err)
	}
	fmt.Print("Using message via yaml.Unmarshal: ")
	if err := enc.Encode(viaYAML); err != nil {
		panic(err)
	}
}

Output:

Using unescaped message: "A < B"
Using message via yaml.Unmarshal: "A \u003c B"

I think this is unintentional behavior, but it is not clear from my reading of the code. It seems like a simple fix could be made here, where we could use an encoder configured specifically to disable HTML escaping rather than using json.Marshal.

Bump version of go-yaml to 2.3.0

go-yaml version 2.3.0 fixed long line wrapping.
go-yaml/yaml#571

Please bump to this version

feature request: preverse property order

Hi, I know why the order is not preserved. It's because of the underlying implementation of the go package. However, given that many Kubernetes tools rely on this yaml package, and the output is typically read by humans, I think it would be nice to add an option to preserve the order.

In this issue are some ideas and even concrete implementations on how to achieve this. Something could be done based on those ideas.

As an example use case, when using kustomize, the yaml manifests get all messed up, which is ok when applying it directly to the cluster. But It's not really ok anymore when source controlling this for gitops purposes. It becomes almost unreadable and create huge diffs.

I tried creating a customer formatter that could be run on top of anything that is produced by this package, but it's very hard to get right because this yaml package has so magic handling of Kubernetes objects, which cannot be achieved easily but using other standard packages. For example, decoding a pod into a pod struct and then using the standard yaml package to get conventional ordering does not result in something usable, since it does not implement the special handling for Kubernetes objects.

After all, I think the right place to try to preserve ordering or get conventional ordering, would be this yaml package here.

yaml parsing bug

The following program produces inconsistent yaml parsing results.

package main

import (
	"fmt"

	"google.golang.org/protobuf/types/known/structpb"
	"sigs.k8s.io/yaml"
)

type individualTest struct {
	Context *structpb.Struct
}

type checkTests struct {
	Tests []individualTest
}

func main() {
	yamlInput := `
  tests:
  - context:
       x: 10
       y: 5
  - context:
      "x": 10
      "y": 5`

	var testCases checkTests
	if err := yaml.Unmarshal([]byte(yamlInput), &testCases); err != nil {
		panic(err)
	}

	for _, testCase := range testCases.Tests {
	        fmt.Printf("%v\n", testCase.Context)
	}
}

Output:

fields:{key:"true"  value:{number_value:5}}  fields:{key:"x"  value:{number_value:10}}
fields:{key:"x"  value:{number_value:10}}  fields:{key:"y"  value:{number_value:5}}

Notice that the y field (first output line) gets marshalled as a bool true 🤔

Expected Output:

fields:{key:"y"  value:{number_value:5}}  fields:{key:"x"  value:{number_value:10}}
fields:{key:"x"  value:{number_value:10}}  fields:{key:"y"  value:{number_value:5}}

Move from travis to github actions

Travis is going away, so we need to move out CI jobs from travis to github actions
https://github.com/kubernetes-sigs/yaml/blob/master/.travis.yml

There are lots of other k8s projects that have already moved, so would be easy to pick one of them to emulate:
https://cs.k8s.io/?q=jobs%3A&i=nope&files=%5C.github.*&repos=

example the one from klog:
https://github.com/kubernetes/klog/tree/master/.github/workflows

Avoid hardcoding golang versions in github-actions CI, if possible

Very softly stepping here as I dont know the head or tail of the appcode.

We are being explicit with the version of go in the github-actions ci on line 12 here https://github.com/kubernetes-sigs/yaml/edit/master/.github/workflows/go.yml ;

strategy:
      matrix: 
        go-versions: [1.13.x, 1.14.x, 1.15.x]
    runs-on: ubuntu-latest

This implies a manual task being created with each new release of golang, that the developers start using.

This issue is just to track the above mentioned change in golang releases and once for all automate the choice of go versions during CI, Alternatively we could explicitly decide to manually maintain the golang version in github-action CI, so as to maintain fine grained control on the big picture.

Unmarshal loses type information

When unmarshalling a YAML file to a map[string]interface{}, all integer values are converted to float64 values instead, despite the fact that both YAML and Go support integer values. This is presumably due to the way the package goes from YAML to JSON before finally unmarshalling to "real" Go types, with JSON only having float numeric types.

For an example of this problem in action (and a comparison to gopkg.in/yaml.v2, which does not exhibit this issue) see https://play.golang.org/p/YfO69Z3vi-V.

This is particularly notable when Helm is unmarshalling values.yaml. For an example of that, see helm/helm#8978 (comment)

an error should be thrown for unknown fields

sample code:

package main

import (
	"fmt"

	"sigs.k8s.io/yaml"
)

var data = `
a: test
b: unknown
`

type T struct {
	A string
	// B string // TEST: B is unknown field
}

func main() {
	t := T{}
	fmt.Println("unmarshaling...")
	err := yaml.UnmarshalStrict([]byte(data), &t)
	if err != nil {
		fmt.Printf("error: %v\n", err)
		return
	}
	fmt.Println("done")
}

output is done instead of error for b being unknown.

/kind bug
cc @dims

Bug in anchor handling with UnmarshalStrict: key already set in map

When trying to read this yaml file:

arrayOfThings:
  - &shared
    name: thing-one
    attributeOne: foo
    attributeTwo: bar
  - <<: *shared
    name: thing-two

into these structs:

type YamlToGo struct {
    ArrayOfThings []Thing `yaml:"arrayOfThings,omitempty"`
}

type Thing struct {
    Name         string `yaml:"name,omitempty"`
    AttributeOne string `yaml:"attributeOne,omitempty"`
    AttributeTwo string `yaml:"attributeTwo,omitempty"`
}

with yaml.UnmarshalStrict, the following error appears:

error converting YAML to JSON: yaml: unmarshal errors:
  line 8: key "name" already set in map
exit status 1

Thanks to @Callisto13 for creating this reproducing example.

It seems that using either the non-strict version of Unmarshal or gopkg.in/yaml.v2 get around this error, but neither is really an acceptable thing to do.

Handling blank first line when using YAMLOrJSONDecoder

When the first line of a yaml document is blank, the decoding will not work properly. By "not properly" I mean that it does not throw an error, but it also does not decode into the desired object.

Example:

crdv1 := &extv1.CustomResourceDefinition{} // extv1 is k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1
// for crd I used https://github.com/kubermatic/kubermatic/blob/36282c9f0bda74b2543be05a3317201cf0604592/charts/kubermatic-operator/crd/k8c.io/kubermatic.k8c.io_clusters.yaml
// but I think anything that starts with a blank line, causes the issue
dec := yaml.NewYAMLOrJSONDecoder(crd, 4096)
err := dec.Decode(crdv1) // will not throw an error, but also will not decode the object into it

Normally I would not mind too much and just remove the blank line. However controller-gen generates all of its CRDs to start with a blank line, which makes this a bit troublesome.

Please release fix for CVE-2022-28948

Good morning,

It appears https://pkg.go.dev/sigs.k8s.io/[email protected] does not contain a fix for CVE-2022-28948.
According to go-yaml/yaml#666, a fix has been generated for CVE-2022-28948.

Is it possible then to please release a patch that fixes this CVE within the upcoming weeks?

Expose a function from sigs.k8s.io/yaml that allows for strict unmarshaling to distinguish between strict and nonstrict errors.

Currently, the kjson library exposes an UnmarshalStrict function that returns strict errors and nonstrict errors separately (strict errors are those relating to unknown fields that are only erroneous when strict unmarshaling is desired and still allow marshaling to occur, while nonstrict errors are any other errors that are always fatal to the marshaling).

We want to expose a similar function from kyaml (sigs.k8s.io/yaml) so that the strict unmarshaling that occurs higher up in the stack does not need to unmarshaling does not need to be performed twice.

For example in the applyPatcher of the patch handler, when we want to perform a strict unmarshal, we first do a regular Unmarshal, confirm that there are no errors, and then perform a second StrictUnmarshal to check for any additional errors that must be strict errors.

A better approach is to just do a single UnmarshalStrict and check the non-strict errors to return fatally (if non-nil) like how we do for kjson.

json quoting error in YAMLToJSON

I found this issue whilst investigating kubernetes-sigs/kustomize#843. I'm going to do a little digging around here and see if I can get to a root cause.

func main() {

	yamlFile := []byte(`
  literals:
  - 'v2=[{"path": "var/druid/segment-cache"}]'
  - v3="[{\"path\": \"var/druid/segment-cache\"}]"
`)

	y, err := yaml.YAMLToJSON(yamlFile)

	if( err != nil) {
		fmt.Println("Cannot Parse JSON")
		os.Exit(-1)
	}
	fmt.Print(string(y))
}

The output is:

{
  "literals": [
    "v2=[{\"path\": \"var/druid/segment-cache\"}]",
    {
      "v3=\"[{\\\"path\\\"": "\\\"var/druid/segment-cache\\\"}]\""
    }
  ]
}

So yaml.YAMLToJSON is erroneously putting extra {} arround the 2nd example.

Long strings are marshalled to invalid YAML; incorrect line breaks inserted

When serializing an object that contains a very long string in its value, the resulting YAML has line breaks seemingly at random. This seems to be a bug in gopkg.in/yaml.v2 that is fixed in gopkg.in/yaml.v3, but discussions in #18 and #61 seem to imply that migrating is nontrivial.

To reproduce, put the following in a file repro_test.go:


import (
	"fmt"
	"testing"

	yamlv2 "gopkg.in/yaml.v2"
	yamlv3 "gopkg.in/yaml.v3"
	yamlk8s "sigs.k8s.io/yaml"

	"gotest.tools/assert"
)

func TestYamlV2(t *testing.T) {
	runTest(t, yamlv2.Marshal)
}

func TestYamlV3(t *testing.T) {
	runTest(t, yamlv3.Marshal)
}

func TestYamlK8s(t *testing.T) {
	runTest(t, yamlk8s.Marshal)
}

func runTest(t *testing.T, marshaller func(interface{}) ([]byte, error)) {
	long_string := "a very long string that might have to be broken into multiple lines in order to fit nicely in the output pane"
	j := map[string]string{
		"foo": long_string,
	}
	y, err := marshaller(j)
	if err != nil {
		t.Fatalf("failed converting: %v", err)
	}

	assert.Equal(t, fmt.Sprintf("foo: %s\n", long_string), string(y))
}

then run go mod init; go mod tidy; go test ./... to see the v2 and k8s tests fail, but not v3:

--- FAIL: TestYamlV2 (0.00s)
    repro_test.go:36: assertion failed: 
        --- ←
        +++ →
        @@ -1,2 +1,3 @@
        -foo:·a·very·long·string·that·might·have·to·be·broken·into·multiple·lines·in·order·to·fit·nicely·in·the·output·pane
        +foo:·a·very·long·string·that·might·have·to·be·broken·into·multiple·lines·in·order
        +··to·fit·nicely·in·the·output·pane
         
        
--- FAIL: TestYamlK8s (0.00s)
    repro_test.go:36: assertion failed: 
        --- ←
        +++ →
        @@ -1,2 +1,3 @@
        -foo:·a·very·long·string·that·might·have·to·be·broken·into·multiple·lines·in·order·to·fit·nicely·in·the·output·pane
        +foo:·a·very·long·string·that·might·have·to·be·broken·into·multiple·lines·in·order
        +··to·fit·nicely·in·the·output·pane

Test failed on arm64

$ GOARCH=arm64 go test -v ./ -run TestJSONObjectToYAMLObject
=== RUN   TestJSONObjectToYAMLObject
=== RUN   TestJSONObjectToYAMLObject/nil
=== RUN   TestJSONObjectToYAMLObject/empty
=== RUN   TestJSONObjectToYAMLObject/values
--- FAIL: TestJSONObjectToYAMLObject (0.04s)
    --- PASS: TestJSONObjectToYAMLObject/nil (0.01s)
    --- PASS: TestJSONObjectToYAMLObject/empty (0.00s)
    --- FAIL: TestJSONObjectToYAMLObject/values (0.02s)
        yaml_test.go:489: jsonToYAML() = (yaml.MapSlice) (len=15 cap=15) {
             (yaml.MapItem) {
              Key: (string) (len=4) "bool",
              Value: (bool) true
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "empty map",
              Value: (yaml.MapSlice) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=11) "empty slice",
              Value: ([]interface {}) {
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=7) "float64",
              Value: (float64) 42.1
             },
             (yaml.MapItem) {
              Key: (string) (len=12) "fractionless",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=3) "int",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=5) "int64",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "int64 big",
              Value: (int) 4611686018427387904
             },
             (yaml.MapItem) {
              Key: (string) (len=3) "map",
              Value: (yaml.MapSlice) (len=1 cap=1) {
               (yaml.MapItem) {
                Key: (string) (len=3) "foo",
                Value: (string) (len=3) "bar"
               }
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=18) "negative int64 big",
              Value: (int) -4611686018427387904
             },
             (yaml.MapItem) {
              Key: (string) (len=7) "nil map",
              Value: (interface {}) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "nil slice",
              Value: (interface {}) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=5) "slice",
              Value: ([]interface {}) (len=2 cap=2) {
               (string) (len=3) "foo",
               (string) (len=3) "bar"
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=6) "string",
              Value: (string) (len=3) "foo"
             },
             (yaml.MapItem) {
              Key: (string) (len=10) "uint64 big",
              Value: (int) 9223372036854775807
             }
            }
            , want (yaml.MapSlice) (len=15 cap=15) {
             (yaml.MapItem) {
              Key: (string) (len=4) "bool",
              Value: (bool) true
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "empty map",
              Value: (yaml.MapSlice) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=11) "empty slice",
              Value: ([]interface {}) {
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=7) "float64",
              Value: (float64) 42.1
             },
             (yaml.MapItem) {
              Key: (string) (len=12) "fractionless",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=3) "int",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=5) "int64",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "int64 big",
              Value: (int) 4611686018427387904
             },
             (yaml.MapItem) {
              Key: (string) (len=3) "map",
              Value: (yaml.MapSlice) (len=1 cap=1) {
               (yaml.MapItem) {
                Key: (string) (len=3) "foo",
                Value: (string) (len=3) "bar"
               }
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=18) "negative int64 big",
              Value: (int) -4611686018427387904
             },
             (yaml.MapItem) {
              Key: (string) (len=7) "nil map",
              Value: (interface {}) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "nil slice",
              Value: (interface {}) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=5) "slice",
              Value: ([]interface {}) (len=2 cap=2) {
               (string) (len=3) "foo",
               (string) (len=3) "bar"
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=6) "string",
              Value: (string) (len=3) "foo"
             },
             (yaml.MapItem) {
              Key: (string) (len=10) "uint64 big",
              Value: (uint64) 9223372036854775808
             }
            }
        yaml_test.go:528: yaml.Unmarshal(json.Marshal(tt.input)) = (yaml.MapSlice) (len=15 cap=16) {
             (yaml.MapItem) {
              Key: (string) (len=4) "bool",
              Value: (bool) true
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "empty map",
              Value: (yaml.MapSlice) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=11) "empty slice",
              Value: ([]interface {}) {
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=7) "float64",
              Value: (float64) 42.1
             },
             (yaml.MapItem) {
              Key: (string) (len=12) "fractionless",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=3) "int",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=5) "int64",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "int64 big",
              Value: (int) 4611686018427388000
             },
             (yaml.MapItem) {
              Key: (string) (len=3) "map",
              Value: (yaml.MapSlice) (len=1 cap=1) {
               (yaml.MapItem) {
                Key: (string) (len=3) "foo",
                Value: (string) (len=3) "bar"
               }
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=18) "negative int64 big",
              Value: (int) -4611686018427388000
             },
             (yaml.MapItem) {
              Key: (string) (len=7) "nil map",
              Value: (interface {}) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "nil slice",
              Value: (interface {}) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=5) "slice",
              Value: ([]interface {}) (len=2 cap=2) {
               (string) (len=3) "foo",
               (string) (len=3) "bar"
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=6) "string",
              Value: (string) (len=3) "foo"
             },
             (yaml.MapItem) {
              Key: (string) (len=10) "uint64 big",
              Value: (uint64) 9223372036854776000
             }
            }
            , want (yaml.MapSlice) (len=15 cap=15) {
             (yaml.MapItem) {
              Key: (string) (len=4) "bool",
              Value: (bool) true
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "empty map",
              Value: (yaml.MapSlice) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=11) "empty slice",
              Value: ([]interface {}) {
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=7) "float64",
              Value: (float64) 42.1
             },
             (yaml.MapItem) {
              Key: (string) (len=12) "fractionless",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=3) "int",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=5) "int64",
              Value: (int) 42
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "int64 big",
              Value: (int) 4611686018427388000
             },
             (yaml.MapItem) {
              Key: (string) (len=3) "map",
              Value: (yaml.MapSlice) (len=1 cap=1) {
               (yaml.MapItem) {
                Key: (string) (len=3) "foo",
                Value: (string) (len=3) "bar"
               }
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=18) "negative int64 big",
              Value: (int) -4611686018427388000
             },
             (yaml.MapItem) {
              Key: (string) (len=7) "nil map",
              Value: (interface {}) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=9) "nil slice",
              Value: (interface {}) <nil>
             },
             (yaml.MapItem) {
              Key: (string) (len=5) "slice",
              Value: ([]interface {}) (len=2 cap=2) {
               (string) (len=3) "foo",
               (string) (len=3) "bar"
              }
             },
             (yaml.MapItem) {
              Key: (string) (len=6) "string",
              Value: (string) (len=3) "foo"
             },
             (yaml.MapItem) {
              Key: (string) (len=10) "uint64 big",
              Value: (int) -9223372036854775000
             }
            }
            
            json: {"bool":true,"empty map":{},"empty slice":[],"float64":42.1,"fractionless":42,"int":42,"int64":42,"int64 big":4611686018427388000,"map":{"foo":"bar"},"negative int64 big":-4611686018427388000,"nil map":null,"nil slice":null,"slice":["foo","bar"],"string":"foo","uint64 big":9223372036854776000}
FAIL
FAIL    sigs.k8s.io/yaml        0.104s
FAIL

The diff is

              },
              (yaml.MapItem) {
               Key: (string) (len=10) "uint64 big",
-              Value: (int) 9223372036854775807
+              Value: (uint64) 9223372036854775808
              }
             }

and

-(yaml.MapSlice) (len=15 cap=16) {
+(yaml.MapSlice) (len=15 cap=15) {
              (yaml.MapItem) {
               Key: (string) (len=4) "bool",
               Value: (bool) true
@@ -66,6 +66,7 @@
              },
              (yaml.MapItem) {
               Key: (string) (len=10) "uint64 big",
-              Value: (uint64) 9223372036854776000
+              Value: (int) -9223372036854775000
              }
             }

scalars containing colons not properly handled in flow style

I recently ran into an issue with YAML parsing of the IPv6 localhost address ("::1") in kubeadm. I opened kubernetes/kubeadm#2858 but they suggested I should create an issue here.

I've also opened issue go-yaml/yaml#956 against the go-yaml parser which I think would address the underlying problem.

If I try to parse the following where the contents of the list are treated as strings, it handles it okay as expected.

d: [1::1, 4.1.1.1]

However, if I try to parse this

d: [::1, 4.1.1.1]

then it gives an error "error: yaml: line 4: did not find expected node content"

In practice this means that Go can't handle the IPv6 localhost address in flow style unless it's quoted. It works fine in block style:

d:
- ::1
- 4.1.1.1

According to the YAML devs, the string "::1" should be a valid scalar because the colon is followed by a non-space character.

The YAML devs pointed me at the tool they use for testing, here are 17 different parsers handling the above:

https://play.yaml.io/main/parser?input=ZDogWzo6MSwgNC4xLjEuMV0=

Here's a Go playground example showing the problem: https://go.dev/play/p/Bg3zzizTwHu

Richer typed errors

encoding/json has various error types that Unmarshal can return such as

type UnmarshalTypeError struct {
	Value  string       // description of JSON value - "bool", "array", "number -5"
	Type   reflect.Type // type of Go value it could not be assigned to
	Offset int64        // error occurred after reading Offset bytes
	Struct string       // name of the struct type containing the field
	Field  string       // the full path from root node to the field
}

This contains lots of useful information that could be used when debugging yaml.Unmarshal errors. From what I can tell yaml.Unmarshal currently discards the rich error type returned by json.Unmarshal.

Could this package produce better errors along the lines of those used by encoding/json?

I see #9 is very similar but I'm not sure why that was closed, so I'm creating a new issue to track/discuss this.

Unexpected un-marshal behavior of Bool which is different from its wrapped package

Package sigs.k8s.io/yaml v1.3.0 is a wrapper around gopkg.in/yaml.v2 v2.4.0, but sigs.k8s.io/yaml v1.3.0 has unexpected un-marshal behavior of bool, which is different from the behavior of gopkg.in/yaml.v2 v2.4.0

Package version recorded in go.mod:

require gopkg.in/yaml.v2 v2.4.0

require sigs.k8s.io/yaml v1.3.0

Test function in yaml_test.yaml:

package yaml

import (
	"fmt"
	yamlv2 "gopkg.in/yaml.v2"
	yamlk8s "sigs.k8s.io/yaml"
	"testing"
)

func TestYAML(t *testing.T) {
	y := `test: off`
	a := make(map[string]string)
	var err error
	err = yamlk8s.Unmarshal([]byte(y), &a)
	if err != nil {
		t.Fatalf(err.Error())
	}
	fmt.Println("sigs.k8s.io/yaml: ", a)
	err = yamlv2.Unmarshal([]byte(y), &a)
	if err != nil {
		t.Fatalf(err.Error())
	}
	fmt.Println("gopkg.in/yaml.v2: ", a)
}

Test result:

=== RUN   TestYAML
sigs.k8s.io/yaml:  map[test:false]
gopkg.in/yaml.v2:  map[test:off]

Notice that value off is un-marshaled to false by package sigs.k8s.io/yaml, which is unexpected and different from the behavior of its wrapped package gopkg.in/yaml.v2

Time Duration (un)marshalling only translates to nanoseconds

Reading a yaml from a file (or copying config to a file) only translates to nano seconds, while the type of the param is of Time.Duration.

		Agent: &agentConfig{
			Server:               "https://localhost:3333",
			EnrollmentUi:         "",
			TpmPath:              "",
			FetchSpecInterval:    1 * time.Minute,
			StatusUpdateInterval: 1 * time.Minute,
		},

the last two params always get unmarshalled to this:

agent:
  fetchSpecInterval: 60000000000
  server: https://localhost:3333
  statusUpdateInterval: 60000000000

Using "gopkg.in/yaml.v3" allows to have notation such as 1m30s, while this one doesn't.

Is there any reason for that?
Thank you very much in advance.

Probable Bug - YAMLToJSON - double quotes (empty string) are converted to null

when doing a YAMLToJSON on a map[string]string (in my case it was the nodeSelector of a DaemonSet) and using annotation of the form: node-role.kubernetes.io/worker: "" the result is: {"node-role.kubernetes.io/worker":null}.
I believe this is a mistake and that it should be: {"node-role.kubernetes.io/worker":""}.
This behavior caused issues for me as the resulting object could not validated as annotation values cannot be nil/null.
I was not able to fund a workaround.

improve error reporting JSON vs YAML

the errors of this library need work.
with the strict JSON decoder unknown fields would result in errors such as:

error unmarshaling JSON: .....: json: unknown field "z";

this gives a hint that "z" is unknown.
but the user would first of all be confused why is the error throw to stdout about JSON, while their e.g. manifest is in YAML.

xref: #7

Bug: string fields parsed incorrectly in structs nested with `json:",inline"`

When you have a struct nested with json:",inline" that has a string field, attempting to unmarshal an unquoted number into that string field will fail with cannot unmarshal number into Go struct field [...] of type string. This is a breakage with the YAML spec.

The following minimal example illustrates the problem: https://go.dev/play/p/enZcSLWe9jo

As you can see, unmarshaling Foo succeeds, but Bar fails with:

failed to unmarshal bar:  error unmarshaling JSON: while decoding JSON: json: cannot unmarshal number into Go struct field Bar.fooStr of type string

Changing the spec to quote the fooStr field makes unmarshaling succeed for both: https://go.dev/play/p/_B5QaRpxiPz

Should this library move to yaml.v3

gopkg.in/go-yaml.v3 is already available, should we upgrade to it?

Difference with gopkg.in/yaml.v2 for string like numeric parsing

I had noticed that a difference between this library and gopkg.in/yaml.v2 when parsing string like numeric.
It is a sample code and output of it below.
Could you fix it to get the same result?

code:

package main

import (
	"fmt"

	yaml2 "gopkg.in/yaml.v2"
	"sigs.k8s.io/yaml"
)

type testStruct struct {
	Serial string `json:"serial,omitempty"`
}

var strYaml string = `
serial: 00000002
`

func main() {
	fmt.Println("sigs.k8s.io/yaml")

	str := []byte(strYaml)
	var c1 testStruct
	err := yaml.Unmarshal(str, &c1)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(c1)

	fmt.Println("gopkg.in/yaml.v2")
	var c2 testStruct
	err = yaml2.Unmarshal(str, &c2)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(c2)
}

output:

sigs.k8s.io/yaml
{2}
gopkg.in/yaml.v2
{00000002}

kubernetes-sigs / yaml Goto Github PK

yaml's Introduction

YAML marshaling and unmarshaling support for Go

Introduction

Compatibility

Caveats

Installation and usage

yaml's People

Contributors

Stargazers

Watchers

Forkers

yaml's Issues

🐛 Bug Report

Bug Report

Recommend Projects

Recommend Topics

Recommend Org