I took just a simple yaml and got this error:

$ cat application.yaml | yq .
Traceback (most recent call last):
File "c:\python27\lib\runpy.py", line 174, in run_module_as_main
"main", fname, loader, pkg_name)
File "c:\python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "C:\Python27\Scripts\yq.exe_main.py", line 9, in
File "c:\python27\lib\site-packages\yq_init.py", line 150, in main
parser.exit(msg.format(program_name, type(e).name, e))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 23: ordinal not in range(128)

the default yaml loader will parse ISOFormat strings and create datetime objects which doesn't jsonify well.

Example:

$ yq --version
usage: yq [-h] [--yaml-output] [--width WIDTH] jq_filter [file [file ...]]
yq: error: too few arguments
$ pip freeze | grep yq
yq==2.1.2
$ yq . sec.yaml
yq: Error while running jq: TypeError: datetime.datetime(2016, 12, 20, 22, 7, 36) is not JSON serializable.
parse error: Unfinished JSON term
$ cat sec.yaml
data:
    thing: 1
    thing2: "stuff"
    datething: 2016-12-20T22:07:36Z
    datethingstr: "2016-12-20T22:07:36Z"

this needs to override the default encoder class.

# isoformat
class DateTimeEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, datetime):
            return o.isoformat()
        return json.json.JSONEncoder.default(self, o)

# usage
json.dumps(..., cls=DateTimeEncoder)

https://bitbucket.org/ruamel/yaml is the link to the package's source code repo, it's ruamel.yaml on pypi.

ruamel.yaml is really nice because it can preserve original formatting, meaning comments, and ordering of maps, etc is preserved. If yq could support this as an optional dependency and flag that allows controlling usage, that would be great. I'd even potentially look into implementing it if it doesn't seem too hard.

HTTP/1.1 404 Not Found

After changes made to passing positional parameters to jq, it looks like --from-file filter is broken and the filter for jq must be defined as the jq_filter of yq. This means you can no longer define filters in a file, which is invaluable for more complex constructs.

Example Setup

Define a filter and some yaml data

[yq] cat filter
.foo
[yq] cat data
foo
  a: 1

2.3.4

Positional arguments are broken

[yq] yq -y --arg a b . data
usage: yq [-h] [--yaml-output] [--width WIDTH] [--version]
          jq_filter [files [files ...]]
yq: error: argument files: can't open 'b': [Errno 2] No such file or directory: 'b'

--from-file without jq_filter works

[yq] yq -y --from-file filter data
a: 1

2.3.5

After changes to positional parameters, the --args example now works

[yq] yq -y --arg a b . data
foo:
  a: 1

...but the --from-file example is broken, returning the --help text with a return code 0.

[yq] yq -y --from-file filter data
usage: yq [options] <jq filter> [YAML file...]
          jq_filter [files [files ...]]

yq: Command-line YAML processor - jq wrapper for YAML documents
...
[yq] echo $?
0

trying to define the call with the self identify filter still fails to work, returning no error message but a return code of 2.

[yq] yq -y --from-file filter . data
[yq] echo $?
2

This file:

foo: bar
# This is commented out - baz: bazzz

when passed through yq w test.yaml foo newValue, returns:

foo: newValue

One of the main advantages of using YAML over JSON is to be able to put comments into the file. This functionality is lost with current version of yq.

Documentation in the main readme suggests using a similar syntax to JQ, but the basic example doesn't seem to work:

cat input.yml | yq .foo.bar

On OSX:

$ brew install yq
$ cat > input.yml
foo:
 - bar: "text"
   value: 200
baz:
 - bar: "blah"
   value: 100
^D
$ cat ./input.yml | yq .foo.bar
Error: unknown command ".foo.bar" for "yq"
Run 'yq --help' for usage.
2019/04/12 16:18:29 unknown command ".foo.bar" for "yq"

jq has a homebrew package. It would be good to have one for yq as well.

Is available in a branch (https://github.com/kislyuk/yq/tree/akislyuk-jmespath-support), but I'm not sure if anyone wants it. Please comment here if you have a use for it.

Hi!
I am finding yq very useful, but there is one thing that would make it better.
YAML is a superset of JSON, so I imagined that yq's input format would be a superset of jq's input format. However, this turns out not to be true because yq only parses a single YAML object and raises an error if there are multiple objects as input.

For example, if I run this command:

echo \"Hi\" | jq '{ hey: . }, { ho: . }' | yq . -y

I get the following output:

yq: Error running jq: ParserError: expected '<document start>', but found '{'
  in "<stdin>", line 4, column 1.

However, I'd expect to see the following output:

- hey: Hi
- ho: Hi

Is this something you are interested in supporting?

(edit: I realise that piping jq into yq is pointless. My actual use case is that I am developing a tool which outputs data as a JSON stream, and I would like to be able to pipe its output directly into yq . -y to convert it back to YAML and optionally run some jq filters on it. I can achieve this already by piping the ouptut to jq -s '[ . ]' | yq .[] -y but it would be nice if I could just use yq . -y in place of that)

Converting json back to yaml it strips all double quotes. For strings with special characters this behavior doesn't work.

I'm using ZSH - hence I need to escape the square brackets

yq w package.yaml dependencies\[+\] "new thing"
usage: yq [-h] [--yaml-output] [--width WIDTH] [--version]
          jq_filter [files [files ...]]
yq: error: argument files: can't open 'dependencies[+]': [Errno 2] No such file or directory: 'dependencies[+]'

package.yaml

name:                app
version:             0.1.0.0
license:             BSD3
license-file:        LICENSE
author:              CabalSaneDefault
maintainer:          nobody
build-type:          Simple
extra-source-files:  ChangeLog.md

dependencies:
  - base >=4.10 && <4.11
  - optparse-applicative
  - process

executable:
  main:             Main.hs
  source-dirs: src

JSON should be supported as an input format. My project(s) use YAML internally, but a lot of the external data they source comes in from JSON formats. Considering that it gets processed with yq anyway and yq is making a round trip to JSON, it seems kind of silly that there isn't an --json-input or similar flag to bypass the first half of the round trip. This would allow yq to work as a JSON→YAML conversion tool, reducing my project dependencies.

They all test against "" and pass, but that's because stdout is not actually being captured in the BytesIO/StringIO - this test fails:

self.assertEqual(self.run_yq("foo", ["."]), "\"foo\"")

It was my assumption that slurping multiple inputs should work the same as in jq, but it does not. For example this works:

echo '{ "one": 1 }\n{ "two": 2 }' | jq -s .

But this fails to parse past the first object:

echo '{ "one": 1 }\n{ "two": 2 }' | yq -s .

You can workaround using jq as an intermediary to slurp into array, then pass to yq and strip off the extra array:

echo '{ "one": 1 }\n{ "two": 2 }' | jq -s . | yq -s .[0]

Hi

I've found strange bug where in some scenarios, valid JSON isn't converted back to valid YAML. My use case was to replace a yaml block in one file with a yaml block from another file.
My command was like this:

yq -y ".spec.jobTemplate.spec.template = $(yq '.spec.template' file_b.yaml)" file_a.yaml

Without the -y flag I get valid JSON but if I add the -y flag to output yaml, a new line is being inserted where it shouldn't be causing broken YAML.

Here is a snippet of the outputted JSON:

"args": [
  "/bin/bash",
  "-c",
  "cp -v -r -L /etc/gitlab/.s3cfg $HOME/.s3cfg && while sleep 3600; do :; done"
],

And when I add the -y flag I get this YAML:

args:
- /bin/bash
- -c
- cp -v -r -L /etc/gitlab/.s3cfg $HOME/.s3cfg && while sleep 3600; do
  :; done

It's added a new line inside that last array element.

So I have an ansible playbook which has tags like this:

some_key: !vault |
  $ANSIBLE_VAULT;1.1;AES256
  37663434363236326231303035313330356563343734393664373764333962663539623031346531
  6335383361386661613563386265326264613738353632310a633162393366636362633835633430
  34313038633233646432646335643336643234613133643734303530616137366132313038343035
  3039313262386130350a626364633935643562376366306631626530663037376630613565373164
  6261

When trying to extract this, to pipe it into a decrypt:

$ yq '.some_key' file.yml
null

It would be more useful to just ignore tags that aren't recognised and pull the value out as-is, rather than just nulling the value out entirely. It might be a trivial to expose tags separately if you need them?

$ yq --tags '.some_key' file.yml
[
  "vault"
]

Been trying the following:

cat file.yml | yq --arg foo bar '.metadata.name = $foo'

And I get the following error:

yq: error: argument files: can't open 'bar': [Errno 2] No such file or directory: 'bar'

It would seem that jq's expected syntax for setting args with a space between key and value is tripping things up.

I am packaging yq for availability through conda-forge and I noticed that the LICENSE file is not being packed in the source distribution.

Can you add the LICENSE in your next release of YQ?

Seeing xq and tqI was wondering if you would be interested in implementing hq, an HTML parser, as part of your tool.

I did see https://github.com/jeffbr13/xq and https://github.com/rbwinslow/hq, but it looks they aren't being maintained anymore.

If I try to use --slurp with yq, it gets incorrectly translated into --slurpfile. I am using Python 3.5.3.

> touch 2.yaml 3.yaml
> sudo pip3 install yq==2.3.4
> echo | yq -e --slurp '.[0] == .[1]' - 2.yaml 3.yaml
true
> sudo pip3 install yq==2.6.0
> echo | yq -e --slurp '.[0] == .[1]' - 2.yaml 3.yaml
jq: Bad JSON in --slurpfile .[0] == .[1] -: Could not open -: No such file or directory

This is linked to ArgumentParser's prefix matches as described in this SO question.

Workaround: use -s instead of --slurp.

Given input files such as file.txt:

string from file

f.json:

{ "foo": "bar" }

and f.yaml:

foo: bar

The following command works fine in jq:

❯ jq -M -e -r --rawfile varname file.txt '[$varname, .foo]' -- f.json
[
  "string from file\n",
  "bar"
]

But the following fails in yq:

❯ yq -M -e -r --rawfile varname file.txt '[$varname, .foo]' -- f.yaml
usage: yq [-h] [--yaml-output] [--width WIDTH] [--version]
          jq_filter [files [files ...]]
yq: error: argument files: can't open '[$varname, .foo]': [Errno 2] No such file or directory: '[$varname, .foo]'

Hello, I am receiving the following error when trying the XML output. What am I doing wrong?

yq: Error running jq: ExpatError: not well-formed (invalid token): line 1, column 0.

Simple test to json

echo '{"example": "test"}' | yq .
{
  "example": "test"
}

Simple test to xml

echo '{"example": "test"}' | xq -x .
yq: Error running jq: ExpatError: not well-formed (invalid token): line 1, column 0.

I have tried many different files but nothing compiles to xml.

# pip freeze yq
asn1crypto==0.24.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
cryptography==2.2.2
idna==2.6
Jinja2==2.10
jq==0.1.6
jsonnet==0.10.0
jsonschema==2.6.0
kapitan==0.16.10
MarkupSafe==1.0
pkg-resources==0.0.0
pycparser==2.18
pyparsing==2.2.0
python-gnupg==0.4.2
PyYAML==3.12
reclass==1.5.2
requests==2.18.4
six==1.11.0
ujson==1.35
urllib3==1.22
xmltodict==0.11.0
yq==2.6.0

echo "aaa: 1" | yq ".aaa=\"abc\"" --yaml-output outputs aaa: abc and it is correct.

but if we use ~:
echo "aaa: 1" | yq ".aaa=\"~\"" --yaml-output outputs aaa: '~' which is incorrect.
Expected output is aaa: ~ - without quotation

Awesome project! I'm dealing now with a lot of yml files to be processed in kubernetes...

Is there a way to parse and print the key values in color?

Maybe just the same colors as jq or like in the vim editor?

I have an test.xml file which contains <?xml version="1.0" encoding="UTF-8" standalone="no"?> at its first line.

When I run xq --xml-output '.' test.xml, the generated xml doesn't contain <?xml version="1.0" encoding="UTF-8" standalone="no"?>.

It is possible to modify xq so that it preserves the XML declaration?

man yq No manual entry for yq

I would suggest a man entry.

brew install python-yq does not install xq ?

How do I install xq using Homebrew?

I have the following yaml file foo.yaml:

echo

Now, trying yq .bar.some-data foo.yaml or yq .bar['some-data'] foo.yaml doesn't give me y, but errors:

$ yq .bar.some-data foo.yaml
jq: error: data/0 is not defined at <top-level>, line 1:
.bar.some-data
jq: 1 compile error

$ yq .bar['some-data'] foo.yaml
jq: error: some/0 is not defined at <top-level>, line 1:
.bar[some-data]
jq: error: data/0 is not defined at <top-level>, line 1:
.bar[some-data]
jq: 2 compile errors

What is the correct way to query for those properties?

On FreeBSD with python-3.6:

Installed /usr/ports/textproc/yq/work/yq-2.7.2/.eggs/wheel-0.33.4-py3.6.egg
running egg_info
writing yq.egg-info/PKG-INFO
writing dependency_links to yq.egg-info/dependency_links.txt
writing entry points to yq.egg-info/entry_points.txt
writing requirements to yq.egg-info/requires.txt
writing top-level names to yq.egg-info/top_level.txt
reading manifest file 'yq.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'yq.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 46, in <module>
    "Topic :: Software Development :: Libraries :: Python Modules"
  File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.6/site-packages/setuptools/command/test.py", line 228, in run
    self.run_tests()
  File "/usr/local/lib/python3.6/site-packages/setuptools/command/test.py", line 250, in run_tests
    exit=False,
  File "/usr/local/lib/python3.6/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/local/lib/python3.6/unittest/main.py", line 141, in parseArgs
    self.createTests()
  File "/usr/local/lib/python3.6/unittest/main.py", line 148, in createTests
    self.module)
  File "/usr/local/lib/python3.6/unittest/loader.py", line 219, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/local/lib/python3.6/unittest/loader.py", line 219, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/local/lib/python3.6/unittest/loader.py", line 190, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/local/lib/python3.6/site-packages/setuptools/command/test.py", line 54, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/local/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/usr/local/lib/python3.6/test/bad_coding.py", line 0
SyntaxError: unknown encoding: uft-8
*** Error code 1

yq -y needs to preserve a 'literal block' (using |), or a 'folded block' (using '>')

Original YAML

$ cat y
some:
  file: |
    hello world
    line1
    line2

yaml 2 yaml

It gives different output:

$ cat y | yq -y .
some:
  file: 'hello world

    line1

    line2

    '

Would it be possible to preserve the original YAML 1-1 when yq is instructed to output YAML ?

Hello. I get this error everytime I try to launch yq with or without arguments (also the examples in in README.md don't work).

Traceback (most recent call last):
  File "/usr/bin/yq", line 11, in <module>
    load_entry_point('yq==2.7.2', 'console_scripts', 'yq')()
  File "/usr/lib/python3.7/site-packages/yq/__init__.py", line 119, in main
    parser = get_parser(program_name)
  File "/usr/lib/python3.7/site-packages/yq/__init__.py", line 96, in get_parser
    parser = Parser(**parser_args)
TypeError: __init__() got an unexpected keyword argument 'allow_abbrev'

yq will not install. See below

[root@aks-bastion01-poc /]# pip install yq
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting yq
  Using cached https://files.pythonhosted.org/packages/5c/2e/1df0a8163e19d3699fb745d56ff7d4c8fb0cbfe1d123609dc13dff6170b3/yq-2.7.2-py2.py3-none-any.whl
Requirement already satisfied: xmltodict>=0.11.0 in /usr/lib/python2.7/site-packages (from yq) (0.11.0)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from yq) (0.9.8)
Collecting PyYAML>=3.11 (from yq)
  Using cached https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz
Installing collected packages: PyYAML, yq
  Found existing installation: PyYAML 3.10
Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
[root@aks-bastion01-poc /]# yq
-bash: yq: command not found
[root@aks-bastion01-poc /]# find . -name yq 
[root@aks-bastion01-poc /]# 

yq internals are not intended to be used as a library, as it dumps out to sys.stdout.

It would be useful to have the output stream of jq provided to the caller, so that another Python module can decide what to do with the data.

The YAML standard allows embedding multiple YAML documents into one by separating them with --- lines.

When yq is given these as input, it errors out with:

yq: Error while running jq: ComposerError: expected a single document in the stream
  in "<stdin>", line 2, column 1

I have a tool that can output yaml or json and when it outputs json it returns two objects concatenated, something like:

{
   'thing': 1
}
{
  'thing': 2
}

Interestingly, if I pass these directly to jq it handles them cleanly. It runs the given command on each document.

Because yq wraps jq it seems like it should be able to handle this in a like manner.

$ yq --version
yq 2.7.2
$ jq --version
jq-1.6
$ cat hodor.yml 
- hodor
$ yq read hodor.yml 
jq: error: read/0 is not defined at <top-level>, line 1:
read
jq: 1 compile error

I installed yq and jq via brew (python-yq and jq, respectively).

jq has a feature request open to support sed-like in-place editing of files. It would be lovely if yq were to support that in advance.

e.g. yq '(.foo.bar=3)' -i foo.yaml would replace foo.yaml in-place, safely, with an implied -y. (write a temp file, close the source file, rename the temp file over the source file) [on windows, a delete or mv of the source file is also needed in there because 👎 but hey, close as we can get.

I'd be happy to put a patch up if this feature request seems reasonable.

test.yaml contents:

test: "test output"
my-test: "my-test output"

Error when trying to access my-test:

❯ cat test.yml | yq .test
"test output"
❯ cat test.yml |yq .my-test
jq: error: test/0 is not defined at <top-level>, line 1:
.my-test
jq: 1 compile error

Using yq in a makefile (run with GNU Make 4+) produces a fatal error. For some reason in that case sys.stdin does not have a isatty() function:

AttributeError: 'NoneType' object has no attribute 'isatty'
make[1]: *** [makefile:18: tt] Error 1
make[1]: Leaving directory '<path>'
make[1]: Entering directory '<path>'
Traceback (most recent call last):
  File "/usr/bin/yq", line 11, in <module>
    load_entry_point('yq==2.7.0', 'console_scripts', 'yq')()
  File "/usr/lib/python3.7/site-packages/yq/__init__.py", line 132, in main
    if sys.stdin.isatty() and not args.files:
AttributeError: 'NoneType' object has no attribute 'isatty'

The same command run from a shell works fine.

yq is currently distributed as a Homebrew formula (#11), and that's awesome!

As a suggestion, I think it would be great if yq was distributed as a Debian package as well. This would allow more folks to easily install it in Debian-compatible Linux distributions (like Ubuntu), doing something like:

sudo add-apt-repository ppa:kislyuk/yq
sudo apt update
sudo apt install yq -y

This would also make a real case for using yq more broadly in server scripts, Linux-based developer workstations, etc.

Convert fails with jq: 1 compile error... Tried to convert and got the following

$ yq r sample.json
jq: error: r/0 is not defined at <top-level>, line 1:
r
jq: 1 compile error

• Using jq 1.5_3

$ yq --version
yq 2.4.1

$ jq --version
jq-1.5

input.yml:

a: b
c:
  - d

Command:
yq -y '.c += ["e"]' input.yml

Output:

a: b
c:
- d
- e

Expect output:

a: b
c:
  - d
  - e

The current code makes the assumption that input and output formats are always the same.

I think it would be much more useful if the input and output stages could be separated and options mixed or matched at will. This would turn the project into a general purpose transcoder as well as a query engine for each format. It already does all the transcoding anyway, It seems like an arbitrary restriction to force transcoding back to the same format it came from.

My fix for passthrough JSON input in #57 already introduces one specific scenario of mismatched formats being useful, but a more general purpose solution where input and output formats can be set independently would add a lot more flexibility.

For example, printing null to stdout breaks the following:

my_val="$(echo 'foo: bar' | yq -re '.someotherkey' || echo 'default_val')"

Since this will result in null instead of default_val.

Instead, yq should output an error message to stderr on no match, at least in the case of -e.

A similar project exists in golang, also called yq:

https://github.com/mikefarah/yq

It only handles YAML, however. It would still be nice to resolve the naming conflict between the two programs. I would recommend renaming this one to xq or xyq because it also handles XML. :)

Hi @kislyuk, thanks for maintaining a great project 👍

Would you mind if I contributed a script to build pex binary for yq?
I'd greatly appreciate it if you could build binaries with my script and host these as GitHub releases for easier installation.

It would also helps #11 by simplifying the installation process - you are no longer likely to need running pip install from within the brew. Instead, just install python3 and the pex binary.

For reference, I've done the same thing on physera/onelogin-aws-cli#29.

e.g.

foo.json

{
  "test": {
    "s1": "bar",
    "s2": [
      "snowflake",
      "s3",
      "gcloud"
    ]
  }
}

yq -y . foo.json

test:
  s1: bar
  s2:
  - snowflake
  - s3
  - gcloud

When valid yaml should be:

 test:
   s1: bar
   s2:
     - snowflake
     - s3
     - gcloud