Description

In #189 @marcules is implementing the configuration of yamlfix through file, commandline arguments and environment variables.

It will be interesting to be able to override the configuration in a particular file. I'm thinking of #191 . I know that I would not want that option enabled by default, but I will want it in one specific file. I could not add a yamlfix.ini in that directory because it lives with other files where I don't want the feature.

Possible Solution

We've thought of two solutions (but if you come up with something else that's fine too!):

Special comments on the file

Implement special comments (like vim does) at the end of the document that override the rest of the configurations.

Add an override section on the configuration by path

To be something like:

overrides.{fileNamePattern}.keyPath.{keyPath}.{config_key}: {config_value}
and
overrides.{fileNamePattern}.recursiveKeyPath.{keyName}.{config_key}: {config_value}

and then apply whatever is in config_key/config_value for that specific node (or nodes).
Not sure if that would make it better/easier to configure, but at least the user does not have to modify the files to add metadata to it, and also with patterns in both filename+key the user could configure a set of settings for multiple similar files.

Additionally this could then be just passed as an environment variable YAMLFIX_overrides.values_yaml.keyPath.global_key_subkey.flow_style_sequence=true or something like that (with special handling for underscore -> subpath/glob conversion).

Or maybe a set of matching values would be easier to understand and implement. E.g.:

overrides:
  - file_name: /absolute/path/values.yaml
     match_key: /absolute/jsonPath
     options:
       flow_style_sequence: true
  - file_name: **/**/values-glob.yaml
     match_key: relative_jsonPath[]
     options:
       flow_style_sequence: false

Then only glob + jsonPath handling need to be implemented and maybe env-to-list conversion.

Additional context

Related Issue

Description

Some yaml files in this repo require fixing.
Fix them now and add a pipeline step to check for them later.

Possible Solution

After #215, run yamlfix . in project root and commit.

Additional context

Might also want to think whether section_whitelines needs to be adjusted.

Description

Ansible vault files are YAML files that contain encrypted YAML content in the format described here

Running yamlfix on such files renders them unuseable.

Steps to reproduce

1. Create an ansible-vault file
   echo "program: yamlfix" | ansible-vault encrypt - --output myvault.yml
2. Inadvertantly launch yamlfix *.yml forgetting to exclude this particular file
3. See that the vault is now broken

Current behavior

Screenshots

Before yamlfix

After yamlfix

Broken vault as a result

Desired behavior

The desired behavior would be to not modify files beginning with $ANSIBLE_VAULT; at all.

Environment

          yamlfix version: 0.7.3
           python version: 3.8.10 (default, Sep 28 2021, 16:10:42)  [GCC 9.3.0]
                 platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.29

Description

When changing list style from block to flow, comments are not placed properly.

Steps to reproduce

---
list:
  # Comment
  - echo "Test"

Current behavior

---
list:
  # Comment
[echo "Test"]

Desired behavior

---
# Comment
list: [echo "Test"]

Environment

------------------------------------------------------------------
     yamlfix: 1.3.1
     Python: 3.10.8
     Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.31
------------------------------------------------------------------

Thank you for yamlfix!

Description

I'd like to integrate yamlfix in tox (and thus CI) in a way that signals to the user there are fixes to do only (but don't actually do them). This is similar to the --check for black and isort.

Possible Solution

Check if files would be changed, do not change them and adjust the exit code accordingly.

Description

Once #215 gets merged, yamlfix . will become possible.
When you run it like this on a large repository the output is quite noisy.
We should only log changed files by default.

Possible Solution

Log unchanged files on debug level and only changed files on info level.

Additional context

Can also probably add -v and -vv options to print debug and trace logs.

Hi all, right now I only find time to maintain yamlfix this means:

• Expect to have an answer to your PR/issue in 2/3 weekdays.
• If you open an issue for a new feature or bug fix that I don't feel it's critical, it won't be implemented until you or someone else does a pull request.

If you want to see the project thrive please evaluate becoming a maintainer and help me answering issues, pull requests and develop the program.

This announcement is not meant to prevent you from opening issues! Even though we're in maintenance mode it's always good to do requests, maybe other yamlfix users want to implement it :)

Description

When files contain block strings, yamlfix will squash white space and change the string.

Current behavior

Given the file:

# original.yml
---
key: |
  # This is a value #
  #      X          #

yamlfix original.yml results in:

---
key: |-
  # This is a value  #
  #      X  #

Desired behavior

Strings in block quotes should remain semantically equivalent, or ideally not be modified at all. As it is, white space is removed.

Environment

------------------------------------------------------------------
     yamlfix: 1.9.0
     Python: 3.10.6
     Platform: Linux-5.15.0-27-generic-x86_64-with-glibc2.35
------------------------------------------------------------------

Description

Currently dict entries and sequence / list elements are not sorted. It would be nice to have two configuration options to sort dict entries and list elements alphabetically. This of course does not work with all config files, especially something like kubernetes pod command and args, which need to stay in their original order to make sense, maybe a whitelist/blacklist option for this configuration could be implemented too, so that it is possible to say "in general I want lists ordered, except these" or "in general I don't want lists sorted, except for these lists".

Today:

list:
  - b
  - c
  - a
  - 1
  - null
  - true
  - false

map:
  z: 'key'
  a: 'key'

a: 'key'

After implementing this config option and setting both dict/sequence options to true:

a: 'key'

list:
- null
- true
- false
- 1
- a
- b
- c

map:
  a: 'key'
  z: 'key'

Possible Solution

Additional context

Related Issue

Description

Add ability to print changes on stdout for pipelines

Possible Solution

Preferably would add to --check to output the changes, so that can be consumed and added to a merge request comment

Alternatively add a --diff that would combine --check and the output

Additional information

This could work similarly to how black uses --diff
https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#diffs

Description

Since 1.4.0 all white lines are removed.
This makes it difficult to read long files.

Add an option to specify max number of consecutive white lines.

Possible Solution

Have a configuration option, which would specify behaviour:

max_whitelines = 1

Additional context

I believe, in general, all stylistic choices should be configurable.

Description

yamlfix is transforming this yamllint-compliant block

---
- name: Setup SONiC Build Servers
  hosts: sonic_build_server
  vars:
    # IMPROVE: stuff
    build_user: build

to this, which conflicts with yamllint's expectation that the comment would align with the line after the comment, not before.

---
- name: Setup SONiC Build Servers
  hosts: sonic_build_server
  vars:
  # IMPROVE: stuff
    build_user: build

Perhaps a regression of #55, but I haven't done a git bisect to figure out where it broke, or if it ever worked.

Environment

------------------------------------------------------------------
     yamlfix: 1.1.0
     Python: 3.10.8
     Platform: macOS-12.6.1-arm64-arm-64bit
------------------------------------------------------------------

Description

Currently whitelines are respected even though if there are too many.

Steps to reproduce

Run yamlfix on the next code

---
list: [item, item]



key: value

Current behavior

Source code is not changed.

Desired behavior

The unneeded lines are removed leaving the source code as:

---
list: [item, item]
key: value

We may need to decide what to do with comments, maybe add a white line above?

We could also add a configuration to specify the number of whitelines between "sections" of the yaml

Description

When using this configuration, unicode characters are printed using the escape sequence, rather than the unicode characters itself:

quote_basic_values=true
quote_representation='"'

Steps to reproduce

Run:

echo "a: 📈" | YAMLFIX_quote_basic_values=true YAMLFIX_quote_representation='"' yamlfix -  2>/dev/null

Current behavior

---
a: "\U0001F4C8"

Desired behavior

---
a: "📈"

Environment

python -c "import yamlfix.version; print(yamlfix.version.version_info())"
------------------------------------------------------------------
     yamlfix: 1.8.1
     Python: 3.10.9
     Platform: macOS-13.2-x86_64-i386-64bit
------------------------------------------------------------------

yaml ignores underscores in numbers, so these are equivalent. I use the underscores for readability with large numbers. It'd be nice to have this as an option in yamlfix.

key1: 1435498
key2: 1_435_498

Description

Valid YAML file containing anchors and aliases/merge points is broken by yamlfix

Current behavior

Source yaml: This is a snippet from docker-compose.yml, trimmed to focus on the issue

x-node-volumes: &node-volumes
  node3_data:

x-vault-volumes: &vault-volumes
  vault_data:

x-mongo-volumes: &mongo-volumes
  mongo_data:

x-certmgr-volumes: &certmgr-volumes
  cert_data:

volumes:
  <<: *node-volumes
  <<: *vault-volumes
  <<: *mongo-volumes
  <<: *certmgr-volumes

Result:

---
x-node-volumes: &node-volumes
  node3_data:

x-vault-volumes:
  vault_data:

x-mongo-volumes: &mongo-volumes
  mongo_data:

x-certmgr-volumes:
  cert_data:

volumes:
  <<: *node-volumes
  <<: *mongo-volumes

Desired behavior

• no change. Document is already formatteed

In this specific example, during YAML processing, the "volumes" map no longer contains all the volumes once yamlfix has processed the file.

Side node: Using a different method to aggregate the anchors produces correct results

volumes:
  <<: [*node-volumes, *vault-volumes, *mongo-volumes, *certmgr-volumes]

Environment

• Ubuntu 20.04
• Yamlfix version

$ yamlfix --version
              yamlfix version: 0.7.2
               python version: 3.8.10 (default, Jun  2 2021, 10:49:15)  [GCC 9.4.0]
                     platform: Linux-5.8.0-64-generic-x86_64-with-glibc2.29

Description

Lists that are at first level get indented two spaces.

Steps to reproduce

1. Create a document with the following text:

---
- name: Fail task
  fail:
    msg: You failed

2. Run yamlfix

Current behavior

The content gets indented

---
  - name: Fail task
    fail:
      msg: You failed

Desired behavior

The code is not indented

Description

Line cropping is not working for comments.

Steps to reproduce

Transform the next document:

# this_is_a_test: asdfasdfasfd asdf adsfasdfasdfasfd asdfasdfasdfasdfa adsfasdfasfdasdf asdfasdfasdfasdf asdfasdfasdfa asdfasdfasdf asdfasdfads asdfasdfasdf asdfasdfasdf asdfasfda
this_is_a_test: asdfasdfasfd asdf adsfasdfasdfasfd asdfasdfasdfasdfa adsfasdfasfdasdf asdfasdfasdfasdf asdfasdfasdfa asdfasdfasdf asdfasdfads asdfasdfasdf asdfasdfasdf asdfasfda

Current behavior

The non commented line gets cropped but the comment doesn't

---
# this_is_a_test: asdfasdfasfd asdf adsfasdfasdfasfd asdfasdfasdfasdfa adsfasdfasfdasdf asdfasdfasdfasdf asdfasdfasdfa asdfasdfasdf asdfasdfads asdfasdfasdf asdfasdfasdf asdfasfda
this_is_a_test: asdfasdfasfd asdf adsfasdfasdfasfd asdfasdfasdfasdfa adsfasdfasfdasdf
  asdfasdfasdfasdf asdfasdfasdfa asdfasdfasdf asdfasdfads asdfasdfasdf asdfasdfasdf
  asdfasfda

Desired behavior

The comment line is also cropped

Description

After implementation of #189 I ran it on our internal helm charts repository and found an edge case where currently yamlfix would mangle the produced lists/yaml files. When an empty list is defined (always has to be flow-style) with a comment on the same line, the comment remains on the line, but the flow-style list is moved to the next line, without any indentation at all, which breaks any further yaml in the file. @lyz-code I'll fix this the coming days.

Steps to reproduce

Run yamlfix on a file with a list that is commented like this:

indented:
  key: value
  list: []  # comment on the same line
  map:
    anotherKey: anotherValue

Current behavior

indented:
  key: value
  list:  # comment on the same line
[]
  map: anotherKey: anotherValue

Desired behavior

File stays the same.

Environment

1.2.0

Description

yamllint specifies that is forbidden to use non-explictly typed truthy values other than allowed ones (by default: true and false), for example YES or off.

Possible Solution

Dive into the ruamel.yaml options to see if they support it, or create a function to fix it afterwards.

Description

Problem in the fix_files function in services.py

It's either a bug, if docstrings are the reference, or an incorrect docstring if the code is the reference...

Steps to reproduce

The fix_files' docstring in services.py says If the input is taken from stdin, it will output the value to stdout. but in fact nothing is ever output to stdout by fix_files, and instead fix_files returns the fixed stdin input as a string.

Current behavior

The docstring doesn't match what fix_files does.

Desired behavior

The docstring or the function has to be modified in order for both to match.

Environment

          yamlfix version: 0.7.3
           python version: 3.8.10 (default, Sep 28 2021, 16:10:42)  [GCC 9.3.0]
                 platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.29

Description

Many applications have a default user location for their configuration files, such as $XDG_CONFIG_HOME/<application_name>/config. That way whenever yamlfix runs, it's not necessary to pass it each time an argument to the configuration file location. This is useful for editor integration, such as formatting and linting frameworks that usually just call the application without extra configuration arguments.

Possible Solution

The YAMLlint configuration can be a good inspiration:

To use a custom configuration file, use the -c option:

yamllint -c /path/to/myconfig file-to-lint.yaml

If -c is not provided, yamllint will look for a configuration file in the following locations (by order of preference):

• a file named .yamllint, .yamllint.yaml, or .yamllint.yml in the current working directory
• a filename referenced by $YAMLLINT_CONFIG_FILE, if set
• a file named $XDG_CONFIG_HOME/yamllint/config or ~/.config/yamllint/config, if present

Finally if no config file is found, the default configuration is applied.

Additional context

https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

Thanks.

Description

Prepare the repository so it can be used as a pre-commit.

Possible Solution

Add the required files.

Additional context

Related Issue

Description

We use autoupdate which updated the tag to v0.8.0, but the pre-commit file for that release has the wrong types and was fixed in this commit c25c110

Can you release a new v0.8.1 tag?

As a temporary workaround, the file patterns can be overwritten:

repos:
  - repo: https://github.com/lyz-code/yamlfix/
    rev: 0.8.0
    hooks:
      - id: yamlfix
        exclude: .copier-answers.yml
        types_or: []
        types: [file, yaml]

Description

I have a yaml file with this format :

name: &alias key
*alias: value

which correspond to :

name: key
key: value

I have the following error:

[+] YamlFix: Fixing files
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Traceback (most recent call last):
  File "/home/jordan/.local/bin/yamlfix", line 8, in <module>
    sys.exit(cli())
  File "/home/jordan/.local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/jordan/.local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/jordan/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jordan/.local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/jordan/.local/lib/python3.8/site-packages/yamlfix/entrypoints/cli.py", line 86, in cli
    fixed_code, changed = services.fix_files(files_to_fix, check, config)
  File "/home/jordan/.local/lib/python3.8/site-packages/yamlfix/services.py", line 82, in fix_files
    fixed_source = fix_code(source, config)
  File "/home/jordan/.local/lib/python3.8/site-packages/yamlfix/services.py", line 154, in fix_code
    source_code = fixer.fix(source_code=source_code)
  File "/home/jordan/.local/lib/python3.8/site-packages/yamlfix/adapters.py", line 362, in fix
    source_code = fixer(source_code)
  File "/home/jordan/.local/lib/python3.8/site-packages/yamlfix/adapters.py", line 380, in _ruamel_yaml_fixer
    for source_dict in source_dicts:
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/main.py", line 477, in load_all
    yield constructor.get_data()
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/constructor.py", line 134, in get_data
    return self.construct_document(self.composer.get_node())
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/composer.py", line 63, in get_node
    return self.compose_document()
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/composer.py", line 96, in compose_document
    node = self.compose_node(None, None)
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/composer.py", line 142, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/composer.py", line 215, in compose_mapping_node
    while not self.parser.check_event(MappingEndEvent):
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/parser.py", line 150, in check_event
    self.current_event = self.state()
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/parser.py", line 636, in parse_block_mapping_key
    if self.scanner.check_token(KeyToken):
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/scanner.py", line 1826, in check_token
    self.fetch_more_tokens()
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/scanner.py", line 240, in fetch_more_tokens
    return self.fetch_stream_end()
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/scanner.py", line 462, in fetch_stream_end
    self.remove_possible_simple_key()
  File "/home/jordan/.local/lib/python3.8/site-packages/ruyaml/scanner.py", line 401, in remove_possible_simple_key
    raise ScannerError(
ruyaml.scanner.ScannerError: while scanning a simple key
  in "<unicode string>", line 2, column 1:
    *alias: value
    ^ (line: 2)
could not find expected ':'
  in "<unicode string>", line 2, column 14:
    *alias: value
                 ^ (line: 2)

Description

yamlfix has started removing whitespace between list elements that it previously would leave untouched. I think this is a step too far for a default configuration, because a lot of uses and tools use whitespace to separate items in a list.

Steps to reproduce

Given a YAML file like this

---
hosts: all
tasks:
  - name: Do something
    command: |
      echo something

  - name: Do another thing
    command: |
      echo "another thing"

Current behavior

Yamlfix removes all empty lines

---
hosts: all
tasks:
  - name: Do something
    command: |
      echo something
  - name: Do another thing
    command: |-
      echo "another thing"

Desired behavior

Yamlfix should (at least by default) ignore at least one blank line between items in a list.

Environment

------------------------------------------------------------------
     yamlfix: 1.9.0
     Python: 3.9.16
     Platform: macOS-12.4-arm64-arm-64bit
------------------------------------------------------------------

Description

yamlfix can only be run with the default settings, which are hard-coded.

In some cases it might be necessary to override a rule. I think black and yamllint could be good models for this: sensible defaults out-of-the-box, but if you really need or want to, you can override or even ignore some of them, such as maximum line length.

However, since yamlfix is meant to be opinionated, we need to keep sensible defaults working without any need for configuration.

Possible Solution

@Jasha10 has implemented this functionality already on autoimport in this PR. It should be easier now to implement it for yamlfix.

As for the spec, it could be a good idea to use the same config options and keys as yamllint, whenever possible.

It may make sense to add a configuration option for each of the changes that we make so that users can choose what transformations to apply potentially speeding up the program run.

Related Issue

After #4 and #116 are taken care of (the default needs to be working first), we could make line-length configurable.

Description

Yamllint specifies a maximum line length.

Yamlfix does not automatically crop long lines, nor join together short multilines, nor convert back to one line the multilines that do no longer need cropping, like black does.

Possible Solution

Dive into the ruamel.yaml options to see if they support it, or create a function to fix it afterwards.

Additional context

If you understand something else of the library, try to share it in this issue

Once complete comment in this and this PRs

Description

Since #208 white lines are allowed on your yaml to separate the content in sections. However you can't add whitelines if the comment is not in the first level of indentation so:

---
list:
  - item

  # Comment: desired: 0 new lines before this comment
  - item
key: value

Gets transformed to:

---
list:
  - item
  # Comment: desired: 0 new lines before this comment
  - item
key: value

Possible Solution

• Replace this regex by "\n\n+[\t ]{0,}[#]". Where the expression [\t ]{0,} matches tab or space between 0 and unlimited number of times.
• Instead of using a fixed string here (i.e. desired_whitelines_with_comments), use a method that: Replaces all of the \n by the desired number of \n and keep all of the trailing \t, space and the final character (#).

Additional context

This PR touches code that @marcules and @fortin-alex have created. They may have some insights on the implementation too

Related Issue

#193

Description

yamlfix is removing quotes around URLs which causes parser errors

Steps to reproduce

Running yamlfix on this file (from here):

---
repos:
  - repo: https://github.com/commitizen-tools/commitizen
    rev: v2.20.4
    hooks:
      - id: commitizen
        # yamllint disable-line rule:line-length
        additional_dependencies: [
          "git+https://github.com/KyleKing/cz_legacy.git@d2e273003ac302f6583f0c5d4c0f2a29aaeee73b"
        ]
        stages: [commit-msg]

Current behavior

Is changed to this:

additional_dependencies: [git+https://github.com/KyleKing/cz_legacy.git@d2e273003ac302f6583f0c5d4c0f2a29aaeee73b]

Error from pre-commit's check-yaml (and also leads to errors with ruamel.yaml):

check yaml...............................................................Failed
- hook id: check-yaml
- exit code: 1

while scanning a plain scalar
  in ".pre-commit-config.yaml", line 44, column 35
found unexpected ':'
  in ".pre-commit-config.yaml", line 44, column 44

Desired behavior

I don't think yamlfix should be removing quotes on strings that contain colons

Environment

Using yamlfix latest from master (~0.8.1) on macOS and Python 3.8

Description

Currently list values are not being quoted if it is not necessary to distinguish the type -> i.e. strings that look like integers will get quoted. It would be nice to have an option to quote all lists, that only contain simple values.

Today:

list:
  - a
  - b
  - c
  - '1'
  - 2
  - true
  - 'false'

list_two: [a, b, c, '1', 2, true, 'false']

complex_list:
  - simple_value
  - complex_value:
      key: 'value'

With this config option implemented:

list:
  - 'a'
  - 'b'
  - 'c'
  - '1'
  - 2
  - true
  - 'false

list_two: ['a', 'b', 'c', '1', 2, true, 'false']

# stays the same
complex_list:
  - simple_value
  - complex_value:
      key: 'value'

Possible Solution

Additional context

Related Issue

Description

We don't know how the list is being used, transforming it with commas can be incorrect.
For example, GitLab uses yaml for CI pipeline.
A list represents a set of commands to be executed one after another with && in between.
By changing to flow and using commas, the logic is changed.

Steps to reproduce

N/A

Current behavior

sequence_style: YamlNodeStyle = YamlNodeStyle.FLOW_STYLE

Desired behavior

sequence_style: YamlNodeStyle = YamlNodeStyle.KEEP_STYLE

Environment

------------------------------------------------------------------
     yamlfix: 1.3.1
     Python: 3.10.8
     Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.31
------------------------------------------------------------------

Description

If not files are provided, run on all yaml files recursively.

Possible Solution

from pathlib import Path
if not files:
	files = [Path.cwd().rglob(f"*.{ext}") for ext in ["yml", "yaml"]]
	files = [file for l in files for file in l]

Additional context

This will make it work similar to black, isort and other tools and simplify usage.

At the same time, it might be a good idea to add include and exclude options to config to simplify usage.
In addition, if no files are found just say so and don't error.

Description

Is there any command or feature to run yamlfix on multiple files of the project? I am looking for something like yamlfix . or yamlfix *.yml

Possible Solution

Additional context

Related Issue

Description

When a file has only comments, the comments are removed

Steps to reproduce

Run fix_code over:

---
# Keep comments!

Current behavior

The file is transformed to:

---
...

Desired behavior

The file is unchanged

Additional context

@rsnodgrass created the unit test in this PR which was closed due to inactivity, if you want to implement this, you can follow in that branch

I have the following yaml file:

---
- key: ['value?']

which loads correctly in python with pyyaml's yaml.safe_load.

yamlfix changes the file to

---
- key: [value?]

which cannot be loaded by pyyaml's yaml.safe_load, error being "expected ',' or ']', but got '?'"

yq breaks similarly with error "parse error: Invalid numeric literal at line 2, column 0"

It would be nice if yamlfix did not remove quotes on strings with question marks (and whatever other set of characters can cause this issue), or if there was a way to disable the feature to remove quotes entirely.

Description

When I have comment wirh a link containing an anchor (i.e. `# https://docs.gitlab.com/ee/ci/merge_request_pipelines/index.html#prerequisites), yamlfix wrongly formats the link

Steps to reproduce

1. Use this yaml file as example:

---
# https://docs.gitlab.com/ee/ci/merge_request_pipelines/index.html#prerequisites
foo: bar

2. Run yamfix on it

Current behavior

• The link gets wrongly indented with an additional space
• The anchor at the end of the URL gets wrongly indented and detached from the link

Desired behavior

yamlfix should not change anything

Environment

yamlfix v. 0.12.3.1

Description

It is common (if unfortunate) in the k8s world to have a single file with multiple API documents:

foo:
...
---
bar:
...
---
quux:

Possible Solution

Dive into the ruamel.yaml options to see if they support it, or split the documents inside the yaml file and run fix_code on each of them and then join them at the end.

Additional context

If you understand something else of the library, try to share it in this issue

Related Issue

yamlfmt original issue.

Description

When a long line contains a jinja variable, it get's splitted in two lines, breaking it. Related to #4

Steps to reproduce

The output is:

vars:
  - service_name: '{{ test_docker_project }}_{{ test_very_long_backend_service_name
      }}_1'

Desired behavior

The line is split after the variable definition

vars:
  - service_name: >
      '{{ test_docker_project }}_{{ test_very_long_backend_service_name }}_1'

Environment

              yamlfix version: 0.7.1
               python version: 3.7.3 (default, Jan 22 2021, 20:04:44)  [GCC 8.3.0]
                     platform: Linux-4.19.0-5-amd64-x86_64-with-debian-10.10

Description

When running yamlfix over a file the comments are deindented one level.

Steps to reproduce

- key: 
  # Comment

Current behavior

The coment is deindented

Desired behavior

Comment indentation is respected

Environment

Description

Currently it is not possible to pass a "None" (Optional) value through toml or the environment to disable the flow-style option for sequences through the cli. If using as a library the user can just pass None.

As flow-style for sequences is currently the only Optional value that has not None as the default value, it is not possible to pass the third option "leave lists as is" to yamlfix.

Possible Solution

Change type of flow_style_sequence from Optional[bool] to an enum that can be flow-style, block-style, as-is and possible change the field-name from flow_style_sequence to sequence_style.

Additional context

Related Issue

Description

Right now, the yamlfix output does not include a newline at the end of file nor does it preserve it if it was there before.
It would be nice to enforce it to comply with the newline POSIX standard.

Possible Solution

This happens because _ruamel_yaml_fixer deletes newlines at end of file by default, through loading and dumping the yaml contents.
A good solution could be to add a new fixer function to add/restore it.

Description

When formatting lists that have a comment the indentation goes wrong.

Steps to reproduce

1. Create a document with the following text:

---
# Comment
- name: Fail task
  fail:
    msg: You failed

2. Run yamlfix

Current behavior

---
# Comment
  - name: Fail task
    fail:
      msg: You failed

Desired behavior

---
# Comment
- name: Fail task
  fail:
    msg: You failed

Environment

Description

yamlfix breaks YAML files which start with a shebang line when adding the '---' YAML beginning of file mark, this mark being already present or not.

We use YAML files which start with a shebang line with Ansible.

Steps to reproduce

Consider the following YAML file, which passes just fine through yamllint v1.26

#!/usr/bin/env ansible-playbook
---
#

- import_playbook: playbooks/blah.yml

when you launch yamlfix on this file you'll get

---
#

- import_playbook: playbooks/blah.yml

So the shebang line was dropped, although the file should not have been modified at all.

Now consider the following file, which only has a shebang line but not YAML BOF marker

#!/usr/bin/env ansible-playbook
#

- import_playbook: playbooks/blah.yml

And here's the result when passed through yamlfix

---
#!/usr/bin/env ansible-playbook
#

- import_playbook: playbooks/blah.yml

Now the shebang line is preserved but moved to line 2, so the behavior is broken since the YAML file can't be executed anymore through the interpreter defined in the shebang line.

In conclusion, yamlfix should skip existing shebang lines before adding the YAML BOF marker if needed.

Hoping this helps.

Environment

yamlfix version: 0.7.2
python version: 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0]
platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.29

Description

Currently the fix_files function overwrites files even if their YAML content was left unmodified by fix_code

This could be optimized by only overwriting files which content was fixed.

Possible Solution

Wrapping a simple conditional around the file overwriting code would prevent yamlfix from doing unwanted disk I/O.

I'll create a PR for this.

Description

yamlfix has a couple of fixers that operate on string replacement basis and there could be cases where those fixers mangle the yaml files or change them in a way that is not compliant.

Possible Solution

A general approach / solution could be to generate the newly fixed/linted yaml file into a temp file, parse the output with either ruyaml itself or another yaml parser, and if an error occurs don't modify the file. If no error occurs just move the modified file over the target file.

This would enable the user to be sure, that the file is 100% always parsable after yamlfix ran.
This would also enable yamlfix to output a diff if the file was corrupted to warn the user of the reason why the file could not be fixed/linted.

Additional context

Related Issue

When fixing a file using the Windows Python versions, the line endings change from LF style to CRLF style, and only when there is some other change in the file.

I suggest to keep the style of the line endings (keep LF or CRLF whatever it was originally) - or enable forcing a line ending.

yamlfix --version

 yamlfix: 1.8.0
 Python: 3.10.9
 Platform: Windows-10-10.0.22621-SP0

Description

Yamllint specifies that there must be a space between the # and the first character.

Yamlfix does not automatically add this space, or transform the multiple spaces into one.

Possible Solution

Dive into the ruamel.yaml options to see if they support it, or create a function to fix it afterwards.

Description

Quoted strings containing the # character shouldn' t be modified but they are.

Steps to reproduce

Consider the following YAML code

---                                                                                                                                                     
somestring: "Here # is not a comment marker"  

When running yamlfix on this file the result is

---
somestring: 'Here  # is not a comment marker'

So a space character was wrongly inserted in front of the hash character while the quotes were preserved (but changed from double to simple which is fine).

Current behavior

the _fix_comment function disregards the fact that some hash characters might be embedded in quoted strings and treat them as comment markers. This has a potential to break some user files.

Desired behavior

The desired behavior is to not modify quoted strings containing hash characters.

Not sure how to best fix this without doing real YAML parsing though, so maybe the fix should be done in ruyaml and yamlfix shouldn't do this sort of fix.

Environment

          yamlfix version: 0.7.3
           python version: 3.8.10 (default, Sep 28 2021, 16:10:42)  [GCC 9.3.0]
                 platform: Linux-5.4.0-90-generic-x86_64-with-glibc2.29

Description

yamlfix crashes when run.

Steps to reproduce

• pip install yamlfix
• git clone https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
• cd pubsec-declarative-toolkit
• yamlfix .

Current behavior

Crashes.

[+] Fixed solutions/guardrails-policies/12-cloud-market-place/constraint.yaml
Traceback (most recent call last):
  File "/home/bain/.local/bin/yamlfix", line 8, in <module>
    sys.exit(cli())
  File "/home/bain/.local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/bain/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/bain/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/bain/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/bain/.local/lib/python3.10/site-packages/yamlfix/entrypoints/cli.py", line 86, in cli
    fixed_code, changed = services.fix_files(files_to_fix, check, config)
  File "/home/bain/.local/lib/python3.10/site-packages/yamlfix/services.py", line 82, in fix_files
    fixed_source = fix_code(source, config)
  File "/home/bain/.local/lib/python3.10/site-packages/yamlfix/services.py", line 154, in fix_code
    source_code = fixer.fix(source_code=source_code)
  File "/home/bain/.local/lib/python3.10/site-packages/yamlfix/adapters.py", line 362, in fix
    source_code = fixer(source_code)
  File "/home/bain/.local/lib/python3.10/site-packages/yamlfix/adapters.py", line 380, in _ruamel_yaml_fixer
    for source_dict in source_dicts:
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/main.py", line 477, in load_all
    yield constructor.get_data()
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 134, in get_data
    return self.construct_document(self.composer.get_node())
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 151, in construct_document
    for _dummy in generator:
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 1627, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 1506, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 174, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 216, in construct_non_recursive_object
    for _dummy in generator:
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 1627, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 1506, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 174, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 216, in construct_non_recursive_object
    for _dummy in generator:
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 1627, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 1507, in construct_mapping
    if self.check_mapping_key(node, key_node, maptyp, key, value):
  File "/home/bain/.local/lib/python3.10/site-packages/ruyaml/constructor.py", line 314, in check_mapping_key
    raise DuplicateKeyError(*args)
ruyaml.constructor.DuplicateKeyError: while constructing a mapping
  in "<unicode string>", line 20, column 5:
        config.kubernetes.io/local-confi ... 
        ^ (line: 20)
found duplicate key "config.kubernetes.io/local-config" with value "true" (original value: "true")
  in "<unicode string>", line 21, column 5:
        config.kubernetes.io/local-confi ... 
        ^ (line: 21)

To suppress this check see:
    http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys

bain@DESKTOP-N2H6GLJ:~/pubsec-declarative-toolkit$ 

Desired behavior

either autofix the issue or present user with a simple error code

Screenshots
If applicable, add screenshots to help explain your problem.
-->

Environment

-->