Thank you for putting together this awesome package.

I'm struggling to figure out how to yaml.dump a file containing an !If tag (in this case, within a CommentedSeq).

I'm probably missing something obvious but can't seem to figure it out. Any help much appreciated. Thank you!

Hi,

is there a way to join two lists? E.g.:

!Defaults
lamps:
  - {name: "lamp a"}
  - {name: "lamp b"}
sockets:
  - {name: "socket a"}
---
all_devices:
  !Loop
    over: !JoinLists lamps sockets
    as: device
    template:
      name: !Lookup device.name

Or, even better join them direcly in the Defaults section:

!Defaults
lamps:
  - {name: "lamp a"}
  - {name: "lamp b"}
sockets:
  - {name: "socket a"}
devices: !JoinLists lamps sockets
---

Did I miss something or is there currently no way to do this?

Hi,

I would like to propose using entry_points as another way of finding and loading custom tags.

This makes sharing and using tag implementations easier - avoids the need to specify them on every emrichen call (with -m mytag) and users can just pip install the package with the tag implementation (effectively becoming plugins)

If maintainers agree with the feature I would be happy to open a PR

Template:

!Defaults
x:
  y: 5
z: !Var x
---
workie: !Lookup x.y
no_workie: !Lookup z.y

Expected output:

workie: 5
no_workie: 5

Actual output:

Traceback (most recent call last):
  File "/usr/local/bin/emrichen", line 11, in <module>
    load_entry_point('emrichen', 'console_scripts', 'emrichen')()
  File "/Users/sapaj1/Hobby/emrichen/emrichen/__main__.py", line 111, in main
    output = template.render(context, format=args.output_format)
  File "/Users/sapaj1/Hobby/emrichen/emrichen/template.py", line 22, in render
    enriched = self.enrich(context)
  File "/Users/sapaj1/Hobby/emrichen/emrichen/template.py", line 19, in enrich
    return context.enrich(self.template)
  File "/Users/sapaj1/Hobby/emrichen/emrichen/context.py", line 37, in enrich
    item = self.enrich(item)
  File "/Users/sapaj1/Hobby/emrichen/emrichen/context.py", line 30, in enrich
    val = self.enrich(val)
  File "/Users/sapaj1/Hobby/emrichen/emrichen/context.py", line 21, in enrich
    return self.enrich(value.enrich(self))
  File "/Users/sapaj1/Hobby/emrichen/emrichen/tags/lookup.py", line 19, in enrich
    raise KeyError(f'{self}: no matches for {self.data}')
KeyError: "Lookup('z.y'): no matches for z.y"

CloudFormation has its own functions such as !Subst and !Join etc. In order to be able to input and output CloudFormation templates, Emrichen needs a way to co-operate with foreign tags.

(Bad) options:

0. Keep disallowing foreign tags and require CloudFormation users to use the { "Fn::Join": […] } form of CFN functions instead of the tag-based !Join form
1. Prefix our own tags with some prefix, reducing the risk of collision, and pass foreign tags to the output as-is.
2. Require foreign tags to be prefixed and strip the prefix upon processing

sorry if this is just pebcak but .. i tried for 2 hours now and cant get anything to work.. i am trying to do something similar to #42 where my input is

!Defaults
butane_version: 1.1.0
---
variant: flatcar
version: !Var butane_version

storage:
  links: 
    !Include "links-ALL.yaml"
    !Include "links-docker_compose.yaml"
    - path: /tmp/test
      target: /tmp/blubb
      hard: false

links-ALL.yaml:

---
- path: /etc/localtime
  overwrite: true
  target: /usr/share/zoneinfo/Europe/Vienna
- path: /etc/systemd/system/multi-user.target.wants/docker.service
  target: /usr/lib/systemd/system/docker.service
  hard: false
  overwrite: true

links-docker_compose.yaml:

---
- path: /etc/extensions/docker_compose.raw
  target: /opt/extensions/docker_compose/docker_compose.raw
  hard: false

the expected outcome would be (reduced to relevant part):

storage:
  links: 
  - path: /etc/localtime
    overwrite: true
    target: /usr/share/zoneinfo/Europe/Vienna
  - path: /etc/systemd/system/multi-user.target.wants/docker.service
    target: /usr/lib/systemd/system/docker.service
    hard: false
    overwrite: true
  - path: /etc/extensions/docker_compose.raw
    target: /opt/extensions/docker_compose/docker_compose.raw
    hard: false
    - path: /tmp/test
      target: /tmp/blubb
      hard: false

however, regardless of what i try, nothing seems to work.
i tried various combinations of !Merge and !Concat and !IncludeGlob (with a non wildcard specific filename) but all of it just throws various errors..

can someone enlighten me what the right way to do this would be? :) much appreciated...

Consider the following files:

# a.yaml
a: 1
b: 2

# a.in.yaml – template
!Merge
  - !Include a.yaml
  - !Include a.update.yaml

# a.update.yaml – an incremental update to a.yaml
b: 3

Now, if you run emrichen --output-file a.yaml a.in.yaml, the resulting a.yaml will only contain

b: 3

instead of

a: 1
b: 3

because --output-file a.yaml is opened at application startup with w mode, truncating the file, and !Include a.yaml enriches to {}.

I would like to change this so that !Include a.yaml here would enrich to the original contents of a.yaml, and a.yaml would be opened for writing only after enriching the template.

I just had a Loop replace the german umlaut "ü" with \xFC which is ugly.

When input is

image: !Format "{replaceme_with_var} {{dont_replaceme}}"

Should output as:

image: foo {dont_replaceme}

is there support for this?

Use cases:

• If the condition is met, return this value, otherwise return that value
• If the condition is met, define this key in a dict, otherwise leave it out
• If the condition is met, include this element in a list, otherwise leave it out

Simple conditions (Python truthiness of a value) are easy to implement. How about complex predicates (==, <, <=, >, >= etc.)?

Switching to a JSONPath implementation that supports ? is one option. Then we could just use !Lookup in the condition.

We do not want to execute Python to resolve the condition.

Have the YAML parser use OrderedDict instead of dict. Our JSON parser already does this.

Hi,

I've noticed that the typecheck functions are always returning false on my install.

I'm running Ubuntu 22.04 with Python 3.10.12 and have installed emrichen 0.4.0 via pip install emrichen at version.

# x.yml
!Defaults

number: 1234
dict:
  key: 4321

---
test_number_isnumber: !IsNumber number
test_dict_isdict: !IsDict dict

test_number_isinteger: !IsInteger number

test_number_isstring: !IsString number
test_dict_isstring: !IsString dict

$ emrichen < x.yml 
> test_number_isnumber: false
> test_dict_isdict: false
> test_number_isinteger: false
> test_number_isstring: true
> test_dict_isstring: true

It was suggested the template file could include defaults for variables if the author so desires.

Perhaps it could leverage multi-document YAML files and look like this:

!Defaults
  foo: bar
  quux: 5
---
this_is_foo: !Var foo
also_quux: !Var quux

Output:

this_is_foo: bar
also_quux: 5

The document containing !Defaults may contain only that, it does not matter where in the document !Defaults is specified, there may be multiple documents containing !Defaults, latest wins, and any document containing !Defaults is erased from the output.

is this tool yaml-injection safe?

I have recently been using emrichen to template yml concourse configuration files but I have run into a snag with !If.
I want to be able to omit a value in a list based on a variable. Here is the file I'm trying to template:

resources:
  - name: gitflow-test-git
    type: git
    icon: github
    source:
      uri: <SNIP>
      branch: master
      username: concourse
      password: <SNIP>

jobs:
  - name: build
    public: true
    plan:
      - get: gitflow-test-git
        trigger: true
      - task: build
        file: gitflow-test-git/ci/build.yml
        params:
          VERSION: !Var VERSION
          VERSION_MAJOR: !Var VERSION_MAJOR
          VERSION_MINOR: !Var VERSION_MINOR
          VERSION_PATCH: !Var VERSION_PATCH
          BRANCH: !Var BRANCH

      !If
        test: !Op [master, eq, !Var BRANCH]
        then:
          - task: deploy-prod
            file: gitflow-test-git/ci/deploy-prod.yml
        else: !Void

Ignoring the concourse jargon I want to be able to omit the deploy-prod task when branch resolves to non-master. I have been able to get the !If block to work when it is the only thing in the file. Eg:

echo "!If
           test: !Op [master, eq, !Var BRANCH]
           then:
             - task: deploy-prod
               file: gitflow-test-git/ci/deploy-prod.yml
           else: !Void" | emrichen --define "BRANCH=master"

resolves correctly to

- task: deploy-prod
  file: gitflow-test-git/ci/deploy-prod.yml

or

This may or not be a bug, I am somewhat new to yml and even newer to emrichen so please let me know the correct usage to achieve my desired functionality.

The hash functions do not allow for chained tags.

There is probably a call to context.enrich(...) missing at

Other tags like Base64 allow for chaining tags.

collections:
  !Include ./configs/card1.yaml

Should indent each line of card1.yaml by 2, but it doesn't, which makes invalid YAML out of two valid YAMLs, right?

The name emrichen is easily confused with the very popular emscripten project. This project name is sharing the same start, middle character, and ending of the other word, which makes differentiating the two more difficult than it should be. Please consider a different name to make recall and reference of this project easier for everybody.

Thank you for releasing this tool.

Would it be possible to make something like this work without a Max recursion error?

#original.yaml
!Defaults
name: John
---
greeting: !Format "Hi {name}!" 


#composed.yaml
!With
template: !Include original.yaml
vars: 
    name: !Format "{name} and Mark" 

I assume it is because we are referring to name in the definition of name: in composed.yaml, but I would expect that it would use the then already existing John default from original.yaml, resulting in "Hi John and Mark"

I'm trying to use emrichen as a preprocessor for HomeAssistant configuration files.
I need to be able to generate YAML-code like this:

homeassistant:
  customize:
    light.asdf:
      hidden: true
    light.foobar:
      hidden: true

My current emrichen template looks like this:

!Defaults
lamps:
  - {name: asdf, id: 123}
  - {name: foobar, id: 456}
---
homeassistant:
  customize:
	!Loop2
      over: !Var lamps
      as: lamp
      template:
        !Format "light.{lamp.name}":
          hidden: true

Is there a way to achieve this?

Consider this, a.yml:

a: 1

And b.yml:

b: 2

Template ab.yml:

ab:
  !Include 1.yml
  !Include 2.yml

emrichen -f ab.yml is stuck and hangs.

But expected, ab.yml:

ab:
   a: 1
   b: 2

It might be related to this #20

Looking to use templates to generate templates, but the challenge is supporting the yaml serialization of those that implement BaseTag, especially with nested expressions. Not sure if it makes sense to adopt Yamlable for better serialization support.

Eg.

!Loop
  over: domain_names
  as: domain_name
  index_as: domain_index
  template:
    apiVersion: v1
    kind: DomainName
    metadata:
      name: !Var domain_name
      …

Always produces a list.

Perhaps with an explicit tag like

!Format "{repository}:{tag}"

Or even make all string literals Jinja2 templates? It would certainly be pragmatic, but it would also reduce the ideological purity of the tool greatly.

Hello,

Thank you for this tool, it's awesome.

Is there a way to reference the full list of Defaults to achieve something like:

apiVersion: v1
kind: ConfigMap

metadata:
  namespace: !Var namespace
  name: !Var name

data: !Var AllTheDefaults

I've been trying

• !LookupAll "$" -- this should be the reference to the root dict but it errors out with "maximum recursion depth exceeded"
• !LookupAll "*" -- this would likely create a new list of values from the root dict (I want the names too)
• !Index {over: !LookupAll "$", ...} -- maximul recursion again

Hey guys,

Any plans to release it to pypi?

Thanks!

This appears, at least with the limited testing I've done, to remove comments from my template files. This wouldn't be such a big deal, but there is one specific use case where it is, which is making the user-data file for cloud-init images. The first line of the file, must be #cloud-config. I am not sure if this is a packer requirement, or cloud init or AutoInstall, but it needs to be there none the less. Is it possible to ensure that comments are left in the output file?

i am trying to render multiple deployments based on a variable

vars.yaml

backend_mutations:
  - name: backend
    command: /run.sh
    replicas: 1
  - name: backend-editors
    command: /run.sh
    replicas: 1
  - name: backend-cron
    command: /cron.sh
    replicas: 1

template.yaml
(redacted the rest of the deployment spec.)

!Loop
  over: !Var backend_mutations
  as: backend
  index_as: id
  template:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: !Lookup backend.name
      labels:
        krn: !Lookup backend.name

the problem is that !Loop produces a list. which is not apply'able to K8s.

ideally i would like to generate a single output file with multiple yaml files in it. seperated by ---

like my supposed output would be like:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  labels:
    krn: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-editors
  labels:
    krn: backend-editors
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-cron
  labels:
    krn: backend-cron
---

is there anything i am missing - or is it currently not possible?

I ran into an issue where I was trying to make a kubernetes manifest for a volumeMount and I wanted to make it configurable whether or not the volume was readOnly. I'm just using --include-env which means that when I provide the environment variable false it gets turned into the string 'false'.

I ended up doing

!Defaults
read_only_bool: !Op [!Var read_only, eq, 'true']

But it seems like it would be nice to have like a

read_only_bool: !Coerce [Boolean, !Var read_only]

That could apply standard yaml type guessing to the string and throw an error if it wasn't right (e.g. if you provided read_only=5 that would be a number and so would throw an error, or if you Coerced to numeric then you would end up with an error if you provided the string 'false'.)

Hi,

Considering a template like

# generic.yaml
!Defaults
name: Unknown name
greeting: Unknown greeting
---
phrase: !Format "{greeting} {name}"

Which is templated once, with results as expected (phrase: Hello Mark):

# hello_mark.yaml
!With
template: generic.yaml
vars:
    name: Mark
    greeting: Hello

and then another template on top of the previous one

# hello_john.yaml
!With
template: hello_mark.yaml
vars:
    name: John

I expected it to result in phrase: Hello John but rather I got still phrase: Hello Mark. So it seems that the first (lower level) variables within the !With tag cannot be overriden by higher level !With templates.

Is this by design? Thanks!

Currently the argument to !Format is passed as-is to Python .format_map(). This means you can do {foo.__class__.__name__} but not {person.first_name}.

Implement our own format string parser that processes JSONPath expressions from within {}.

$ emrichen postgres/secret.local.yaml | kc apply -n outline-ropecon -f -
Traceback (most recent call last):
  File "/Users/japsu/Hobby/venv3.10-emrichen/bin/emrichen", line 33, in <module>
    sys.exit(load_entry_point('emrichen', 'console_scripts', 'emrichen')())
  File "/Users/japsu/Hobby/emrichen/emrichen/__main__.py", line 35, in main
    args.output_file.seek(0)
io.UnsupportedOperation: underlying stream is not seekable
error: no objects passed to apply

Regression in #55 #54

hi - struggling again, asking for help :)

  systemd:
    units: !Concat
      - !Include units/ALL.yaml
      - !Include units/docker_compose.yaml
      - !Include units/mount_docker.yaml
      - !Include units/mount_data.yaml
      - !If
        test: !Op [!Void, ne, !Var etcd_token]
        then: !Include units/etcd.yaml

i want to check if one of the "parent" files set the variable etcd_token - and if so, include the file.. is this doable?
TIA!