tanbro / pyyaml-include Goto Github PK

View Code? Open in Web Editor NEW

77.0 4.0 18.0 398 KB

yaml include other yaml

Home Page: https://pypi.org/project/pyyaml-include/

License: GNU General Public License v3.0

Python 98.22% Shell 1.78%

yaml pyyaml

pyyaml-include's Issues

How to make the package work with setuptools lower than 61.0.0

I'm building the package from scratch.

For some reason, I work on a system with an old setuptools which does not support PEP621 and is unable to extract name and version metadata in pyproject.toml as pypa/setuptools#3269 (comment).

Is there any way to achieve the compatibility with the old setuptools? Thanks!

is it currently supported to set an anchor for the included mapping?

I was trying to have some inheritance, by including the shared yaml-file:

`
file1: !include include.d/1.yaml & shared

section1:
<<: *shared
foo: bar
`

readers_map not getting picked up: RuntimeError: Un-supported file name

I have the following code:

from yamlinclude import YamlIncludeConstructor

readers = {"yaml": "YamlReader", "yml": "YamlReader", "j2": "YamlReader", "yaml.j2": "YamlReader"}

YamlIncludeConstructor(reader_map=readers).add_to_loader_class(loader_class=yaml.FullLoader)

# Read the main yaml file and include the other yaml files.
with open(file_path) as f:
    data = yaml.load(f.read(), Loader=yaml.FullLoader)

I'm attempting to read files named "filename.yaml.j2". With the above code, I'm still getting the:
RuntimeError: Un-supported file name

What am I doing wrong here?

Could you update the PyPI to new version to add PyYAML 6.0 support?

Thanks for the good plugin to support include YAML file, it's very useful.

But the latest version in https://pypi.org/project/pyyaml-include/, is still the old version, which meets the problem to support PyYAML6.0, could you update it to a new version? Thanks.

Yaml anchor support for include yaml files

Hi tanbro, I thought it would be nice if this library also supported anchors when including a yaml file so I wrote this PR #32.

Ability for default value

Ability to use default value if file doesn't exist.

Example:

Somelist: !include { pathname: path.yaml, default: [] }

two files in one dict?

I'm trying to get it mapped with more files, but it doesn't work for me.

I have

file foo-1.yml
one: a
file foo-2.yml
two: b

using

foo: 
  !include foo-1.yml

Works fine, I get a {'foo' :{one: a}} - thats what I want. But adding another line, like

foo:
  !include foo-1.yml
  !include foo-2.yml

gives an error: FileNotFoundError: [Errno 2] No such file or directory: '/home/martin/Projects/toki/toki/data/yaml/items-area-1.yml !include items-area-2.yml'

And using

foo:
  !include foo-*.yml

makes a list of two dicts..
what can I doo to have just {foo: {'one': a, 'two': b}} as a result here?

Include and achors

I've got the following files:

# main.yaml
included: &included
    !include include.yaml

rendering:
  - << : *included

and

# include.yaml
-
    output: ignore.txt
    template: ignore.tpl

The following code doesn't work:

In [1]: import yaml 
   ...: from yamlinclude import YamlIncludeConstructor                                                                                                                       

In [2]: YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='/tmp')                                                                            
Out[2]: <yamlinclude.constructor.YamlIncludeConstructor at 0x7f4c3adcfbb0>

In [3]: import pathlib                                                                                                                                                       

In [4]: data = yaml.load(pathlib.Path('/tmp/main.yaml').open(), Loader=yaml.FullLoader)                                                                                      
---------------------------------------------------------------------------
ConstructorError                          Traceback (most recent call last)
<ipython-input-4-f98f7c4c3bd9> in <module>
----> 1 data = yaml.load(pathlib.Path('/tmp/main.yaml').open(), Loader=yaml.FullLoader)

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/__init__.py in load(stream, Loader)
    112     loader = Loader(stream)
    113     try:
--> 114         return loader.get_single_data()
    115     finally:
    116         loader.dispose()

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in get_single_data(self)
     49         node = self.get_single_node()
     50         if node is not None:
---> 51             return self.construct_document(node)
     52         return None
     53 

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_document(self, node)
     58             self.state_generators = []
     59             for generator in state_generators:
---> 60                 for dummy in generator:
     61                     pass
     62         self.constructed_objects = {}

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_yaml_map(self, node)
    414         data = {}
    415         yield data
--> 416         value = self.construct_mapping(node)
    417         data.update(value)
    418 

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_mapping(self, node, deep)
    218     def construct_mapping(self, node, deep=False):
    219         if isinstance(node, MappingNode):
--> 220             self.flatten_mapping(node)
    221         return super().construct_mapping(node, deep=deep)
    222 

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in flatten_mapping(self, node)
    205                         merge.extend(value)
    206                 else:
--> 207                     raise ConstructorError("while constructing a mapping", node.start_mark,
    208                             "expected a mapping or list of mappings for merging, but found %s"
    209                             % value_node.id, value_node.start_mark)

ConstructorError: while constructing a mapping
  in "/tmp/main.yaml", line 5, column 5
expected a mapping or list of mappings for merging, but found scalar
  in "/tmp/main.yaml", line 1, column 11

The library looks to by typed, but the `py.typed` file is missing

See: https://peps.python.org/pep-0561/

Wildcard with Hash/Dict/Map

More of a question, but also a request if it is possible. Can i include wildcard and instead of a list fill a object with filename (with/without suffix) as the key and loaded value as the value?

files: !include {path_name: include.d/*.yml, map: full_name/without_prefix/custom_function}

files: !include {path_name: include.d/*.yml, map: without_prefix}

This should be:

files:
  1:
    name: 1
 2:
    name: 2

files: !include {path_name: include.d/*.yml, map:  true}

This should be:

files:
  1.yaml:
    name: 1
 2.yaml:
    name: 2

def custom_function(file:str, value:any):
  return Path(file).name

How to concatenate two yaml files?

Hello, I want to concatenate two yaml files, what should I do?

for example, I have two different yaml files,

# file1.yaml
name: 1

# file2.yaml
class: 2

I want to concatenate these two files into one file after using pyyaml-include, as shown below

# merge_file.yaml
name: 1
class: 2

I would like to ask, what should I do to get this result?

Thank you very much.

Multiple YAML documents in a single file

Is there support forr !include a single yaml file that has three yaml documents? A new document in YAML starts with:

Example

---
name: doc-a

---
name: doc-b

---
name: doc-c

This would be extremely useful if it is not supported. I read through the docs but did not see that this feature is supported.

If it is supported, could you provide an example?

It seems that I can't rewrite the inherited argments from base `yaml` file.

When I use a test case like:
base.yaml:

# base.yaml
name: John Doe
age: 30
address:
  city: New York
  state: NY

derived.yaml:

# derived.yaml
!include base.yaml
age: 35
address:
  state: CA

test.py

import yaml
from yamlinclude import YamlIncludeConstructor

YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='./')

with open('derived.yaml') as f:
    data = yaml.load(f, Loader=yaml.FullLoader)

print(data)

I expect the output is {'name': 'John Doe', 'age': 35, 'address': {'city': 'New York', 'state': 'CA'}} but there is a error occured. The traceback is as follows:

Traceback (most recent call last)
[d:\SelfDatas\SJTU\Projects\Labs\yaml\read_yaml.ipynb](file:///D:/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb) Cell 1 in 7
      [4](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=3) YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='[./](https://file+.vscode-resource.vscode-cdn.net/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/)')
      [6](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=5) with open('derived.yaml') as f:
----> [7](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=6)     data = yaml.load(f, Loader=yaml.FullLoader)
      [9](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=8) print(data)

File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\__init__.py:81, in load(stream, Loader)
     79 loader = Loader(stream)
     80 try:
---> 81     return loader.get_single_data()
     82 finally:
     83     loader.dispose()

File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\constructor.py:49, in BaseConstructor.get_single_data(self)
     47 def get_single_data(self):
     48     # Ensure that the stream contains a single document and construct it.
---> 49     node = self.get_single_node()
     50     if node is not None:
     51         return self.construct_document(node)

File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\composer.py:36, in Composer.get_single_node(self)
     34 document = None
     35 if not self.check_event(StreamEndEvent):
...
    583 # the parser.
    584 if not self.flow_level:

ScannerError: mapping values are not allowed here
  in "derived.yaml", line 4, column 4

Does not support top level include with other keys

Here is my setup:

PyYAML: 5.1
pyyaml-include: 1.1.1
python: 2.7.15

The python script (from the README):

import yaml
from yamlinclude import YamlIncludeConstructor

YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader)

with open('0.yml') as f:
     data = yaml.load(f, Loader=yaml.FullLoader)

print(data)

0.yml:

!include 1.yml
foo: bar

1.yml:

name: "1"

It seems that mixing toplevel include and other value is not (yet?) supported.

I got this error:

Traceback (most recent call last):
  File "../test.py", line 32, in <module>
    data = yaml.load(f, Loader=yaml.FullLoader)
  File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 43, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python2.7/site-packages/yaml/composer.py", line 58, in compose_document
    self.get_event()
  File "/usr/local/lib/python2.7/site-packages/yaml/parser.py", line 118, in get_event
    self.current_event = self.state()
  File "/usr/local/lib/python2.7/site-packages/yaml/parser.py", line 193, in parse_document_end
    token = self.peek_token()
  File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 129, in peek_token
    self.fetch_more_tokens()
  File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 579, in fetch_value
    self.get_mark())
yaml.scanner.ScannerError: mapping values are not allowed here
  in "0.yml", line 2, column 4

Python 3.10 unsopported?

Hello, I have been using pyyaml-include with Python3.8 and it worked well, but when I install the package on python3.10, the python source files are empty?

I get this error:
" from yamlinclude import YamlIncludeConstructor
ImportError: cannot import name 'YamlIncludeConstructor' from 'yamlinclude' (/usr/local/lib/python3.10/dist-packages/yamlinclude/init.py)"

The files "constructor.py", "readers.py", "version.py" inside "/usr/local/lib/python3.10/dist-packages/yamlinclude" are empty

Use tag with anchors across a single yaml config file

Hi,

I've started playing around with this interesting library and I have a question. I'm not sure if this can be an issue or maybe I missed the point. So let me explain what I want to build.

My environment:
python=3.10.10
PyYAML=6.0
pyyaml-include=1.3

What I've tried so far:
I want to create some defaults that would be use across the YAML config files without need to copy it multiple times in different blocks with possibly with anchors. So, if I have /test/path/custom-defaults/person.yaml file:

.default_description: &default-description
    hair_color: grey
    eye_color: blue
    age: 123
    gender: N/A

And I'd like to include it in other config files, e.g. in /test/path/defaults/describe-people.yaml:

!include ../custom-defaults/person.yaml

persons:
   person.with.custom.defaults: 
       <<: *default-description
   person.with.overrided.defaults: 
       <<: *default-description
       hair_color: blue

When try this I'm getting an error: raise ParserError(None, None, yaml.parser.ParserError: expected '<document start>', but found '<block mapping start>'

On the other side, I can fix it by:

Updating person.yaml file a bit, by removing anchors
Use !include tag only in blocks, like person.with.custom.defaults: !include ../custom-defaults/person.yaml

Questions:
Is it even possible to use it as a document tag and if yes, can someone provide an example?

Thank you.

Recursive includes do not work with nested directories

Here is the test case:

        os.makedirs("foo/bar")
        with open("foo.yml","w") as fd:
            fd.write("!include foo/bar.yml\n")
        with open("foo/bar.yml","w") as fd:
            fd.write("!include bar/zot.yml\n")
        with open("foo/bar/zot.yml","w") as fd:
            fd.write("foo: 42\n")
        YamlIncludeConstructor.add_to_loader_class(yaml.FullLoader)
        with open("foo.yml") as fd:
            self.assertEqual({"foo":42}, yaml.load(fd, Loader=yaml.FullLoader))
        os.remove("foo/bar/zot.yml")
        os.remove("foo/bar.yml")
        os.remove("foo.yml")
        os.removedirs("foo/bar")

it would work if I write the full path foo/bar/zot.yml into foo/bar.yml, but this is counterproductive (what if bar.yml is included from another place or used separately).

Would it be possible for include to take the path relative to the directory of the node that does the include? (cf. https://gist.github.com/joshbode/569627ced3076931b02f).
Or maybe it is already possible?
Thanks!

Publish sdist to PyPI

Could you please publish 1.2.post2 sdist to PyPI just like 1.2 (pyyaml-include-1.2.tar.gz)? Thanks.

Load multiple documents

If I have one document such as (foo.yaml):

!include ./baz.yaml
!include ./baz.yaml

and baz.yaml looks like:

baz1: x
---
baz2: y

I would like this to be interpreted as:

baz1: x
---
baz2: y
---
baz1: x
---
baz2: y

Is there any way to achieve this? Currently, even if I use load_all() my current setup fails with error.

Can we use alias with include?

For example:

something like:

vars: &vars !inc test.yaml

x:
<< *vars

Does not support top level include with other keys

According to this answer

Till now, !include constructor can NOT be used at the TOP level, unless it's the only one node in whole YAML document.

The question is why?

Idea: include from remote storages (notably git)

Hi,
I like this project very much, this is exactly what I need for one of mine.
The only thing I would miss is ability to include from remote storage types (especially git).
I thought I would ask whether you would consider this to be a good idea before I'll start programming something by my own or before making a pull request here if you would agree.

I also might appreciate comments support that is present in ruamel. How feasible do you think it would be to port your library to ruamel?

Thanks a lot,
kind regards
Roman

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Python 3.6 unsupported after releasing `1.2.post1`

Hi,
re.Pattern is Python 3.7+

On your latest release, 1.2.post1, you've included:

...
  File "/usr/local/lib/python3.6/site-packages/yamlinclude/__init__.py", line 7, in <module>
    from .constructor import *
  File "/usr/local/lib/python3.6/site-packages/yamlinclude/constructor.py", line 24, in <module>
    class YamlIncludeConstructor:
  File "/usr/local/lib/python3.6/site-packages/yamlinclude/constructor.py", line 46, in YamlIncludeConstructor
    reader_map: Collection[Tuple[Union[str, re.Pattern], Reader]] = None
AttributeError: module 're' has no attribute 'Pattern'

I guess either by mistake, or you removed support for 3.6 and didn't include that in your release notes anywhere :)

FYI

yaml writer?

This is an interesting project that I'd like to integrate into my tools. I'd like to know if there's any support or interest for a writer class that respects includes.

As an example I'll refer to the first example in the README. Once the data has been read and potentially changed I may want to change the value of file2 name to be "3".

When I use yaml to dump the new dictionary to disk it will ignore the file structure and overwrite the values into one file.

I'd like to know if there is a way to handle this such that the writer respects the existing structure. If not, is that something this project would be interested in?

Relative include: relativeto the current file?

I think (IMHO) it would make more sense if relative include searches for the file in the current file's directory rather than current working directory.

Use case: when I pass config files in the command line I might have them in different directories. And if they're independent - they could load their own local extensions.

The solution to not break backward compatibility would be, for example to add a new parameter to the constructor or to initialize base_dir=None instead of default empty string.

EDIT: perhaps this is not possible to get information about current file from the node object passed to the constructor?
But as soon as we can redefine reader - I see that yaml.Reader class can handle stream names for file-like objects.

include specific components from other yamls?

Does this package let me include specific components from other yamls? For cases where I don't want to include the entire content of the yaml file. E.g something like

a.yaml

param:
  - name: A
    type: A1
  - name: B
    type: B1

stuff: 123

b.yaml

other:
   - param: !include a.yaml/param[0]
  
stuff: !include a.yaml/stuff

such that reading in b.yaml would result in

{
 "other": [{"param": {"name": 'A', "type": 'A1'}}]
 "stuff": 123 
}

Jinja2 support

Hello,

How likely are you to support jinja templates?

I have YAML templates that have jina logic in them. For example:

 containers:
    - name: "{{application_name}}"
      image: "{{business_unit}}-docker-{{environment}}.artifactory.svc.aws.gartner.com/{{application_name}}:{{version}}"
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 8080
          protocol: TCP
      resources:
        limits:
          memory: {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}

The last line of the code, is a jinja2 expression.

The probably is, I run the template through the pyyaml-include, to get a full single document from multiple templates. Then, I run the single template through the template engine.

If this:

memory:" {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}"

when pyyaml-include operates on this line, you get:

memory: '{{ vars.eks.memory_limits if vars.eks.memory_limits else ''512mb'' }}'

And the jinja2 engine can't understand it. What jinja2 is expecting is this:

memory: {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}

Are you willing to support a jinja2 template language?

Thanks

tanbro / pyyaml-include Goto Github PK

pyyaml-include's Issues

Recommend Projects

Recommend Topics

Recommend Org