tanbro / pyyaml-include Goto Github PK
View Code? Open in Web Editor NEWyaml include other yaml
Home Page: https://pypi.org/project/pyyaml-include/
License: GNU General Public License v3.0
yaml include other yaml
Home Page: https://pypi.org/project/pyyaml-include/
License: GNU General Public License v3.0
According to this answer
Till now, !include constructor can NOT be used at the TOP level, unless it's the only one node in whole YAML document.
The question is why?
This is an interesting project that I'd like to integrate into my tools. I'd like to know if there's any support or interest for a writer class that respects includes.
As an example I'll refer to the first example in the README. Once the data has been read and potentially changed I may want to change the value of file2 name to be "3".
When I use yaml to dump the new dictionary to disk it will ignore the file structure and overwrite the values into one file.
I'd like to know if there is a way to handle this such that the writer respects the existing structure. If not, is that something this project would be interested in?
The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.
Thanks for the good plugin to support include YAML file, it's very useful.
But the latest version in https://pypi.org/project/pyyaml-include/, is still the old version, which meets the problem to support PyYAML6.0, could you update it to a new version? Thanks.
When I use a test case like:
base.yaml:
# base.yaml
name: John Doe
age: 30
address:
city: New York
state: NY
derived.yaml:
# derived.yaml
!include base.yaml
age: 35
address:
state: CA
test.py
import yaml
from yamlinclude import YamlIncludeConstructor
YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='./')
with open('derived.yaml') as f:
data = yaml.load(f, Loader=yaml.FullLoader)
print(data)
I expect the output is {'name': 'John Doe', 'age': 35, 'address': {'city': 'New York', 'state': 'CA'}}
but there is a error occured. The traceback is as follows:
Traceback (most recent call last)
[d:\SelfDatas\SJTU\Projects\Labs\yaml\read_yaml.ipynb](file:///D:/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb) Cell 1 in 7
[4](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=3) YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='[./](https://file+.vscode-resource.vscode-cdn.net/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/)')
[6](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=5) with open('derived.yaml') as f:
----> [7](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=6) data = yaml.load(f, Loader=yaml.FullLoader)
[9](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=8) print(data)
File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\__init__.py:81, in load(stream, Loader)
79 loader = Loader(stream)
80 try:
---> 81 return loader.get_single_data()
82 finally:
83 loader.dispose()
File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\constructor.py:49, in BaseConstructor.get_single_data(self)
47 def get_single_data(self):
48 # Ensure that the stream contains a single document and construct it.
---> 49 node = self.get_single_node()
50 if node is not None:
51 return self.construct_document(node)
File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\composer.py:36, in Composer.get_single_node(self)
34 document = None
35 if not self.check_event(StreamEndEvent):
...
583 # the parser.
584 if not self.flow_level:
ScannerError: mapping values are not allowed here
in "derived.yaml", line 4, column 4
Hi tanbro, I thought it would be nice if this library also supported anchors when including a yaml file so I wrote this PR #32.
I'm building the package from scratch.
For some reason, I work on a system with an old setuptools which does not support PEP621 and is unable to extract name and version metadata in pyproject.toml as pypa/setuptools#3269 (comment).
Is there any way to achieve the compatibility with the old setuptools? Thanks!
I've got the following files:
# main.yaml
included: &included
!include include.yaml
rendering:
- << : *included
and
# include.yaml
-
output: ignore.txt
template: ignore.tpl
The following code doesn't work:
In [1]: import yaml
...: from yamlinclude import YamlIncludeConstructor
In [2]: YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='/tmp')
Out[2]: <yamlinclude.constructor.YamlIncludeConstructor at 0x7f4c3adcfbb0>
In [3]: import pathlib
In [4]: data = yaml.load(pathlib.Path('/tmp/main.yaml').open(), Loader=yaml.FullLoader)
---------------------------------------------------------------------------
ConstructorError Traceback (most recent call last)
<ipython-input-4-f98f7c4c3bd9> in <module>
----> 1 data = yaml.load(pathlib.Path('/tmp/main.yaml').open(), Loader=yaml.FullLoader)
~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/__init__.py in load(stream, Loader)
112 loader = Loader(stream)
113 try:
--> 114 return loader.get_single_data()
115 finally:
116 loader.dispose()
~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in get_single_data(self)
49 node = self.get_single_node()
50 if node is not None:
---> 51 return self.construct_document(node)
52 return None
53
~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_document(self, node)
58 self.state_generators = []
59 for generator in state_generators:
---> 60 for dummy in generator:
61 pass
62 self.constructed_objects = {}
~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_yaml_map(self, node)
414 data = {}
415 yield data
--> 416 value = self.construct_mapping(node)
417 data.update(value)
418
~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_mapping(self, node, deep)
218 def construct_mapping(self, node, deep=False):
219 if isinstance(node, MappingNode):
--> 220 self.flatten_mapping(node)
221 return super().construct_mapping(node, deep=deep)
222
~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in flatten_mapping(self, node)
205 merge.extend(value)
206 else:
--> 207 raise ConstructorError("while constructing a mapping", node.start_mark,
208 "expected a mapping or list of mappings for merging, but found %s"
209 % value_node.id, value_node.start_mark)
ConstructorError: while constructing a mapping
in "/tmp/main.yaml", line 5, column 5
expected a mapping or list of mappings for merging, but found scalar
in "/tmp/main.yaml", line 1, column 11
More of a question, but also a request if it is possible. Can i include wildcard and instead of a list fill a object with filename (with/without suffix) as the key and loaded value as the value?
files: !include {path_name: include.d/*.yml, map: full_name/without_prefix/custom_function}
files: !include {path_name: include.d/*.yml, map: without_prefix}
This should be:
files:
1:
name: 1
2:
name: 2
files: !include {path_name: include.d/*.yml, map: true}
This should be:
files:
1.yaml:
name: 1
2.yaml:
name: 2
def custom_function(file:str, value:any):
return Path(file).name
Hi,
re.Pattern
is Python 3.7+
On your latest release, 1.2.post1, you've included:
...
File "/usr/local/lib/python3.6/site-packages/yamlinclude/__init__.py", line 7, in <module>
from .constructor import *
File "/usr/local/lib/python3.6/site-packages/yamlinclude/constructor.py", line 24, in <module>
class YamlIncludeConstructor:
File "/usr/local/lib/python3.6/site-packages/yamlinclude/constructor.py", line 46, in YamlIncludeConstructor
reader_map: Collection[Tuple[Union[str, re.Pattern], Reader]] = None
AttributeError: module 're' has no attribute 'Pattern'
I guess either by mistake, or you removed support for 3.6 and didn't include that in your release notes anywhere :)
FYI
I think (IMHO) it would make more sense if relative include searches for the file in the current file's directory rather than current working directory.
Use case: when I pass config files in the command line I might have them in different directories. And if they're independent - they could load their own local extensions.
The solution to not break backward compatibility would be, for example to add a new parameter to the constructor or to initialize base_dir=None
instead of default empty string.
EDIT: perhaps this is not possible to get information about current file from the node object passed to the constructor?
But as soon as we can redefine reader - I see that yaml.Reader
class can handle stream names for file-like objects.
Is there support forr !include a single yaml file that has three yaml documents? A new document in YAML starts with:
Example
---
name: doc-a
---
name: doc-b
---
name: doc-c
This would be extremely useful if it is not supported. I read through the docs but did not see that this feature is supported.
If it is supported, could you provide an example?
I have the following code:
from yamlinclude import YamlIncludeConstructor
readers = {"yaml": "YamlReader", "yml": "YamlReader", "j2": "YamlReader", "yaml.j2": "YamlReader"}
YamlIncludeConstructor(reader_map=readers).add_to_loader_class(loader_class=yaml.FullLoader)
# Read the main yaml file and include the other yaml files.
with open(file_path) as f:
data = yaml.load(f.read(), Loader=yaml.FullLoader)
I'm attempting to read files named "filename.yaml.j2". With the above code, I'm still getting the:
RuntimeError: Un-supported file name
What am I doing wrong here?
Hi,
I like this project very much, this is exactly what I need for one of mine.
The only thing I would miss is ability to include from remote storage types (especially git).
I thought I would ask whether you would consider this to be a good idea before I'll start programming something by my own or before making a pull request here if you would agree.
I also might appreciate comments support that is present in ruamel. How feasible do you think it would be to port your library to ruamel?
Thanks a lot,
kind regards
Roman
Hello, I have been using pyyaml-include with Python3.8 and it worked well, but when I install the package on python3.10, the python source files are empty?
I get this error:
" from yamlinclude import YamlIncludeConstructor
ImportError: cannot import name 'YamlIncludeConstructor' from 'yamlinclude' (/usr/local/lib/python3.10/dist-packages/yamlinclude/init.py)"
The files "constructor.py", "readers.py", "version.py" inside "/usr/local/lib/python3.10/dist-packages/yamlinclude" are empty
Hi,
I've started playing around with this interesting library and I have a question. I'm not sure if this can be an issue or maybe I missed the point. So let me explain what I want to build.
My environment:
python=3.10.10
PyYAML=6.0
pyyaml-include=1.3
What I've tried so far:
I want to create some defaults that would be use across the YAML config files without need to copy it multiple times in different blocks with possibly with anchors. So, if I have /test/path/custom-defaults/person.yaml file:
.default_description: &default-description
hair_color: grey
eye_color: blue
age: 123
gender: N/A
And I'd like to include it in other config files, e.g. in /test/path/defaults/describe-people.yaml:
!include ../custom-defaults/person.yaml
persons:
person.with.custom.defaults:
<<: *default-description
person.with.overrided.defaults:
<<: *default-description
hair_color: blue
When try this I'm getting an error: raise ParserError(None, None, yaml.parser.ParserError: expected '<document start>', but found '<block mapping start>'
On the other side, I can fix it by:
Questions:
Is it even possible to use it as a document tag and if yes, can someone provide an example?
Thank you.
Could you please publish 1.2.post2 sdist to PyPI just like 1.2 (pyyaml-include-1.2.tar.gz)? Thanks.
Here is the test case:
os.makedirs("foo/bar")
with open("foo.yml","w") as fd:
fd.write("!include foo/bar.yml\n")
with open("foo/bar.yml","w") as fd:
fd.write("!include bar/zot.yml\n")
with open("foo/bar/zot.yml","w") as fd:
fd.write("foo: 42\n")
YamlIncludeConstructor.add_to_loader_class(yaml.FullLoader)
with open("foo.yml") as fd:
self.assertEqual({"foo":42}, yaml.load(fd, Loader=yaml.FullLoader))
os.remove("foo/bar/zot.yml")
os.remove("foo/bar.yml")
os.remove("foo.yml")
os.removedirs("foo/bar")
it would work if I write the full path foo/bar/zot.yml
into foo/bar.yml
, but this is counterproductive (what if bar.yml
is included from another place or used separately).
Would it be possible for include
to take the path relative to the directory of the node that does the include? (cf. https://gist.github.com/joshbode/569627ced3076931b02f).
Or maybe it is already possible?
Thanks!
Does this package let me include specific components from other yamls? For cases where I don't want to include the entire content of the yaml file. E.g something like
a.yaml
param:
- name: A
type: A1
- name: B
type: B1
stuff: 123
b.yaml
other:
- param: !include a.yaml/param[0]
stuff: !include a.yaml/stuff
such that reading in b.yaml
would result in
{
"other": [{"param": {"name": 'A', "type": 'A1'}}]
"stuff": 123
}
I was trying to have some inheritance, by including the shared yaml-file:
`
file1: !include include.d/1.yaml & shared
section1:
<<: *shared
foo: bar
`
Here is my setup:
The python script (from the README):
import yaml
from yamlinclude import YamlIncludeConstructor
YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader)
with open('0.yml') as f:
data = yaml.load(f, Loader=yaml.FullLoader)
print(data)
0.yml
:
!include 1.yml
foo: bar
1.yml
:
name: "1"
It seems that mixing toplevel include and other value is not (yet?) supported.
I got this error:
Traceback (most recent call last):
File "../test.py", line 32, in <module>
data = yaml.load(f, Loader=yaml.FullLoader)
File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 114, in load
return loader.get_single_data()
File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 43, in get_single_data
node = self.get_single_node()
File "/usr/local/lib/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/usr/local/lib/python2.7/site-packages/yaml/composer.py", line 58, in compose_document
self.get_event()
File "/usr/local/lib/python2.7/site-packages/yaml/parser.py", line 118, in get_event
self.current_event = self.state()
File "/usr/local/lib/python2.7/site-packages/yaml/parser.py", line 193, in parse_document_end
token = self.peek_token()
File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 129, in peek_token
self.fetch_more_tokens()
File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
return self.fetch_value()
File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 579, in fetch_value
self.get_mark())
yaml.scanner.ScannerError: mapping values are not allowed here
in "0.yml", line 2, column 4
I'm trying to get it mapped with more files, but it doesn't work for me.
I have
file foo-1.yml
one: a
file foo-2.yml
two: b
using
foo:
!include foo-1.yml
Works fine, I get a {'foo' :{one: a}}
- thats what I want. But adding another line, like
foo:
!include foo-1.yml
!include foo-2.yml
gives an error: FileNotFoundError: [Errno 2] No such file or directory: '/home/martin/Projects/toki/toki/data/yaml/items-area-1.yml !include items-area-2.yml'
And using
foo:
!include foo-*.yml
makes a list of two dicts..
what can I doo to have just {foo: {'one': a, 'two': b}}
as a result here?
If I have one document such as (foo.yaml):
!include ./baz.yaml
!include ./baz.yaml
and baz.yaml looks like:
baz1: x
---
baz2: y
I would like this to be interpreted as:
baz1: x
---
baz2: y
---
baz1: x
---
baz2: y
Is there any way to achieve this? Currently, even if I use load_all()
my current setup fails with error.
Ability to use default value if file doesn't exist.
Example:
Somelist: !include { pathname: path.yaml, default: [] }
For example:
something like:
vars: &vars !inc test.yaml
x:
<< *vars
Hello, I want to concatenate two yaml files, what should I do?
for example, I have two different yaml files,
# file1.yaml
name: 1
# file2.yaml
class: 2
I want to concatenate these two files into one file after using pyyaml-include, as shown below
# merge_file.yaml
name: 1
class: 2
I would like to ask, what should I do to get this result?
Thank you very much.
Hello,
How likely are you to support jinja templates?
I have YAML templates that have jina logic in them. For example:
containers:
- name: "{{application_name}}"
image: "{{business_unit}}-docker-{{environment}}.artifactory.svc.aws.gartner.com/{{application_name}}:{{version}}"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
memory: {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}
The last line of the code, is a jinja2 expression.
The probably is, I run the template through the pyyaml-include, to get a full single document from multiple templates. Then, I run the single template through the template engine.
If this:
memory:" {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}"
when pyyaml-include operates on this line, you get:
memory: '{{ vars.eks.memory_limits if vars.eks.memory_limits else ''512mb'' }}'
And the jinja2 engine can't understand it. What jinja2 is expecting is this:
memory: {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}
Are you willing to support a jinja2 template language?
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.