Git Product home page Git Product logo

Comments (8)

tanbro avatar tanbro commented on August 10, 2024

@ba1dr Thanks for your advice!

Yes, it's nice and natural to parse the relative file from where the current YAML is.

But i can not to get information about current file, as you wrote.

For the case of that "config files in different directories.", i think current API can't do that beautifully, but to use absolute path is workable.

from pyyaml-include.

tanbro avatar tanbro commented on August 10, 2024

jinjyaml is a Jinja2 template engine integration for PyYAML.

We can include files by Jina2's include instruction:

Consider we have below YAML:

parent: !j2 |
    {% include "child-1.yml" %}
    {% include "child-2.yml" %}

then execute:

import jinja2
import jinjyaml

j2_env = jinja2.Environment(
    loader = jinja2.FileSystemLoader(searchpath=your_base_dir)
)

j2_ctor = jinjyaml.Constructor()
yaml.add_constructor('!j2', j2_ctor)

doc = yaml.full_load(yaml_string)

data = jinjyaml.extract(doc, env=j2_env)

Jinja2's FileSystemLoader would load child-1.yml and child-2.yml, relative to it's search path.

And we can even write a custom Jinja2 file loader, for particular purpose.

from pyyaml-include.

ba1dr avatar ba1dr commented on August 10, 2024

Hmm, no, I think Jinja2 would be an overkill. If using template engines - I'd better use config file on Python generated with Jinja2 rather than yaml.
Even with this include feature I am not sure if it is a good idea to use it as it breaks compatibility with other languages or scripts that do not support this tag..

from pyyaml-include.

tanbro avatar tanbro commented on August 10, 2024

Perhaps a YMAL's Json Pointer (if there be one) could be more fit for the case.

from pyyaml-include.

1ace avatar 1ace commented on August 10, 2024

In case anyone's interested, my current workaround is this:

import contextlib
import os
import pathlib

import yaml
from yamlinclude import YamlIncludeConstructor


YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.SafeLoader)


@contextlib.contextmanager
def working_directory(path: pathlib.Path):
    prev_cwd = pathlib.Path.cwd()
    os.chdir(path)
    try:
        yield
    finally:
        os.chdir(prev_cwd)


def load_config_file(file_path: pathlib.Path):
    with working_directory(file_path.parent):
        with file_path.open("r") as config_file:
            return yaml.safe_load(config_file)

But obviously, the limitation is that any 2nd+ level include is relative to the first file, not any intermediate files, but luckily that's good enough for us right now 🙂

from pyyaml-include.

idantene avatar idantene commented on August 10, 2024

But i can not to get information about current file, as you wrote.

Actually @tanbro, you can! :)
One just had to change the base_dir as they travel along, and extract the name of the file from the stream, then patch yaml.load specifically.

EDIT: Updated the snippet, this is what we now use internally in an __init__.py.

import yaml
from yamlinclude import YamlIncludeConstructor

YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader)
YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.SafeLoader)
YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.Loader)
YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.BaseLoader)

include_tag = YamlIncludeConstructor.DEFAULT_TAG_NAME
yaml_load = yaml.load  # Save original load function

def load_yaml(stream, Loader):
    from pathlib import Path
    path = Path(stream.name)
    if include_tag not in Loader.yaml_constructors:
        return yaml_load(stream, Loader=Loader)
    previous_base = Loader.yaml_constructors[include_tag].base_dir
    Loader.yaml_constructors[include_tag].base_dir = path.parent.as_posix()
    res = yaml_load(stream, Loader=Loader)
    Loader.yaml_constructors[include_tag].base_dir = previous_base
    return res

yaml.load = load_yaml  # Use new one

del YamlIncludeConstructor
del yaml

from pyyaml-include.

idantene avatar idantene commented on August 10, 2024

The above would fail on strings (if used with e.g. yaml.load(f.read()), or some local definitions).
One can add an isinstance(stream, io.TextIOWrapper) for validation as needed.

EDIT: Like so:

yaml_load = yaml.load  # Save original load function

def load_yaml(stream, Loader):
    from pathlib import Path
    from yamlinclude import YamlIncludeConstructor
    from io import TextIOWrapper
    tag = YamlIncludeConstructor.DEFAULT_TAG_NAME
    if tag not in Loader.yaml_constructors or not isinstance(stream, TextIOWrapper):
        # If tag is included in the stream but we can't get the file location, we can't assume
        # anything about the relative file location
        return yaml_load(stream, Loader=Loader)
    path = Path(stream.name)
    previous_base = Loader.yaml_constructors[tag].base_dir
    Loader.yaml_constructors[tag].base_dir = path.parent.as_posix()
    res = yaml_load(stream, Loader=Loader)
    Loader.yaml_constructors[tag].base_dir = previous_base
    return res

yaml.load = load_yaml  # Use new one

@ba1dr @1ace You might be interested ^

from pyyaml-include.

tanbro avatar tanbro commented on August 10, 2024

#26 provides a way to include files relatively

from pyyaml-include.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.