Multiple Directory Search Path, Combining Multiple Pillar Git Repos about gitstack-pillar HOT 7 OPEN

I was just coming here to raise this same issue.

We operate a delegated setup where we do this using the standard git ext_pillar, ansd use the mountpoint option to merge other repos in at specific directories. it would be useful to have a single 'top' stack repo which contained stack.cfg, and then multiple other repos that contained the yaml files. It would look something like:

stack.cfg:

# sourced from the 'top' repo
defaults.yml
# sourced from the relevant dept repo, mounted in a subdir
{{ __grains__['dept'] }}/defaults.yml 
{{ __grains__['dept'] }}/nodes/{{ minion_id }}.yml

top.git

defaults.yml

finance.git

defaults.yml
nodes/finance-server.yml

admin.git

defaults.yml
nodes/admin-server.yml

from gitstack-pillar.

Hi @amendlik, thanks for the project!

What do you think of this feature request?

This would enable a breakthrough in our ability to modularize and publish much more of our work, which is something I'm keen to do.

If you're not opposed to the idea, would you accept a pull request?

from gitstack-pillar.

@nickgarber: I like the idea very much and would be happy to take a PR for this. Before you put too much time into it, can you propose a configuration format (preferably one that is backwards compatible)? How do we know which repo is the "top" repo?

from gitstack-pillar.

Sweet!

Here's what comes to mind for me...

ext_pillar:
  - gitstack: 
    - master https://github.com/org/myrepo.git:
      ## perhaps some config to control overlay file merge handling, (not sure about these yet)
      #- permit_submount_to_shadow_merge: false
      #- permit_using_stack_cfg: false
      ## these configs would remain unchanged
      - stack:
          pillar:environment:
            dev: _stack/stack.cfg
            prod: _stack/stack.cfg
          grains:custom:grain:
            value:
              - _stack/stack1.cfg
              - _stack/stack2.cfg
          opts:custom:opt:
            value: _stack/stack0.cfg
      ## 'submount' is the first term that comes to mind, but maybe there's something better?
      - submounts:
        - master https://github.com/org/myrepo2.git:
          - root: pillarstack_config.d
          - mountpoint: org2
        - master https://github.com/org/myrepo3.git
          ## 'root' would default to the top directory of the repo
          ## 'mointpoint' would default to the top level of the merge directory
  - git: 
    - master https://github.com/org/myrepo.git
    - master https://github.com/org/myrepo2.git
    - master https://github.com/org/myrepo3.git

from gitstack-pillar.

@nickgarber: This all looks very good to me

from gitstack-pillar.

As I look into this my idea of the ideal approach has changed a bit so wanted to check in.

Originally I thought of using symlinks in the checkout at the git_pillar cache, though quickly realized that would be impractical and barbaric.

Next I thought that perhaps the right approach would be to have gitstack perform a "first-found wins" ordered search through candidate directories and only pass the paths that it new to exist. This still seems feasible, but there's one final method thats caught my attention that I'm learning about now...

The mountpoint option is mentioned in the https://github.com/saltstack/salt/blob/develop/salt/pillar/git_pillar.py file, so I continued reading in https://github.com/saltstack/salt/blob/develop/salt/utils/gitfs.py, (the latter is imported from the former), and where mountpoint is frequently mentioned.

It was in the salt/utils/gitfs.py file that I saw linkdirs mentioned, so I'm wrapping my head around that concept in order to understand if it would be a more full featured solution. Also reading through the info here, https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.git_pillar.html#how-multiple-remotes-are-handled .

Based on what I come away understanding about creating and using the features provided in salt/utils/gitfs.py from within salt/pillar/gitstack.py, I may favor option 2 (with a single unified view) or options 3 (with greater re-use of existing feature code).

It feels like option 3 is the more promising way to go.

Thoughts?
Cheers :)

from gitstack-pillar.

I've struggled to make progress on this issue. I'm not giving up, but want to communicate that I'm open to assistance and guidance.

from gitstack-pillar.