[Feature Request] Refactor to work with ansible galaxy about ansible-pfsense HOT 15 CLOSED

I really have no idea why ansible is not interested in my PR as there has been zero replies to it. I suspect they are just overwhelmed and pfsense is not a priority for them.

I am starting to think that a galaxy collection will likely be the way forward for the foreseeable future. I've created a new github organization (pfsensible) and a new repo (core) https://github.com/pfsensible/core. I've also requested the pfsensible namespace in galaxy (ansible/galaxy#2194). I'm sad that we loose the git history in the pfsensible/core repo, but maybe this can be addressed later. The core repo is generated via a simple script that I'll check in at some point when ready.

@tvories thanks again for getting this started!

from ansible-pfsense.

Closing this. We're going to maintain the separate repos for now. Thank you again for getting this started.

from ansible-pfsense.

I tried to reproduce the error by creating your role, collection build then collection install, but when I run the role, I get another error on the task:

Could not find imported module support code for pfsense_vlan.  Looked for either PFSenseModule.py or pfsense.py

Are you installing the collection into a particular place ?

from ansible-pfsense.

I do have my collection path set to collections_paths=collections in my ansible.cfg. Also, make sure you are referencing the FQCN for the plugin. IE: opoplawski.pfsense.pfsense_vlan instead of pfsense_vlan.

from ansible-pfsense.

Ok, I took a little more time to understand. I think my error message is quite the same as yours (I'm just not sure why it's looking so different).

The issue comes from the import in module_utils/network/pfsense/pfsense_vlan.py which is not refactored. It works when it is.

So it should be:

from ansible_collections.opoplawski.pfsense.plugins.module_utils.network.pfsense.pfsense import PFSenseModule, PFSenseModuleBase

Let me know if it works for you.

Also, if I may suggest a couple things:

• get rid of the module_utils sub-directories (network and pfsense), just put everything in module_utils
• exclude the misc and test directories
• pfsense.modules seems a better collection name imo

from ansible-pfsense.

I thought the same thing about removing the sub-folders. It makes sense when it matches the ansible plugin architecture but in a contained collection it's just extra folders.

Let me try your fix.

Edit - Working! Repo updated at https://github.com/tvories/ansible-pfsense/tree/galaxy

OK, I figured out my issue. I actually had a few issues. First, you are correct, I was not properly importing the module_utils packages. I had initially had those there but when I was having other issues I removed them. I fixed those imports and then discovered that I had accidentally overwritten the imports for some valid ansible.module_utils imports. I corrected those and everything is now working!

I also removed the extra module_utils structure to clean up the project. I moved tests out of the module folder and back to the top level. I am not sure that the tests are actually correct and I'm not sure how to validate them.

The misc and test directories are excluded, but you must build the collection using ansible 2.10+. The build_ignore: feature in the galaxy.yml file does not work in 2.9.

pfsense.modules seems a better collection name imo

You and @opoplawski have far more to say about the structure than I do, but it would make the module name in the actual ansible tasks very verbose. For example, the ansible task would go from:

- name: 192.168.60.0/24 - DMZ
  opoplawski.pfsense.pfsense_vlan:
    interface: "{{ pfsense_lan_nic }}"
    vlan_id: 60
    descr: 192.168.60.0/24 - DMZ
    state: present

to

- name: 192.168.60.0/24 - DMZ
  opoplawski.pfsense.module.pfsense_vlan:
    interface: "{{ pfsense_lan_nic }}"
    vlan_id: 60
    descr: 192.168.60.0/24 - DMZ
    state: present

Not the end of the world but definitely a long plugin name.

from ansible-pfsense.

I just realized that if this PR gets accepted, it will require ansible 2.9+ to use this plugin. Either that, or there will be duplicate docs for using it with galaxy or stand-alone. Any input on that decision @opoplawski or @f-bor? I can definitely add docs for both use cases if you think that's the best way.

from ansible-pfsense.

I added documentation for using ansible-galaxy as well as the previous setup. I added a release to my repo with a zipped binary of the unmodified repo to allow people to still install the module even if they don't have ansible 2.9.

from ansible-pfsense.

What I meant for the collection name was "namespace: pfsense" and "name: modules". So the wording to call pfsense_vlan would be pfsense.modules.pfsense_vlan which seems natural to me (and easier to spell than Orion's name :D)

For the PR, I'm not used to ansible galaxy but I would prefer a script to do the refactoring in a temporary folder and generate the archive from there. The script should be quite easy to write (grep the collection's name from galaxy.yml, copy whatever is required into the temp folder, find -exec sed the imports and launch ansible-galaxy in the end) and it would not break the existing playbooks. For the doc, it should then be an extra wiki page with a link in the project's readme.

@opoplawski any thoughts ?

from ansible-pfsense.

I think the namespace is tied to a user/github account, though I'm not positive on that, so I'm not sure if pfsense would be available over opoplawski. I agree that it would be much more straightforward, though.

Good ideas on the script generated refactoring, although I did found the original layout to be a bit confusing and didn't like the idea of having to run my infrastructure code inside of this repo and preferred it to be completely stand-alone. It was difficult for me initially to install this module. I'm definitely open to suggestions, though.

It's possible we could further refactor it and keep the playbooks working in the collection and add a role folder in addition to the plugin folder, similar to https://github.com/freeipa/ansible-freeipa . That would keep each piece (roles and plugins) self contained but would allow for both if a user wanted to use the included roles.

from ansible-pfsense.

First, thanks for working on this. I just haven't had any extra time recently.

I think it's time to create a github organization to host this. We could go with a very short name (pfans?) or something like 'pfansible'. I'm almost irresistibly drawn to 'pfsensible' as it is too cute :). Then the project/repo name could be something like 'core', leaving open the possibility of other collections for say managing openvpn, haproxy, and other packages. I could even see dropping the pfsense_ from the module names and so have 'pfsensible.core.rule'.

I really like the idea of building the collection via script, at least for a while to test the waters. But I suspect that this will really be the way forward for this project. We've gotten basically zero response to our attempts to start to move this into ansible proper.

from ansible-pfsense.

I wonder if we create a new repo just for the galaxy version for now and then any changes in the main repo get scripted down to the galaxy project. I think it's good to have the galaxy release to have its own repo, as that seems to be the way to doc and request changes to it.

I also like pfsensible 🤣

from ansible-pfsense.

I just read up on the ansible PR you guys submitted to merge it into ansible base. That makes a lot of sense now as to the way the repository was set up. I would agree that you probably wouldn't want to totally refactor the plugin if there's a chance it would be included in ansible at some point.

Sorry for any confusion here. I was unaware of the history of this project. I think the downstream separate galaxy repo may be a better idea.

from ansible-pfsense.

@opoplawski I think they didn't answer because of the aggregated scheme. If we remove that, merge the related files between library and module_utils, they would probably accept your PRs for the modules.

If I'm the only user of those modules (aggregate & ipsec_aggregate), it's probably the best solution. I can just write a program to regenerate the modules as they are today for my own usage.

from ansible-pfsense.

Well, our approach to implement modules is quite dirty. We don't have much choice since there is no CLI or API. In the end, we have to rewrite pfSense piece by piece to implement proper parameters validation and changes. I fear the day they will do some major changes and force us to maintain different behaviours following pfSense version. It would be easier for everyone if they release a proper CLI or API.

FWIW, I don't see the point in having an organization. Enabling tags here and having a simple packaging script would'nt be enough to push to galaxy ?

from ansible-pfsense.