Yum fails with http 403 error on installing MySQL repo (Akamai CDN) about ansible HOT 6 CLOSED

Files identified in the description:

None

If these files are incorrect, please update the component name section of the description or use the component bot command.

from ansible.

You can accomplish this by downloading the RPM using the get_url module which allows you to set the agent https://docs.ansible.com/ansible/latest/collections/ansible/builtin/get_url_module.html#parameter-http_agent. Then you can install the downloaded RPM using the yum module.

from ansible.

@s-hertel

I knew that there was a way to avoid this by customizing the user agent with a different approach but I supposed that this could be solved at the root level by making the default process more reliable in these situation with widely used CDN like Akamai, Cloudflare etc.

The yum command itself works as intended by itself on the target machine if I simply issue

yum install https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm

So the first thing that one does is translate it (by following the docs) to

- name: Install Mysql 5.7 Community Repo
  yum:
    name: "https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm"
    state: present

which doesn't work because the UA ansible-httpget gets a bad score.

I was thinking to send a PR that sets an UA with more context and details like the module name (if available):

def get_user_agent(module=None):
    """Returns a user agent used by open_url"""
    python_version = sys.version_info
    module_part = ''
    if not module is None and hasattr(module, '_name') and module._name:
        module_part = "; %s" % module._name
    return u'ansible-httpget/{ansible_version} ({platform}{module_part}; python:{py_major}.{py_minor}.{py_micro})'.format(
        ansible_version=ansible_version,
        platform=platform.system(),
        module_part=module_part,
        py_major=python_version.major,
        py_minor=python_version.minor,
        py_micro=python_version.micro,
    )

which, in this case, translates to ansible-httpget/2.16.3 (Linux; ansible.legacy.yum; python:2.7.5) that works by default.

from ansible.

@lucalooz It could be configurable by adding an http_agent option to the arg spec:

diff --git a/lib/ansible/module_utils/yumdnf.py b/lib/ansible/module_utils/yumdnf.py
index 7eb9d5fc2b..b70e937cf4 100644
--- a/lib/ansible/module_utils/yumdnf.py
+++ b/lib/ansible/module_utils/yumdnf.py
@@ -35,6 +35,7 @@ yumdnf_argument_spec = dict(
         enable_plugin=dict(type='list', elements='str', default=[]),
         enablerepo=dict(type='list', elements='str', default=[]),
         exclude=dict(type='list', elements='str', default=[]),
+        http_agent=dict(type='str', default='ansible-httpget'),
         installroot=dict(type='str', default="/"),
         install_repoquery=dict(type='bool', default=True),
         install_weak_deps=dict(type='bool', default=True),

but I don't think we want to add every option supported by open_url to every module that might fetch a URL. The yum module fetches the URL first to see if it's installed. I also don't think we can change the default without potentially breaking playbooks.

from ansible.

@s-hertel I concur that we can't add every url setting to every module, maybe just a global configuration for the default UA.

I was just proposing to have a more detailed UA respecting the RFC that contains information about the calling module of fetch_url in urls.py.

I don't know if there are tests that we can run to ensure that's not breaking anything or if it's important to keep the same exact UA for historical reasons because some server may have an hardcoded rule for ansible-httpget.

from ansible.

@s-hertel I concur that we can't add every url setting to every module, maybe just a global configuration for the default UA.

We generally avoid new global configuration options that change how the playbook runs, because it breaks content portability. This seems a little niche for a new play keyword. A get_url task in a playbook also seems easier to maintain than some keyword or config that affects some unknown tasks.

It's unfortunate the module fetches the file at all, since there are probably a lot of ways our URL-fetching differs from yum. Making User-Agent configurable and nothing else seems arbitrary - I see lots of HTTP related config options in https://man7.org/linux/man-pages/man5/yum.conf.5.html. Ideally we would defer to the yum CLI to see if any changes would be made (but it doesn't support that).

I was just proposing to have a more detailed UA respecting the RFC that contains information about the calling module of fetch_url in urls.py.

HTTP RFC 7231 specifies that the version is optional.

from ansible.