Python is not loading the correct certificate bundle in the docker image about signalfx-agent HOT 7 CLOSED

Do you mind sharing the domain that the agent is trying to verify but can't? The docker image for the agent uses the system certs from Ubuntu 16.04 and we make sure to pull the latest base image every time we do a build, so I am surprised something is missing.

from signalfx-agent.

Do you mind sharing the domain that the agent is trying to verify but can't? The docker image for the agent uses the system certs from Ubuntu 16.04 and we make sure to pull the latest base image every time we do a build, so I am surprised something is missing.

I'm hesitant to post the full hostname publicly, but it's of the form *.mongodb.net. The certificate chain can be found here: https://gist.github.com/jalaziz/3a9ec245c13cc479e9e75cc52592e959

I should add, it may not be the /etc/ssl/certs/ca-certificates.crt file that is the problem, but rather the SSL bundle used by python or pymongo (which should be /etc/ssl/certs/ca-certificates.crt, but may not be?).

from signalfx-agent.

Ok, so I did some more testing and setting SSL_CERT_FILE to /etc/ssl/certs/ca-certificates.crt fixes the problem as well. I had assumed the cert file was being set properly, but it appears that it wasn't.

The real issue appears to be that Python (or at least PyMongo) is not loading the cert bundle.

from signalfx-agent.

Ok, that clarifies things a lot, thanks. I made the linked PR to hopefully fix it. If running outside of a Docker container it shouldn't be a problem.

We probably also need to update the pymongo version at some point to better support Mongo 4.x but that will require some more in-depth testing.

from signalfx-agent.

Ok, that clarifies things a lot, thanks. I made the linked PR to hopefully fix it. If running outside of a Docker container it shouldn't be a problem.

Yeah. I need to look deeper into it, but I think it's probably something that is lost when going from Ubuntu to the scratch base image. Not quite sure if it's the SSL_CERT_FILE env var that is set normally or if there is something else that tells the libraries where to look. Will dig deeper tonight. In either case, your PR should fix it.

We probably also need to update the pymongo version at some point to better support Mongo 4.x but that will require some more in-depth testing.

That would be great. Happy to help testing, but we're only running Mongo 3.4 currently. We should be upgrading to 3.6 soon. So while I can't promise testing on 4.x, I can at least promise that we will run the check with the updated driver for an extended period.

from signalfx-agent.

Spent some more time trying to understand why this was broken in the first place and this is what I discovered.

First, by default, the Python ssl library is looking at /usr/lib/ssl/cert.pem and /usr/lib/ssl/certs. These paths actually come from the underlying OpenSSL library.

>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')

If we look at the ubuntu 16.04 container, and check the OpenSSL path, wee see:

openssl version -a
OpenSSL 1.0.2g  1 Mar 2016
...
OPENSSLDIR: "/usr/lib/ssl"

If we check /usr/lib/ssl, we see:

ls -al /usr/lib/ssl/
total 16
drwxr-xr-x 3 root root 4096 Oct 11 19:41 .
drwxr-xr-x 1 root root 4096 Oct 11 19:42 ..
lrwxrwxrwx 1 root root   14 Feb 27  2019 certs -> /etc/ssl/certs
drwxr-xr-x 2 root root 4096 Oct 11 19:41 misc
lrwxrwxrwx 1 root root   20 Feb 27  2019 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx 1 root root   16 Feb 27  2019 private -> /etc/ssl/private

As it turns out, OpenSSL uses its own cert store by default. The directory is specified with --openssldir when building OpenSSL. Ubuntu and Debian systems will usually then symlink the OpenSSL cert directories to the system-level directories.

The SSL_CERT_FILE is actually used by OpenSSL libraries to override the default location, so the committed fix should hopefully help with anything else that relies on OpenSSL.

However, a lesser "band-aid" fix may be to copy over "/usr/lib/ssl/certs" and apply the same symlink.

from signalfx-agent.

Thanks for looking into this more -- that is a very clear explanation.

As long as the SSL_CERT_FILE is officially supported in OpenSSL, as it appears to be, I'm fine with it. As far as I know there is only the Go x509 implementation (which looks in /etc/ssl/certs by default), Java (which bundles its own CA cert list in the JRE) and OpenSSL (collectd/Python) being used in the agent, so I think everything should be covered.

from signalfx-agent.