Issue with GKE, Grafana, CertManager, SonarQube latest version and SHA tag and old metric about version-checker HOT 2 OPEN

@AleksandarMis thank you for raising such a detailed issue for us. I need a bit of time to digest all the issues presented here and determine if we already have similar or duplicated issues open, such as #138 as you already mentioned.

Just wanted to say we've seen your issue and I plan to take a look in the next few weeks.

from version-checker.

Hi @AleksandarMis, I've taken an extensive look into the reported issues and found the following:

1. GKE

Thank you for this, I'd totally forgotten that images were now moved under the gke.gcr.io and have opened a PR to resolve this.

As GKE images are suffixed with metadata, you MUST ensure that the pods are annotated with use-metadata.version-checker.io/kube-proxy="true" for the versions to be successfully matched.

See PR #202 that should resolve this issue.

2. Grafana

This issue is a tricky one, primarily caused by Grafana pushing an image that doesn't match semver and also has no metadata (which by default version-checker would have voided the version).

I've tried a number of days to try and see if I could tweak version-checkers sorting and selection process to no-avail, however I have resolved this by adding the following annotation to grafana:

~$ curl -s localhost:8080/metrics | grep 'grafana/grafana'
version_checker_is_latest_version{container="grafana",container_type="container",current_version="10.3.4",image="grafana/grafana",latest_version="9799770991",namespace="kubecost",pod="kubecost-grafana-585f95598c-mqq8m"} 0
~$ k annotate pods finops-grafana-5f67d6fc4-dqkfc match-regex.version-checker.io/grafana='(\d+)\.(\d+)\.(\d+)'
pod/finops-grafana-5f67d6fc4-dqkfc annotated
~$ curl -s localhost:8080/metrics | grep 'grafana/grafana'
version_checker_is_latest_version{container="grafana",container_type="container",current_version="10.3.4",image="docker.io/grafana/grafana",latest_version="11.0.0@sha256:a80bc3848cf5d4b2958ea25dbeb36fa9442ef4be8c73fe4bff11340307c32919",namespace="davidtest",pod="finops-grafana-5f67d6fc4-dqkfc"} 0

Which correctly registers the valid latest image as 11.0.0

3. Cert-Manager

This is similar to that of grafana, the 608111629 image is taken as 608111629.0.0 and therefore always registered as the latest version.

To resolve this, the easiest approach is to annotate these pods with: match-regex.version-checker.io/cert-manager='v(\d+)\.(\d+)\.(\d+), This ensures that tags not starting with v are discarded.

4. Sonarqube

To re-write the URI - you would set the value to be docker.io/sonarqube I.E: override-url.version-checker.io/sonarqube: docker.io/sonarqube, if debugging is enabled -v=debug then you should see the following log line:

overriding image lookup docker.io/grafana/grafana -> sonarqube  module=version_getter

Taking a look at Sonarqube's tagging strategy, it looks like you'll also need to enable the metadata usage:

use-metadata.version-checker.io/sonarqube="true"

(I used my Grafana instance as an example here)

Assuming you're only interested in say the community editions, you can still use the regex match (from above) like so:

k annotate pods kyverno-admission-controller-969cddd96-dh8ht match-regex.version-checker.io/kyverno='(\d+)\.(\d+)\.(\d+)-community'

(This didn't pick up the correct output due to their being metadata in the versions of the container I'm testing against, but you can see that version-checker detected the correct latest tag within the metric labels):

version_checker_is_latest_version{container="kyverno",container_type="container",current_version="v1.12.3",image="docker.io/sonarqube",latest_version="10.5.1-comm
unity",namespace="davidtest",pod="kyverno-admission-controller-969cddd96-dh8ht"} 1

I've raised #204 to help make this a little clearer of where the versions actually come from when using the overrideURI option/annotation.

5. SHA Tags

I have seen this locally and been trying to address this issue more thoroughly, however this is more challenging that expected.
I have noticed that this some what resolves it self for many images, the SHA256 is used to detect if the container/image being used
at the time of version-checkers checking, has had a newer version pushed (I.E: re-tagged) for what ever purpose (patching/CVE resolution).

The annotation use-sha.version-checker.io/${container} is mainly used for the version checking based of SHA, and uses the timestamp in which
the image was pushed to the registry. This can be used if you want to use latest or v3 and ensure that you have that version up to date.

6. Send old metrics

I suspect this is related to the image-cache-timeout flag being set to the default of 30 Minutes. Meaning metrics can still be reported after their deletion. I need to do some more thorough investigation on this one, I also suspect that this could be related to annotations changing and/or that version-checker is missing container deletion and therefore missing the deletion of the metrics from the metrics registry.

I've raised the following PR to attempt to address this: #203

End Note:

Once again, thank you for such a detailed and long issue, but with some really good example use-cases. The information you provided have been incredibly interesting and do highlight, just how difficult checking versions can be.

from version-checker.