Git Product home page Git Product logo

Comments (5)

mmontes11 avatar mmontes11 commented on June 18, 2024 1

I've added an extra step at the end of the upgrade guide that should fix this issue:

As part of the release v0.0.24 we've unified the headless service to inconditionally be <mariadb>-internal, as we are relying on it for the metrics now.

The problem is that the spec.serviceName of the StatefulSet is inmutable:

# statefulsets.apps "mariadb" was not valid:
# * spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

So unfortunately we have to do fix this by hand.

Closing! I've manged to reproduce and fix the issue, but feel free to reopen if you still encounter problems!

from mariadb-operator.

mmontes11 avatar mmontes11 commented on June 18, 2024

Thanks for reporting @outductor !

First of all, does the mariadb-internal Service have associated Endpoints?:

kubectl get endpoints mariadb-internal -o yaml

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2023-12-31T16:45:59Z"
  creationTimestamp: "2023-12-31T16:45:53Z"
  labels:
    service.kubernetes.io/headless: ""
  name: mariadb-internal
  namespace: mariadb-operator
  resourceVersion: "135950330"
  uid: 46f09d01-a86d-49f8-a2ec-9ef0f60a70d4
subsets:
- addresses:
  - hostname: mariadb-0
    ip: 10.10.2.17
    nodeName: k8s-worker1
    targetRef:
      kind: Pod
      name: mariadb-0
      namespace: mariadb-operator
      uid: f620d8b6-03dd-4bd2-8577-f403342e1665
  ports:
  - name: mariadb
    port: 3306
    protocol: TCP

If that's the case, are you able to resolve mariadb-0.mariadb-internal.namespace.svc instead of mariadb-0.mariadb-internal.namespace.svc.cluster.local?

nslookup mariadb-0.mariadb-internal.mariadb-operator.svc
Server:         10.90.0.10
Address:        10.90.0.10#53

Name:   mariadb-0.mariadb-internal.mariadb-operator.svc.cluster.local
Address: 10.10.2.17

from mariadb-operator.

outductor avatar outductor commented on June 18, 2024

@mmontes11 Thansk for the response. The name you provided did not get resolved either.

kubectl get endpoints -n namespace mariadb-internal -o yaml | kubectl neat
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2023-12-30T22:45:33Z"
  labels:
    service.kubernetes.io/headless: ""
  name: mariadb-internal
  namespace: namespace
subsets:
- addresses:
  - ip: 10.96.132.31
    nodeName: wk-3
    targetRef:
      kind: Pod
      name: mariadb-0
      namespace: namespace
      uid: 85e6bc38-cc21-40c5-9712-a17368621632
  ports:
  - name: mariadb
    port: 3306
    protocol: TCP

# dig mariadb-0.mariadb-internal.namespace.svc

; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> mariadb-0.mariadb-internal.namespace.svc
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 47745
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: ad7a3cfa4535e621 (echoed)
;; QUESTION SECTION:
;mariadb-0.mariadb-internal.namespace.svc. IN A

;; AUTHORITY SECTION:
.			30	IN	SOA	a.root-servers.net. nstld.verisign-grs.com. 2023123101 1800 900 604800 86400

;; Query time: 28 msec
;; SERVER: 10.96.64.10#53(10.96.64.10) (UDP)
;; WHEN: Mon Jan 01 08:36:10 JST 2024
;; MSG SIZE  rcvd: 169

from mariadb-operator.

evanrich avatar evanrich commented on June 18, 2024

@mmontes11 I just found this while seeing issues with the exporter, same as the OP. I did the steps you mentioned above, but still have an issue. after rolling out the statefuleset again, i can see the service name now points to the -internal name, but the exporter logs show:

ts=2024-04-06T05:17:27.606Z caller=mysqld_exporter.go:220 level=info msg="Starting mysqld_exporter" version="(version=0.15.1, branch=HEAD, revision=cc349684494b5038ec5a52233bdca9eb9291e6f2)"
ts=2024-04-06T05:17:27.606Z caller=mysqld_exporter.go:221 level=info msg="Build context" build_context="(go=go1.21.5, platform=linux/amd64, user=root@d89c15b9f5ad, date=20231212-07:55:09, tags=unknown)"
ts=2024-04-06T05:17:27.606Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=global_status
ts=2024-04-06T05:17:27.606Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=global_variables
ts=2024-04-06T05:17:27.606Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=slave_status
ts=2024-04-06T05:17:27.606Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=info_schema.innodb_cmp
ts=2024-04-06T05:17:27.606Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=info_schema.innodb_cmpmem
ts=2024-04-06T05:17:27.606Z caller=mysqld_exporter.go:233 level=info msg="Scraper enabled" scraper=info_schema.query_response_time
ts=2024-04-06T05:17:27.607Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9104
ts=2024-04-06T05:17:27.607Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9104
ts=2024-04-06T05:17:43.161Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"
ts=2024-04-06T05:17:45.027Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"
ts=2024-04-06T05:17:45.343Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"
ts=2024-04-06T05:17:45.566Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"
ts=2024-04-06T05:17:45.810Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"
ts=2024-04-06T05:17:46.043Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"
ts=2024-04-06T05:17:46.243Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"
ts=2024-04-06T05:17:51.540Z caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused

I've verified that the mysql username/password are correct via phpmyadmin any idea whats up? I'm using the following MariaDB def:

apiVersion: k8s.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
spec:
  database: mariadb
  env:
    - name: TZ
      value: SYSTEM
  envFrom:
    - configMapRef:
        name: mariadb
  image: mariadb:11.1.4
  imagePullPolicy: IfNotPresent
  livenessProbe:
    exec:
      command:
        - bash
        - -c
        - mariadb -u root -p"${MARIADB_ROOT_PASSWORD}" -e "SELECT 1;"
    initialDelaySeconds: 20
    periodSeconds: 10
    timeoutSeconds: 5
  metrics:
    enabled: true
    exporter:
      image: prom/mysqld-exporter:v0.15.1
      port: 9104
      resources:
        limits:
          cpu: 300m
          memory: 512Mi
        requests:
          cpu: 50m
          memory: 64Mi
    username: mariadb-metrics
    passwordSecretKeyRef:
      name: mariadb-metrics-password
      key: password
    serviceMonitor:
      interval: 10s
      prometheusRelease: kube-prometheus-stack
      scrapeTimeout: 10s
  myCnf: |
    [mysqld]
    bind-address=0.0.0.0
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    max_allowed_packet=256M
  myCnfConfigMapKeyRef:
    key: my.cnf
    name: config-mariadb
  passwordSecretKeyRef:
    key: MYSQL_PASSWORD
    name: mariadb-db-secrets
  podDisruptionBudget:
    maxUnavailable: 50%
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 999
  port: 3306
  readinessProbe:
    exec:
      command:
        - bash
        - -c
        - mariadb -u root -p"${MARIADB_ROOT_PASSWORD}" -e "SELECT 1;"
    initialDelaySeconds: 20
    periodSeconds: 10
    timeoutSeconds: 5
  replicas: 1
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 128Mi
  rootEmptyPassword: false
  rootPasswordSecretKeyRef:
    key: MYSQL_ROOT_PASSWORD
    name: mariadb-db-secrets
  securityContext:
    allowPrivilegeEscalation: false
    runAsNonRoot: true
  service:
    type: ClusterIP
  updateStrategy:
    type: RollingUpdate
  username: mariadb
  volumeMounts:
    - mountPath: /tmp
      name: tmp
  volumes:
    - emptyDir: {}
      name: tmp
  serviceAccountName: mariadb
  storage:
    size: 30Gi
    volumeClaimTemplate:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 30Gi

thanks!

Edit: Actually, it seems to be working now. I was trying to test via the exporters UI (forwarding :9104 and then /metrics). however grafana is picking up metrics, so it looks like it all works within the cluster, but not via the UI. I'm ok with that :)

from mariadb-operator.

mmontes11 avatar mmontes11 commented on June 18, 2024

exporters UI (forwarding :9104 and then /metrics)

Actually we don't use the /metrics endpoint, we use multi-target endpoint /probe?target=foo:3306:
https://github.com/prometheus/mysqld_exporter?tab=readme-ov-file#multi-target-support

Just tested locally and I'm getting the same error when using the /metrics endpoint, so I guess our configuration is not backwards compatible. But anyway, the multi-target approach should work!

from mariadb-operator.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.