Git Product home page Git Product logo

custom-metrics-apiserver's Introduction

Custom Metrics Adapter Server Boilerplate

Go Reference

Purpose

This repository contains boilerplate code for setting up an implementation of the Metrics APIs:

  • Custom metrics (k8s.io/metrics/pkg/apis/custom_metrics)
  • External metrics (k8s.io/metrics/pkg/apis/external_metrics)

It includes the necessary boilerplate for setting up an implementation (generic API server setup, registration of resources, etc), plus an implementation for testing that allows setting custom metric values over HTTP.

How to use this repository

This repository is designed to be used as a library. First, implement one or more of the metrics provider interfaces in pkg/provider (for example, CustomMetricsProvider), depending on which APIs you want to support.

Then, use the AdapterBase in pkg/cmd to initialize the necessary flags and set up the API server, passing in your providers.

More information can be found in the getting started guide, and the testing implementation can be found in the test-adapter directory.

Prerequisites

Go: this library requires the same version of Go as Kubernetes.

Test Adapter

There is a test adapter in this repository that can be used for testing changes to the repository, as a mock implementation of the APIs for automated unit tests, and also as an example implementation.

Note that this adapter should not be used for production. It's for writing automated e2e tests and serving as a sample only.

To build and deploy it:

# build the test-adapter container as $REGISTRY/k8s-test-metrics-adapter-amd64
export REGISTRY=<some-prefix>
make test-adapter-container

# push the container up to a registry (optional if your cluster is local)
docker push $REGISTRY/k8s-test-metrics-adapter-amd64

# launch the adapter using the test adapter deployment manifest
kubectl apply -f test-adapter-deploy/testing-adapter.yaml

When the deployment is ready, you can define new metrics on the test adapter by querying the write endpoint:

# set up a proxy to the API server so we can access write endpoints
# of the testing adapter directly
kubectl proxy &
# write a sample metric -- the write paths match the same URL structure
# as the read paths, but at the /write-metrics base path.
# data needs to be in json, so we also need to set the content-type header
curl -X POST \
  -H 'Content-Type: application/json' \
  http://localhost:8001/api/v1/namespaces/custom-metrics/services/custom-metrics-apiserver:http/proxy/write-metrics/namespaces/default/services/kubernetes/test-metric \
  --data-raw '"300m"'
# you can pipe to `jq .` to pretty-print the output, if it's installed
# (otherwise, it's not necessary)
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2" | jq .
# fetching certain custom metrics of namespaced resources
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/default/services/kubernetes/test-metric" | jq .

If you wanted to target a simple nginx-deployment and then use this as an HPA scaler metric, something like this would work following the previous curl command:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-deployment
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-deployment
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      metric:
        name: test-metric
      describedObject:
        apiVersion: v1
        kind: Service
        name: kubernetes
      target:
        type: Value
        value: 300m

You can also query the external metrics:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .
# fetching certain custom metrics of namespaced resources
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/my-external-metric" | jq .

Compatibility

The APIs in this repository follow the standard guarantees for Kubernetes APIs, and will follow Kubernetes releases.

Community, discussion, contribution, and support

Learn how to engage with the Kubernetes community on the community page.

You can reach the maintainers of this repository at:

Code of Conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

Contribution Guidelines

See CONTRIBUTING.md for more information.

custom-metrics-apiserver's People

Contributors

44past4 avatar alb0t avatar answer1991 avatar astefanutti avatar catherinef-dev avatar damemi avatar dashanji avatar dependabot[bot] avatar dgrisonnet avatar directxman12 avatar dttung2905 avatar hjet avatar johanneswuerbach avatar jorturfer avatar jsturtevant avatar k8s-ci-robot avatar kawych avatar luxas avatar mbssaiakhil avatar mirake avatar olivierlemasle avatar s-urbaniak avatar sanwishe avatar skydiator avatar srinivasrk avatar uthark avatar wanlinghao avatar whitebear009 avatar yangjunmyfm192085 avatar zroubalik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

custom-metrics-apiserver's Issues

Release v1.22.0

## Changes since v1.21.0

### Changes
* Make metrics provider context-aware (@answer1991)
* Move module to sigs.k8s.io/custom-metrics-apiserver (@arajkumar)

### Improvements
* Allow users to configure audit logs (@dgrisonnet)
* Bump Kubernetes dependencies to v0.22.0 (@dgrisonnet)

/cc @s-urbaniak

Multiple adapters in cluster?

Is there a way how to deploy multiple adapters in a cluster? Eg. there's already deployed custom metrics server Foo and I need to deploy my own implementation Bar which provides different set of metrics.

If I understand it correctly, to be able to provide custome/external metrics API, the adapter needs to deploy cluster wide APIService with a specific name. Deploying second adapter in the cluster will results in reconfiguring the original APIService object.

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: custom-metrics
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

Multiple redundant calls to `ListAllMetrics()` and `ListAllExternalMetrics()`

Is there a particular reason why the provider does multiple calls to both ListAllMetrics() and ListAllExternalMetrics() in a short period of time? It is happenning periodically each couple of seconds.

See the log snippet below, I have modified test-adapter to print log message every time it is called, see lines with Got request to List All Metrics and Got request to List All External Metrics.

 1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta2" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854588       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854469       1 provider.go:353] Got request to List All External Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.855354       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="986.14ยตs" userAgent="Go-http-client/2.0" audit-ID="f490b89a-a949-49c9-bfff-90194f94c3a8" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.855398       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/external.metrics.k8s.io/v1beta1" latency="1.466749ms" userAgent="Go-http-client/2.0" audit-ID="0995f5d1-83aa-4d11-8f1e-b2fae1267103" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854599       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.855905       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856024       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="1.586697ms" userAgent="Go-http-client/2.0" audit-ID="9b1f7a5a-cbd8-4459-bdc5-8f9323dba109" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854232       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856172       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856439       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="2.164326ms" userAgent="Go-http-client/2.0" audit-ID="f88707c2-d8ab-47cd-8753-d8dc2304f183" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854355       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854368       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta2" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854399       1 handler.go:143] test-adapter: GET "/apis/external.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/external.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854177       1 handler.go:143] test-adapter: GET "/apis/external.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/external.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854510       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta2" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854538       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854685       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854690       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta2" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854797       1 handler.go:143] test-adapter: GET "/apis/external.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/external.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.854797       1 handler.go:143] test-adapter: GET "/apis/custom.metrics.k8s.io/v1beta2" satisfied by gorestful with webservice /apis/custom.metrics.k8s.io
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.855030       1 handler.go:143] test-adapter: GET "/apis/external.metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/external.metrics.k8s.io/v1beta1
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856559       1 provider.go:353] Got request to List All External Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856671       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/external.metrics.k8s.io/v1beta1" latency="2.015598ms" userAgent="Go-http-client/2.0" audit-ID="19e03792-9977-44a7-90d1-94ab0c72fbcb" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856867       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856988       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857256       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="3.102552ms" userAgent="Go-http-client/2.0" audit-ID="5ba1fd51-b7a2-471e-a417-38cda27534f9" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857019       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="2.47638ms" userAgent="Go-http-client/2.0" audit-ID="cde309d4-c258-4604-8805-4d3060db5a25" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856930       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857047       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857820       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="3.312865ms" userAgent="Go-http-client/2.0" audit-ID="8b6f5805-0bf3-47e0-a24a-ffb10c3dead0" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857053       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857944       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="3.554138ms" userAgent="Go-http-client/2.0" audit-ID="d14f4d02-fdb0-4139-92b5-b382ff05e5e8" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857071       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.858062       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="3.498993ms" userAgent="Go-http-client/2.0" audit-ID="d954a78f-66eb-47bc-a520-b19ef5e060af" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.856867       1 provider.go:313] Got request to List All Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.858119       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta2" latency="3.726064ms" userAgent="Go-http-client/2.0" audit-ID="a5eaf1a5-0839-4933-8db9-281f384883d5" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857099       1 provider.go:353] Got request to List All External Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.858187       1 httplog.go:131] "HTTP" verb="GET" URI="/apis/external.metrics.k8s.io/v1beta1" latency="4.140927ms" userAgent="Go-http-client/2.0" audit-ID="be03e5ce-4da2-4c62-85ec-35f91392427a" srcIP="10.244.0.1:26586" resp=200
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857103       1 provider.go:353] Got request to List All External Metrics
custom-metrics-apiserver-bb57cc947-dc2dz custom-metrics-apiserver I0102 11:35:05.857146       1 provider.go:353] Got request to List All External Metrics

During an inteval of one second (time 11:35:05) I can count:

  • 5 calls to List All External Metrics
  • 10 calls to List All (Custom) Metrics

Is this really neccessary? The default horizontal-pod-autoscaler-sync-period is 15 seconds and I really wonder whether we need to update exposed metrics that often.

Access Denied while doing curl -XPOST

I build the test-adapter-container with make build and got the container and the custom-metrics pod is up and running as shown.

kubectl get pods -n custom-metrics
NAME READY STATUS RESTARTS AGE
custom-metrics-apiserver-69d885cc57-49w6n 1/1 Running 0 69m

and then i did

kubectl proxy
Starting to serve on 127.0.0.1:8001

and on on another tab when i do
curl -XPOST http://localhost:8001/api/v1/namespaces/custom-metrics/services/custom-metrics-apiserver:http/proxy/write-metrics/namespaces/default/services/kubernetes/test-metric --data-raw '"300m"'

i am getting
localhost couldnot be resolved by dns
although the curl http://localhost:8001/api/ it returns the list of apis.

so i changed the kubectl proxy to
kubectl proxy --port 6000 --address=project-hpa.csacloud.local --accept-hosts='^.*$' --accept-paths='^.*'
Starting to serve on 10.2.16.36:6000

and now if i do

curl -XPOST http://10.2.16.36:6000/api/v1/namespaces/custom-metrics/services/custom-metrics-apiserver:http/proxy/write-metrics/namespaces/default/services/kubernetes/test-metric --data-raw '"300m"'

i get
`
Access Denied (policy_denied)

Your system policy has denied access to the requested URL.
For assistance, contact your network support team. `

`
and the logs for custom-metrics pod is as follows

POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 4 milliseconds I0116 12:39:43.988689 1 round_trippers.go:411] Response Headers: I0116 12:39:43.988700 1 round_trippers.go:414] Content-Type: application/json I0116 12:39:43.988709 1 round_trippers.go:414] Content-Length: 294 I0116 12:39:43.988721 1 round_trippers.go:414] Date: Wed, 16 Jan 2019 12:39:43 GMT I0116 12:39:43.988806 1 request.go:942] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false,"reason":"no RBAC policy matched"}} I0116 12:39:43.988974 1 authorization.go:73] Forbidden: "/", Reason: "no RBAC policy matched" I0116 12:39:43.989141 1 wrap.go:42] GET /: (5.977247ms) 403 [Go-http-client/2.0 10.32.0.1:59492] I0116 12:39:48.518217 1 request.go:942] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:generic-garbage-collector","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}} I0116 12:39:48.518379 1 round_trippers.go:386] curl -k -v -XPOST -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews' I0116 12:39:48.521342 1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 2 milliseconds I0116 12:39:48.521377 1 round_trippers.go:411] Response Headers: I0116 12:39:48.521389 1 round_trippers.go:414] Content-Type: application/json I0116 12:39:48.521400 1 round_trippers.go:414] Content-Length: 534 I0116 12:39:48.521411 1 round_trippers.go:414] Date: Wed, 16 Jan 2019 12:39:48 GMT
@DirectXMan12 can u please help me

Update the Compatibility Section in the Documentation to Explicitly Mention Supported kube-apiserver Skew

This is an Enhancement Request

What would you like to be added
Update the compatibility section in the README.md of this repository to explicitly mention the supported kube-apiserver skew for each version of the metrics server.

Why is this needed
It's essential to provide this information to ensure administrators understand the compatibility requirements and limitations when deploying different versions of components in their Kubernetes clusters.

References
kubernetes/website#46109
kubernetes/kubernetes#124533

Release v1.19.0

## Changes since kubernetes-1.11.2

### Changes
* Move custom-metrics-apiserver module to github.com/kubernetes-sigs/custom-metrics-apiserver (@dgrisonnet)

### Improvements
* Support metric label selector for custom metrics (@astefanutti)
* Expose all supported versions of metrics APIs (@44past4)
* Serve OpenAPI spec (@johanneswuerbach)
* Bump Kubernetes dependencies to v0.19.3 and Golang to v1.15 (@dgrisonnet)

cc @serathius @s-urbaniak

no endpoints available for service \"custom-metrics-apiserver\"

Hi, I'm trying to run the test-adapter. I've successfully built the docker image and deployed the testing-adapter.yaml. But I cannot write a sample metric nor can I get the existing metrics:

$ kubectl get all --namespace=custom-metrics
NAME                                           READY     STATUS    RESTARTS   AGE
pod/custom-metrics-apiserver-dd9455bc9-c2vh5   1/1       Running   0          22s

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/custom-metrics-apiserver   ClusterIP   10.100.237.125   <none>        443/TCP,80/TCP   23s

NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/custom-metrics-apiserver   1         1         1            1           23s

NAME                                                 DESIRED   CURRENT   READY     AGE
replicaset.apps/custom-metrics-apiserver-dd9455bc9   1         1         1         23s

$ curl -XPOST http://localhost:8001/api/v1/namespaces/custom-metrics/services/custom-metrics-apiserver/proxy/write-metrics/namespaces/default/services/kubernetes/test-metric --data-raw '"300m"'

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "no endpoints available for service \"custom-metrics-apiserver\"",
  "reason": "ServiceUnavailable",
  "code": 503
}

It's very strange because I can see the pod is running.

I also observed that there custom.metrics.k8s.io/v1beta1 doesn't have any resources:

$ curl http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": []
}

Any clues? I'm using Kubernetes v1.12.1. Thanks!

[Feature request] Provide binary/image

Can we provide a binary/image that simulates the APIs maintained by the repository that can be used by e2e to test components that rely on those APIs and provide a good command example for custom-metrics-apiserver?

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

How to deploy custom-metrics-apiserver in EKS

Please suggest how to deploy custom metrics api server in EKS.
We are using prometheus-operator.
We are facing issue mentioned here - #37

We used to boot custom metrics in opensource Kubernetes using prometheus operator and its version was: image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.2.0 (it works)

Empty response warning with latest kube-client (v0.26)

Recent changes in https://github.com/kubernetes/client-go have changed the behavior of discovery client.
These changes are released as part of v0.26 and produced a scenario where the tooling like kubectl or helm shows a warning during the resource discovery:

E1212 15:54:39.565250   21101 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1

As a first action, I reviewed if we are doing something wrong in KEDA but after debugging helm and kubectl I found that the reason isn't related with KEDA (prometheus-adapter produces the same error but about custom.metrics) itself.
After the changes, the discovery client list all the api resources and also all the resources inside the api resources, so the scenario where a fresh installation of KEDA/prometheus-adapter without any metric exposed automatically shows this warning on every update tooling, which is annoying (and looks hideous).

In KEDA, we return the current metrics available in response of listing requests, and I have seen that other tools like prometheus-adapter does the same for external metrics and custom metrics

Is this approach the correct? I mean, should we return the metrics instead of the metric type (like metrics.k8s.io does). Exposing all the metrics available as part of listing requests can produce timeouts if the amount of available metrics is huge, and can produce this annoying (and unrelated) warning in case of non metrics for scaling available.
I can see that this approach is also used in the example code, but if this is how we should do the things exposing metrics, the discovery client should consider this to ignore those errors.

Could you provide any help/guidance about this topic. We have received some notifications about this warning in KEDA project and users think it's related with a KEDA problem, but we follow the available example to develop our metrics server.

Is correct from custom-metrics-apiserver usage if we return a fixed collection of generic types (in our case, scaledobjects) like metrics.k8s.io api does (with 'pods' and 'nodes'). This change would solve the timeouts in huge clusters and also the issue without any metric. Considering that these metrics aren't browsable like metric-k8s-io are, maybe this could be acceptable from custom-metrics-apiserver implementation pov

Have a question about this framework and AWS EKS clusters

I know AWS EKS was lacking a lot in HPA subject, and I know you guys helped them sort stuff with metrics-server, so my question is:

Is it possible to create External metrics API server with this framework and make it work on AWS EKS?

Or AWS EKS still have issues with authentication methods and webhooks/client-ca file?

Error while building the custom metrics server

While building the custom metrics server getting this issue. Can anyone help please

sigs.k8s.io/custom-metrics-apiserver/pkg/registry/external_metrics
..\..\..\..\..\..\go\pkg\mod\sigs.k8s.io\[email protected]\pkg\registry\external_metrics\reststorage.go:40:22: cannot use &REST{} (value of type *REST) as type rest.Storage in variable declaration:
        *REST does not implement rest.Storage (missing Destroy method)
sigs.k8s.io/custom-metrics-apiserver/pkg/registry/custom_metrics
..\..\..\..\..\..\go\pkg\mod\sigs.k8s.io\[email protected]\pkg\registry\custom_metrics\reststorage.go:41:22: cannot use &REST{} (value of type *REST) as type "k8s.io/apiserver/pkg/registry/rest".Storage in variable declaration:
        *REST does not implement "k8s.io/apiserver/pkg/registry/rest".Storage (missing Destroy method)

Release v1.21.0

## Changes since v1.20.0

A release process following Kubernetes releases was added to the project.
The major and minor versions of custom-metrics-apiserver are in sync with
upstream Kubernetes and the patch version is reserved for changes in
custom-metrics-apiserver that we want to release without having to wait for the 
next version of Kubernetes.

Since the project is in sync with upstream Kubernetes releases, it is following
the same support policies, so alongside this release will be cut two other
releases (v1.19, v1.20) to have the three most recent releases of Kubernetes.

### Improvements
* Bump Kubernetes dependencies to v0.21.2 and Golang to v1.16 (@dgrisonnet)

### Tooling
* Replace Travis CI by prow (@dgrisonnet)

cc @serathius @s-urbaniak

HPA- custom -metrics-issue

Hi ,
I am new to k8s and have a huge confusion on creating HPA for the below query
we have a prometheus query : (sum(jvm_memory_bytes_used{area="heap"}) ) / (sum(jvm_memory_bytes_max{area="heap"}) )

Problem: Need to get that metrics added to prometheus adapter , so custom metrics can check those and we can implement HPA.
and we are getting those metrics from istio-system prometheus (i dnt know if this information helps)

Kops version : 1.13.1
AWS :provider

Can we start adding tags that correspond their matching K8s release?

Knative makes use of this cusom-metrics-apiserver. We also try to pin our K8s dependencies & other components to specific K8s versions because of the version skew policy.

We're not able to do that with this metrics server repo since there aren't any tags and branches. It currently involves visiting the repo and skimming through the go.mod across commits.

I was hoping this repo (at a minimum) could have branches for each k8s minor release similar kube-openapi repo.

ie. release-1.17, release-1.18, release-1.19 etc.

Metrics Aggregator

In some cases, it would be nice to be able to have a single endpoint which was able to aggregate multiple different providers of the custom metrics API into a single implementation of the custom metrics API.

This could be useful if one is manually sharding metrics collection in a large cluster, or has multiple different types of metrics collection in a cluster.

why i got an error:"F0923 17:02:39.416565 20966 helpers.go:114] Error from server (Forbidden): work-queue-length.custom.metrics.k8s.io is forbidden: User "system:anonymous" cannot list resource "work-queue-length" in API group "custom.metrics.k8s.io" in the namespace "default"""

I deployed the adapter according to the test-adapter directory and setup successfully.
I can get metricsvaluelist(command is: kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"). But when try to get a certain metricsValue(command is: kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/work-queue-length" ) ,I got the error like above,and the log show that : "I0923 08:55:11.715408 1 request.go:1068] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"resourceAttributes":{"namespace":"default","verb":"list","group":"custom.metrics.k8s.io","version":"v1beta1","resource":"work-queue-length"},"user":"system:anonymous","groups":["system:unauthenticated"]},"status":{"allowed":false}}"
How can I change the user when I make a SubjectAccessReview request? why the user is system:anonymous?

Update to K8s 1.15.x libraries

There were a number of changes in 1.15.x that break callsites in this library, so consumers of this library are unable to update to 1.15.x until this library has been updated.

We currently use this as the basis for integrating our request metrics with HPA-based autoscaling in Knative, and I'd like to see us move to 1.15.x libs in our next milestone (starts tomorrow).

Any chance we can get this moved forward?

cc @markusthoemmes @vagababov @josephburnett

Should have a lightweight version of the interface

In some cases a more lightweight version of the provider interface may be used to implement the normal provider interface, something like this:

type CustomMetricsImplementation interface {
	// ValueForMetric returns a value for a given metric for the given time frame
	ValueForMetric(name string, groupResource schema.GroupResource, namespace string, metricName string) (int64, int64)

	// ListAllMetrics provides a list of all available metrics at
	// the current time.  Note that this is not allowed to return
	// an error, so it is recomended that implementors cache and
	// periodically update this list, instead of querying every time.
	ListAllMetrics() []MetricInfo
}

(courtesy of luxas@096b8c1#diff-59a31729c3bbddcc0c8999aec0c96bf3R70).

We should provide an implementation CustomMetricsProvider in terms of this interface, for expediency getting up and running.

Error from server (NotFound): the server could not find the metric http_requests for pods

Hi๏ผŒHow do I add http_requests indicators? thx.

[root@kube-master custom-metrics-apiserver]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq . Error from server (NotFound): the server could not find the metric http_requests for pods [root@kube-master custom-metrics-apiserver]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .|grep http_requests [root@kube-master custom-metrics-apiserver]# [root@kube-master custom-metrics-apiserver]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .|grep fs_usage_bytes "name": "pods/fs_usage_bytes" [root@kube-master custom-metrics-apiserver]#

Is it possible to run it as non root?

Hello,

While setting runasNonRoot true, it throws an error. It is expecting to run as root.

By any chance is it possible to run it as non root because security is concern in production and bypassing policy is not possible.

Kind Regards
Tanul

Documentation on how to use it

Need some documentation on

  • where to add custom metric logic
  • how to use it this custom api server
  • how to deploy this api sever to the kubernetes cluster and how to aggregate it with k8s api server

Proposal: Support multiple custom/external metrics servers in the cluster

Currently only one custom or external metrics server in the cluster is supported (because there can be only one APIService resouce specified), as already discussed in #3 in #68. This way users are not able use multiple projects that are consuming this framework, eg. KEDA, k8s-prometheus-adapter or Datadog etc.

I was thinking about options how to solve this and I'd like to propose, that we will introduce aggregator and child components.

  • aggregator - that's basically the adapter that we have today (ie. it is responsible for communication with k8s api server, APIService registration etc). + additional functionality: it registers child components as they are added in the cluster, transferring the requests for particular metrics to the child components and will maintain a cache of metrics that are provided by individual childs (which could always fall back to calling ListMetrics() on each child).
    Could either use:
    • a new controller + CRD to register new child components and forward the REST call
    • use grpc for communication between aggregator & child, this way the child components could register themselves in the aggregator (by initializing the connection) and we won't need a controller and CRD for this part
  • child - very lightweight component, that's transferring the metrics to the aggregator, user will just have to implement the interfaces

With this approach, in the cluster there will be one aggregator component which will transfer requests to the individual child. Individual projects, that are consuming this framework, will have to just implement the child. If grpc is used, the whole communication could be pretty simple and we can even include this change into the existing codebase and allow some mixed scenario, where aggregator could provide some metrics as well, but I'd like to avoid this scenario.

What do you think about this proposal? Do you think that this is something that could be acceptable by this project? I can elaborate more (and formalize the proposal), if I see an interest in this.

And yeah, I can contribute this ๐Ÿ˜„

Provide default OpenAPI settings & definitions

Since #65, custom-metrics-apiserver serves an OpenAPI spec; however, it requires a configuration and the generation by the project using custom-metrics-apiserver of OpenAPI definitions using openapi-gen.

That leads to duplicated works, each implementation project having to generate its own OpenAPI definitions.

E.g.:

  • test-adapter generates definitions here and configures OpenAPI here;
  • prometheus-adapter generates definitions here and configures OpenAPI here;
  • keda generates definitions here and configures OpenAPI here.

It seams reasonable to provide these generated OpenAPI definitions in custom-metrics-apiserver, along with default OpenAPI serving settings (and let it possible to customize it).

Any thought?

/kind feature
/assign

Synchronize test-adapter and Getting Started Guide

Currently, the Getting Started Guide includes pieces of code which, combined, give a sample adapter, different from the test-adapter.

That leads to two issues:

  • code in the Getting Started Guide is not compiled or tested. The last fix in #108 showed lots of inconsistencies (e.g. different name for a same variable in the code) or outdated code (e.g. the adapter had not been updated for the Interfaces changes).
  • there's 2 "test adapters" to maintain, and both are meant to serve as an example implementation (the "test-adapter" is also used for unit tests and is compiled during CI). These test adapters may diverge.

To tackle the first issue, we could enforce the Markdown from Getting Started Guide to be synced with Go code (e.g. kube-prometheus uses mdox to check that Jsonnet examples from its Markdown documentation is synced with Jsonnet files - cf here).

Ideally, we would have a single test implementation (test-adapter).
However, test-adapter is more complex than the implementation from the Getting Started Guide...

What do you think?

resClient.List should add resourceVersion=0 parameter

resourceVersion=0 is not specified here, causing apiserver to skip the cache and go directly to etcd to read the data.
add resourceVersion=0 parameter, apiserver will read data from the cache, and there is an order of magnitude performance improvement.

error when make test-adapter-container

In the cmake build I get this error. My Go is installed correctly and I can run

srini@dev-main:~$ $GOPATH/bin/hello
Hello, world.

Would appreciate any feedback on how to resolve this

o build -o ./_output/amd64/test-adapter github.com/kubernetes-incubator/custom-metrics-apiserver/test-adapter
can't load package: package github.com/kubernetes-incubator/custom-metrics-apiserver/test-adapter: cannot find package "github.com/kubernetes-incubator/custom-metrics-apiserver/test-adapter" in any of:
        /usr/lib/go-1.10/src/github.com/kubernetes-incubator/custom-metrics-apiserver/test-adapter (from $GOROOT)
        /home/srini/go/src/github.com/kubernetes-incubator/custom-metrics-apiserver/test-adapter (from $GOPATH)

I solved the above problem with go get command however

can't read test-adapter-deploy/manifests/custom-metrics-apiserver-deployment.yaml: No such file or directory

I see this now and I dont see manifests directory in the repo should the Makefile be updated?

#40

Validate metric name while listing available metrics and getting metric values

Problem

Right now, it is possible for custom/external metrics adapter implemented using this library to expose and list metrics with names, which do not match Kubernetes resource naming convention, being as example:

The Pod "Foo" is invalid: metadata.name: Invalid value: "Foo": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

Listing and getting metrics using uppercase characters work fine when you use kubectl get --raw, however, it does not work when one requests the metric e.g. using HPA object. Attempt to do so, results in the following error on HPA object:

invalid metrics (1 invalid out of 1), first error is: failed to get external metric E2E: unable to get external metric newrelic-metrics-adapter-e2e-tests-5rl58/E2E/nil: unable to fetch metrics from external metrics API: getting fresh external metric value: getting metric value: metric "e2e" not configured

This is because either a client or API server downcases the requested resource name, so requesting metric named E2E results in client sending request for resource e2e.

Solution

I think values received from methods in ExternalMetricsProvider should be sanitized before sending this data back to the client.

Notes

If needed, this can be implemented as an opt-in feature, to avoid breaking changes.

kubernetes/kubernetes#72996 also talks about it. Perhaps other characters like / should also be not allowed. However, I'm not able to find a full list of forbidden characters.

EDIT:

It seems to me that ideally https://pkg.go.dev/k8s.io/[email protected]/pkg/api/validation/path#IsValidPathSegmentName should be used.

Release v1.27.0

Changes since v1.25.1

Breaking changes

  • Change CustomMetricsAdapterServerOptions to improve options handling (#123, @olivierlemasle):
    • removed CustomMetricsAdapterServerOptions.Complete
    • changed CustomMetricsAdapterServerOptions.Validate from func([]string) error to func() error

Improvements

Documentation

Why implementing ListAllMetrics is needed?

It needs to implement these interfaces from https://github.com/kubernetes-sigs/custom-metrics-apiserver/blob/master/docs/getting-started.md

type CustomMetricsProvider interface {
    ListAllMetrics() []CustomMetricInfo

    GetMetricByName(ctx context.Context, name types.NamespacedName, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValue, error)
    GetMetricBySelector(ctx context.Context, namespace string, selector labels.Selector, info CustomMetricInfo, metricSelector labels.Selector) (*custom_metrics.MetricValueList, error)}

Where will ResourceLister be used?

ResourceLister: provider.NewCustomMetricResourceLister(s.customMetricsProvider),

panic: Genericapiserver.New() called with config.LoopbackClientConfig == nil

This might be related to the recent dependency updates. When I call "kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver".completedConfig.New() (1) with a string, a customMetricsProvider, and nil as the parameters, the built image panics with:

Error: Genericapiserver.New() called with config.LoopbackClientConfig == nil

which looks like the error is originating from here. I don't know if this is the actual cause, but I think it may have something to do with the new 3rd parameter (externalMetricsProvider) after the dependency updates which, for my use, is nil

Should we rethink the response of listing requests?

Hi to all,

As you know, we have recently faced with the issue produced by the changes in kube-client. Basically, the issue was related with returning an empty list because empty was considered as error.

This has pushed me to think about the current situation and how we expose the custom-metrics information. I'd like to reopen the discussion about what we are returning during listing operations and if we really need to return the current metrics.
Currently, we are exposing custom/external metrics listing all of them or by name. I know that this is the global approach in Kubernetes for aggregated APIs, but I'm not sure if it makes sense (most probably due to my knowledge gap here).
Other aggregated APIs like metrics.k8s.io supports the same actions as us, but in that case, it makes sense because all the metrics available on every the namespace/node. For example, if I query

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/*/pods" | jq

I will get a list with all the available values in the namespace and it makes sense because there eventually are pod metrics on every namespace, pod as metric is generic and it's an abstraction that we can query on every namespace.
My doubts appear with custom/external metrics, for example in KEDA, we list all the metrics and expose them during listing requests. The problem is that each query is totally attached to the specific metric/namespace, I can't query

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/*" | jq

It's because this why I'm not sure about current approach. We are returning a list of metrics that can't be used for discovering anything, as we return only the metric name generated using the specific info but not the namespace. Basically, we are returning a useless list, because nothing can be done with it apart from knowing the whole list of metrics. This listing action can produce a timeout too, if the metric list is too long, printing an error without any useful information for end users. We have an endpoint which doesn't offer too much information, which is queried by the cluster/users really often, and which can produce timeouts easily in some situations. Another important point is that HPA controller doesn't use this listing endpoint at all, it uses directly the get endpoint providing namespace and metric name from the HPA.

Based on the reasons above, I think that following the same approach as metrics.k8s.io could be worth. My proposal is:

  • Listing requests in root (kubectl get --raw "/apis/custom-external.metrics.k8s.io/v1beta1") return a single metric like custom-metric and external-metric (depending on the API)
  • (Optionally) Listing requests for a metric in a namespace return a list of values for that metric on the specific namespace.

Following this approach, we could be safe from changes in discovery processes, and also avoid timeouts listing all the available metrics. I'm not totally sure about the changes required for doing this and neither if they have to be done in this repo or directly in the implementations but I wanted to share my thoughts about this.

BTW, I'm willing to contribute with the changes if we agree with change something

Possibly Doesn't Build

Let me start by saying that I'm not a full-time Go developer (or even a part-time Go developer), so it's entirely possible that what I'm about to say is 100% off-base.

  • I installed the Go tools using the .msi available from the official download page. * I configured my GOPATH.
  • I pulled down the source into the /src subfolder under my GOPATH.
  • I navigated to that folder in a terminal then executed: glide install
  • I executed: go build

I received a fountain of errors that seem like some kind of type-confusion.

The full output is:

# github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer
pkg\apiserver\installer\cmhandlers.go:148:111: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
pkg\apiserver\installer\cmhandlers.go:151:46: cannot use rootScopedHandler (type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(rootScopedPath).To
pkg\apiserver\installer\cmhandlers.go:173:124: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
pkg\apiserver\installer\cmhandlers.go:174:46: cannot use namespacedHandler (type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(namespacedPath).To
pkg\apiserver\installer\cmhandlers.go:197:132: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
pkg\apiserver\installer\cmhandlers.go:198:60: cannot use namespaceSpecificHandler (type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(namespaceSpecificPath).To
pkg\apiserver\installer\emhandlers.go:127:110: cannot use restfulListResource(lister, nil, reqScope, false, a.minRequestTimeout) (type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to metrics.InstrumentRouteFunc
pkg\apiserver\installer\emhandlers.go:129:54: cannot use externalMetricHandler (type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".RouteFunction) as type "github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".RouteFunction in argument to ws.GET(externalMetricPath).To
pkg\apiserver\installer\installer.go:79:41: cannot use ws (type *"github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".WebService) as type *"github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".WebService in argument to versionDiscoveryHandler.AddToWebService
# github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/server/routes
vendor\k8s.io\apiserver\pkg\server\routes\openapi.go:38:91: cannot use c.RegisteredWebServices() (type []*"github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".WebService) as type []*"github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".WebService in argument to handler.BuildAndRegisterOpenAPIService
vendor\k8s.io\apiserver\pkg\server\routes\openapi.go:42:97: cannot use c.RegisteredWebServices() (type []*"github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/k8s.io/apiserver/vendor/github.com/emicklei/go-restful".WebService) as type []*"github.com/kubernetes-incubator/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful".WebService in argument to handler.BuildAndRegisterOpenAPIVersionedService

Again--I'm a total novice here, so I might be way off base.

custom-metrics-apiserver compile failed with k8s.io/apiserver v0.29.2

HI, my go application using

sigs.k8s.io/custom-metrics-apiserver v1.28.0
k8s.io/apiserver v0.29.2

and it can't compile succeed because of this function **func (o *CustomMetricsAdapterServerOptions) ApplyTo(serverConfig *genericapiserver.Config) error **. In this function, it called ApplyTo in package k8s.io/apiserver, but in v0.29.2version, ApplyTo has 3 parameters. Also I compare k8s.io/apiserver between v0.29 and v0.28, in version0.28, ApplyTo has only 1 parameter.

Maybe I think custom-metrics-apiserver should fit kubernetes v1.29

Querying an invalid custom metrics API path results in a panic

Querying an incorrect custom metrics API path may result in a panic from custom-metrics-apiserver. Usually, when the query is malformed the adapter would respond HTTP 404; however, for the following query, it responds HTTP 503 and a panic can be found in the adapter logs. This panic seems to occur whenever only the first half of the query path is valid.

This error was reported by a couple of prometheus-adapter users who were misled by the query output and the logs:

This issue doesn't seem to have a direct impact on the adapter but the error is quite misleading to the end-users.

Query

$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metric_name"
Error from server (ServiceUnavailable): the server is currently unable to handle the request

Note: the valid query would be: kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/metric_name"

Logs

I1119 10:55:58.662995       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/%2A/metric_name" latency="5.664053ms"       userAgent="kubectl/v1.18.1 (linux/amd64) kubernetes/7879fc1" srcIP="192.168.122.188:49898" resp=0
E1119 10:55:58.663068       1 runtime.go:76] Observed a panic: runtime error: slice bounds out of range [2:1]
goroutine 891 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0007d0a20)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x113
panic(0x1b9ee40, 0xc0007fc500)
        /usr/lib/go/src/runtime/panic.go:969 +0x1b9
github.com/kubernetes-sigs/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.getRequestOptions(0xc00094b900, 0x1f632e0, 0xc0007651c0, 0x1f53b00,       0xc0004ac9c0, 0x1f1da80, 0xc000764900, 0x0, 0x0, 0x0, ...)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers/get.go:160 +0x63c
github.com/kubernetes-sigs/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.ListResourceWithOptions.func1(0x1f51ac0, 0xc000378d30, 0xc00094b900)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers/get.go:111 +0x4c7
github.com/kubernetes-sigs/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResourceWithOptions.func1(0xc00087dd40, 0xc0004c0690)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:297 +0xed
k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc00087dd40, 0xc0004c0690)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:384 +0x282
github.com/emicklei/go-restful.(*Container).dispatch(0xc000352ab0, 0x7f09a9ea63c0, 0xc000378d28, 0xc00094b900)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful/container.go:294 +0x65a
github.com/emicklei/go-restful.(*Container).Dispatch(...)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/github.com/emicklei/go-restful/container.go:204
k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1cc26de, 0x16, 0xc000352ab0, 0xc0002f2a10, 0x7f09a9ea63c0, 0xc000378d28, 0xc00094b900)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x539
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f09a9ea63c0, 0xc000378d28, 0xc00094b900)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x563
net/http.HandlerFunc.ServeHTTP(0xc0004e3700, 0x7f09a9ea63c0, 0xc000378d28, 0xc00094b900)
        /usr/lib/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func2(0x7f09a9ea63c0, 0xc000378d28, 0xc00094b900)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:175 +0x4cf
net/http.HandlerFunc.ServeHTTP(0xc00088a780, 0x7f09a9ea63c0, 0xc000378d28, 0xc00094b900)
        /usr/lib/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f09a9ea63c0, 0xc000378d28, 0xc00094b900)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x231d
net/http.HandlerFunc.ServeHTTP(0xc0004e3740, 0x7f09a9ea63c0, 0xc000378d28, 0xc00094b900)
        /usr/lib/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f09a9ea63c0, 0xc000378d28, 0xc00094b800)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:70 +0x672
net/http.HandlerFunc.ServeHTTP(0xc000255c20, 0x7f09a9ea63c0, 0xc000378d28, 0xc00094b800)
        /usr/lib/go/src/net/http/server.go:2042 +0x44
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc0007d0a20, 0xc0004c9680, 0x1f57140, 0xc000378d28, 0xc00094b800)
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:113 +0xb8
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
        /home/damien/github/dgrisonnet/custom-metrics-apiserver/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:99 +0x1cc

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.