Comments (6)
kawych
commented on June 13, 2024
This 1) is disabled by default (enabled by a flag), 2) shouldn't affect direct-to-sd exporter application.
The example still works for me. Please provide more information so I can reproduce your issue:
- Cluster version
- Command you used to create the exporter (e.g.
kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/examples/direct-to-sd/custom-metrics-sd.yaml) - Output of
kubectl get deployment custom-metric-sd -o yaml(assuming you used the command above). - Do you have Stackdriver Monitoring API enabled? Do you have premium stackdriver account?
from k8s-stackdriver.
Glennvd
commented on June 13, 2024
Cluster version: 1.9.3-gke.0
Create command: kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes-engine-samples/master/custom-metrics-autoscaling/direct-to-sd/custom-metrics-sd.yaml
kubectl logs output:
16:46:26 › kubectl logs custom-metric-sd-7b9b98f96d-kg5d9
2018/03/12 15:46:03 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 15:46:08 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 15:46:13 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 15:46:18 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 15:46:23 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 15:46:28 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
Kubectl get output:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: 2018-03-12T15:46:00Z
generation: 1
labels:
run: custom-metric-sd
name: custom-metric-sd
namespace: default
resourceVersion: "13853398"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/custom-metric-sd
uid: 76d59a2e-260c-11e8-8cf9-42010a84007c
spec:
replicas: 1
selector:
matchLabels:
run: custom-metric-sd
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
run: custom-metric-sd
spec:
containers:
- command:
- /bin/sh
- -c
- ./direct-to-sd --metric-name=foo --metric-value=40 --pod-id=$(POD_ID)
env:
- name: POD_ID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
image: gcr.io/google-samples/sd-dummy-exporter:latest
imagePullPolicy: Always
name: sd-dummy-exporter
resources:
requests:
cpu: 100m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2018-03-12T15:46:01Z
lastUpdateTime: 2018-03-12T15:46:01Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
Stackdriver monitoring API is enabled and is successfully working for non container metrics in the same project. We do have a premium account.
from k8s-stackdriver.
kawych
commented on June 13, 2024
Thanks for the details. I noticed that your deployment uses different image than specified in the yaml file: gcr.io/google-samples/sd-dummy-exporter:latest instead of gcr.io/google-containers/sd-dummy-exporter:v0.1.0.
@bskiba do you know what does this image do (e.g. what version of code is it)?
from k8s-stackdriver.
Glennvd
commented on June 13, 2024
Hey @kawych, I just recreated the pod with the different image but the error is the same.
glennvandamme at Glenns-MacBook-Pro.local in [~]
17:16:41 › kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/examples/direct-to-sd/custom-metrics-sd.yaml
deployment "custom-metric-sd" created
horizontalpodautoscaler "custom-metric-sd" created
glennvandamme at Glenns-MacBook-Pro.local in [~]
17:17:12 › kubectl get pods
NAME READY STATUS RESTARTS AGE
custom-metric-sd-757d4b775d-kvdc2 1/1 Running 0 17s
glennvandamme at Glenns-MacBook-Pro.local in [~]
17:17:30 › kubectl logs custom-metric-sd-757d4b775d-kvdc2
2018/03/12 16:17:00 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 16:17:05 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 16:17:10 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 16:17:15 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 16:17:20 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 16:17:25 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
2018/03/12 16:17:31 Failed to write time series data: googleapi: Error 500: One or more TimeSeries could not be written:
An internal error occurred: timeSeries[0], backendError
glennvandamme at Glenns-MacBook-Pro.local in [~]
17:17:34 › kubectl get pod custom-metric-sd-757d4b775d-kvdc2 -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: 2018-03-12T16:16:57Z
generateName: custom-metric-sd-757d4b775d-
labels:
pod-template-hash: "3138063318"
run: custom-metric-sd
name: custom-metric-sd-757d4b775d-kvdc2
namespace: default
ownerReferences:
- apiVersion: extensions/v1beta1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: custom-metric-sd-757d4b775d
uid: c9952901-2610-11e8-8cf9-42010a84007c
resourceVersion: "13856212"
selfLink: /api/v1/namespaces/default/pods/custom-metric-sd-757d4b775d-kvdc2
uid: c996556c-2610-11e8-8cf9-42010a84007c
spec:
containers:
- command:
- /bin/sh
- -c
- ./sd_dummy_exporter --metric-name=foo --metric-value=40 --pod-id=$(POD_ID)
env:
- name: POD_ID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
image: gcr.io/google-containers/sd-dummy-exporter:v0.1.0
imagePullPolicy: IfNotPresent
name: sd-dummy-exporter
resources:
requests:
cpu: 100m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-z7mqb
readOnly: true
dnsPolicy: ClusterFirst
nodeName: gke-mewthree-pre-emptible-standard-a47a9e9c-k83f
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-z7mqb
secret:
defaultMode: 420
secretName: default-token-z7mqb
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2018-03-12T16:16:57Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2018-03-12T16:17:00Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2018-03-12T16:16:57Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://1a3a51022b8a187c746ecfe1479d0872a416429bc63d0530cdf5a2b019f33029
image: gcr.io/google-containers/sd-dummy-exporter:v0.1.0
imageID: docker-pullable://gcr.io/google-containers/sd-dummy-exporter@sha256:71cc99ca8d37fa10d110f77af2ecf580938f08493f7158b270ddf981ffdbf7df
lastState: {}
name: sd-dummy-exporter
ready: true
restartCount: 0
state:
running:
startedAt: 2018-03-12T16:17:00Z
hostIP: 10.240.0.64
phase: Running
podIP: 10.56.41.13
qosClass: Burstable
startTime: 2018-03-12T16:16:57Z
from k8s-stackdriver.
kawych
commented on June 13, 2024
Hi @Glennvd
Sorry for long time without response. I'm not sure yet what the issue is, I think it would help to see the error messages from Stackdriver. Can you try and use API Explorer and query Stackdriver v3 API (TimeSeries.list) for metric custom.googleapis.com/foo for pod with ID c996556c-2610-11e8-8cf9-42010a84007c? If you need help with that, I will provide more details tomorrow, I think it will be also useful to log the full error message from Stackdriver.
from k8s-stackdriver.
Glennvd
commented on June 13, 2024
Hey @kawych, I actually raised this error with google cloud support this weekend and they've confirmed that this was an issue with certain configurations on their side. We can close the issue.
from k8s-stackdriver.
Related Issues (20)
- Need a non-root user for the fluentd image (k8s.gcr.io/fluentd-gcp:1.30)
- Permission to create Metrics of any MetricType
- Issue with metrics.k8s.io/v1beta1 HOT 2
- error calling MarshalJSON / unsupported value: NaN HOT 7
- Custom Metrics Adapter Applicability to GKE Autopilot HOT 3
- custom-metrics-sd-adapter: 403s when fetching metrics from another project
- Autoscaler throws FailedGetExternalMetric after upgrade to 1.22 HOT 1
- HPA doesn't scale down to minimum size when no data is available from external metrics API server
- prometheus-to-sd | 403 Permission error, with correct Service Account permissions. HOT 2
- Custom stack driver pod doesnt seem to have liveness/readiness probe.
- Running custom-metrics-stackdriver-adapter as non-root user HOT 1
- Stackdriver metrics are not supported by HorizontalPodAutoscaler HOT 2
- ClusterRole in stackdriver adapter has namespace defined
- custom-metrics-stackdriver-adapter not working - auth problems? HOT 13
- Security Policy violation Binary Artifacts HOT 5
- Allow custom metrics from a different pod HOT 3
- Custom metrics adapter spewing errors "apiserver was unable to write a fallback JSON response: http2: stream closed" HOT 11
- custom-metrics-stackdriver-adapter - couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1 HOT 3
- istio.io/service/server/response_latencies metric that HPA collected is different to Cloud monitoring HOT 1
- Filtering metrics by labelSelector in external.metrics.k8s.io api doesn't work HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k8s-stackdriver.