Basic python application skeleton with the common bells and whistles:
- HTTP, gRPC support
- prometheus metrics
- opentracing via jaeger
- dockerized
- helm charts
- reduced bullshit
# local
docker/build.sh # build the image, rebuild protos and copy over
vim docker/local-compose.yml # tweak as necessary; by default server won't be started
docker-compose -f docker/local-compose.yml up # start stuff locally
# run tests
docker exec pyskull python3 -m unittest discover /opt/tests/
# install prometheus (use the offical helm repo to get the manifests)
helm install stable/prometheus --name prometheus --namespace monitoring
# if running a local cluster, make sure your metric server has certain flags
stable/metrics-server/values.yaml:
...
args:
- --logtostderr
- --kubelet-insecure-tls # <- this
...
# start the metrics server
helm install stable/metrics-server --name metrics-server --namespace monitoring
# tweak prometheus-adapter configuration:
...
rules:
default: false
custom:
- seriesQuery: '{__name__=~"http_requests_count",pod!="",kubernetes_namespace!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
pod: {resource: "pod"}
name:
matches: ".*"
as: "avg_requests"
metricsQuery: 'sum(irate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
...
# start the adapter
helm install stable/prometheus-adapter --name prometheus-adapter --namespace monitoring
# finally, start the application
helm install chart/pyskull --name my-app
# add load and see it scale ...
- Check the logs of the metrics-server
- Check the logs of the prometheus-adapter and toggle the logging level to 6
- Make sure that the adapter is able to list the data from prometheus, i.e.:
I0917 14:50:59.304159 1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/default/pods?labelSelector=app%3Dpyskull%2Crelease%3Dmy-app 200 OK in 3 milliseconds # GET is succesful
- Make sure that the custom metric is available:
Yields:
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/avg_requests
{ "kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/avg_requests" }, "items": [ { "describedObject": { "kind": "Pod", "namespace": "default", "name": pyskull, "apiVersion": "/__internal" }, "metricName": "avg_requests", "timestamp": "2018-09-17T14:51:21Z", "value": "3604m" }, { "describedObject": { "kind": "Pod", "namespace": "default", "name": pyskull, "apiVersion": "/__internal" }, "metricName": "avg_requests", "timestamp": "2018-09-17T14:51:21Z", "value": "3287m" }, { "describedObject": { "kind": "Pod", "namespace": "default", "name": pyskull, "apiVersion": "/__internal" }, "metricName": "avg_requests", "timestamp": "2018-09-17T14:51:21Z", "value": "3353m" }, { "describedObject": { "kind": "Pod", "namespace": "default", "name": pyskull, "apiVersion": "/__internal" }, "metricName": "avg_requests", "timestamp": "2018-09-17T14:51:21Z", "value": "3070m" } ] }
- Finally, troubleshoot HPA:
kubectl describe hpa kubectl get hpa