The current load-watcher Prometheus pkg was using the metric of instance:node_cpu:ratio to calculate the node utilization
However, when this value is still below 60%, I found another metric instance:node_cpu_utilisation:rate1m was very large and was around 90%. Apparently, the Prometheus metric had some smoothing for the metric, and the one we used may already have a smoothing over a large time window, which might be larger than 1m. Let's guess for 5m.

We are not sure which Prometheus metric is consistent with the metric obtained directly from the metric server, so there needs more testing.

We observe x509: certificate signed by unknown authority error for certain OpenShift clusters.

ERRO[0000] error querying Prometheus for query stddev_over_time(instance:node_cpu:ratio[5m]): Post "https://prometheus-k8s-openshift-monitoring......dev.rhcloud.com/api/v1/query": x509: certificate signed by unknown authority

1. I found baseUrl = "/Watcher" in pkg/watcher/watcher.go, rather than "/watcher" ?
   It can't match the trimaran plugin, which defined as "/watcher"

2. the compile env option should add CGO_ENABLED=0, used in alpine

Currently, no tests exist for each metric provider. These need to be added for code coverage and resilient clients.

version:
load-watcher v0.2.3
prometheus v2.50.1

curl 10.105.174.136:2020/watcher

Self-monitored data

https://github.com/paypal/load-watcher/issues/51 This suggestion should be open to consideration

Currently, the query is done with a host filter, we would also like to add a cluster filter when fetching metadata.

1. Could you provide a Dockerfile to build this as a container?
2. Could you provide an example deployment yaml and add tutorials on how we run this in a k8s cluster?

Add env lookup to set log level in Load Watcher for logrus from code, when running as a service.

The load watcher should work with the Prometheus monitoring stack to provide statistics of resource usage data.
The plan is to support CPU and Memory resource usages and support statistical aggregations, such as the average and the standard deviations for rolling windows include 5m, 10m, and 15m.

The following line of log needs to be logged in debug mode as it is filling up the logs:

log.Debugf("received metrics: %v", metrics)

The load-watcher binary built from Docker file may not set executable permissions, causing permissions issue during execution from entry point.

The current design of libraryClient in the release version includes watcher as follows:

// Client for Watcher APIs as a library
type libraryClient struct {
	fetcherClient watcher.MetricsProviderClient
	watcher       *watcher.Watcher
}

Watcher libraries should only be used for load-watcher. libraryClient should not include starting the watcher when the libraryClient is also used by scheduler plugins.

// Creates a new watcher client when using watcher as a library
func NewLibraryClient(opts watcher.MetricsProviderOpts) (Client, error) {
	var err error
	client := libraryClient{}
	switch opts.Name {
	case watcher.PromClientName:
		client.fetcherClient, err = metricsprovider.NewPromClient(opts)
	case watcher.SignalFxClientName:
		client.fetcherClient, err = metricsprovider.NewSignalFxClient(opts)
	default:
		client.fetcherClient, err = metricsprovider.NewMetricsServerClient()
	}
	if err != nil {
		return client, err
	}

	client.watcher = watcher.NewWatcher(client.fetcherClient)
	client.watcher.StartWatching()
	return client, nil
}

The LibraryClient failed to getLatestMetrics when using it in plugins without select {}.

The code below

is no longer valid in the latest Prometheus library.

The code should be refactored to:

config.NewAuthorizationCredentialsRoundTripper("Bearer", token, rt)

This is found when I wanted to bump the version of scheduler-plugins to k8s 1.22.

Currently, there is no job configured on PRs, which can lead to inadvertently broken builds. The build/test checks must be passed before any PR is merged.

It will be nice to have contribution guidelines defined. Also, a script to check basic code formatting issues will save time in PR reviews and avoid unintentional overlooks. This can be added as an additional check to pass PR.

Currently, there exists no health check API for load-watcher. This can be used for example to have liveness/readiness checks in Kubernetes. This can be an additional API in the existing web server.

Changelog file is missing from repo. Additionally, release notes under releases are not descriptive and lack information. It might be a good idea to automate this with Github tools.

Currently, it is confusing to know which load-watcher version is compatible with which kube-scheduler/scheduler-plugins version. We should have a table to declare the release compatibility.

Currently FetchAllHostsMetrics() function for SignalFx is not implemented.

It took me some time to find out what exactly instance:node_cpu:ratio metirc is. It seems cpu and memory metric is come from helm-charts/charts/kube-prometheus-stack/templates/prometheus/rules/kube-prometheus-node-recording.rules.yaml rule which is is removed and seems be repalced by instance:node_load1_per_cpu:ratio rule in later verison. I think it is better to have detail description about cpu and memory metric and provide a way to configure name of cpu and memory metric.

I wonder if it is in the roadmap to support other metrics in load-watcher. I can help with the development if it would make sense :)

For instance, I would like to include metrics regarding bandwidth retrieved from prometheus:

node_network_transmit_bytes and node_network_receive_bytes

GetLatestWatcherMetrics can return nil, nil in case of e.g. HTTP error code 403. (

@zorro786

The load-watcher can either be used as a microservice and as a library.

• We need to maintain release versions so Trimaran scheduler plugins can always work with a stable release version where others can continue to contribute to other branches.
• Besides, it will make sure a specific version of Trimaran plugins work with the same release version of docker image and library regardless of whether running in client mode or in package mode.
• Need an official docker hub account for the image.

Hi all,
I found this line in README:
kubectl create -f manifests/load-watcher-deployment.yaml
but I did not find the manifests/load-watcher-deployment.yaml in repo.
Maybe a sample deployment file is needed?
thanks.

Currently, logs do not print line numbers which makes it a bit difficult to debug. The logging library we use sirupsen/logrus has the option for enabling it.

Since metricsprovider package is meant for internal use, we should move it to internal/.

In FetchAllHostMetrics() method implementation by SignalFx client, the query is done to fetch all the metrics from SignalFx server with FQDNs, as added by SignalFx to Kubernetes node names. For example,

Node name like "test" is stored with its FQDN as "test.gcp.us-central.com". We would like to strip the node name from this.
We need a filter per cluster to avoid conflicts across clusters when we strip, and also for better query performance.

When trimaran plugins run clients to obtain metrics from a load-watcher service, it can write both HTTP queries to get data from load-watcher service or can use load-watcher client library to get metrics.

1. We do not have load-watcher client library now.
2. We should allow users, such as Trimaran plugins, to use all load-watcher as libraries for all clients load-watcher support.
3. The existing load-watcher clients, such as prometheus, k8s and signalfx, should have the same interface as the new load-watcher client to add.