prometheus-community / pushprox Goto Github PK

In the Dockerfile the go get command pulls github.com/robustperception/pushprox/proxy and github.com/robustperception/pushprox/client

RUN go get github.com/robustperception/pushprox/proxy
WORKDIR $GOPATH/src/github.com/robustperception/pushprox/proxy
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build

RUN go get github.com/robustperception/pushprox/client
WORKDIR $GOPATH/src/github.com/robustperception/pushprox/client
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build

It seems like the directories got moved in github.com/robustperception/pushprox/cmd/{proxy,client}
To get the docker build working I had to change the path in line 6,7,10,11,21 and 22.

Was this the right way to go or did I overlook something?

Two other things I stumbled upon

• in your circleci builds you updated to golang:1.13 while in the dockerfile still golang:1.12 is used.
• in the build stage $GOPATH is used but when the binaries get copied /go is hardcoded. As no GOPATH default is set in the Dockerfile this could break when golangs GOPATH changes

I am looking at running the client on some raspberry pi devices acting as edge proxies into a small on site network for clients.

I can build the ARM client myself, but was just considering that this would be nice to have pre-packaged as part of the official build to ease updates etc.

A good question for a FAQ:

What is the value add of PushProx over a common reverse proxy like nginx?

Hi,I use prometheus operator,but I do not know how to integration with pushProx?
How do I configure the prometheus operator to integration with pushProx?
Any help is appreciate

I'm trying to run pushprox but I'm getting the following error:

caller=main.go:179 msg="Error reading request:" err="malformed HTTP request """

My setup is ALB (http->https redirect) -> Nginx (basic auth) -> pushprox-proxy.
The clients however are on HTTP only.

Since I figured it might be something to do with mixing https and http I got rid of 301 redirect to https and tried serving and connecting to the proxy over HTTP. I also tried getting rid of the Nginx in font that was providing basic auth but the problem remained the same.

This appears to be the only errors I'm seeing. The proxy doesn't appear to be outputting any errors.
And of course no metrics are being scraped.

I would like to run the pushprox-client on my Raspberry PI using Docker. Could you please add arm support for your Docker image?

 user 暭 curl http://engineering:8080/clients
[{"targets":[""],"labels":null}]
 ~/src/go 暭 
 user 暭 

with

 user 暭 bin/client --proxy-url=http://engineering:8080/ --log.level=debug --fqdn="engineering"

and

 user  bin/proxy 
level=info ts=2017-10-03T09:10:19.78520441Z caller=proxy.go:104 msg=Listening address=:8080

Hi,

Does anyone have working nginx reverse proxy config?

I have two problems:

• on virtualhost based configuration, nginx try to evaluate the destination url itsef (something:9100/metrics), so prometheus got 404
• if i setup vhost without servername, then similar problem with subpath (/metrics)

Any ideas?

Thanks.

The check below in the client code at line https://github.com/RobustPerception/PushProx/blob/master/client/client.go#L112 seems incorrect, please confirm.

	if request.URL.Hostname() != fqdnURL.Hostname() {
		c.handleErr(request, client, errors.New("scrape target doesn't match client fqdn"))
		return
	}

If fqdn provided to the client is "xyz" and the request url is "xyz:8000", the fqdnURL.HostName() will be empty ("") while the request.URL.Hostname() will be "xyz".

Hi!
I'm having troubles running PushProx on my staging server.

The most informative error I see is from proxy client receiving bad request:

$ ./pushprox-client --proxy-url=http://prom-proxy-stg.***.com/ --fqdn=`hostname` --metrics-addr=":9368"
level=info ts=2021-09-22T10:26:52.905Z caller=main.go:245 msg="URL and FQDN info" proxy_url=http://prom-proxy-stg.***.com/ fqdn=argon-u1910
level=error ts=2021-09-22T10:27:53.467Z caller=main.go:205 msg="Error reading request:" err="malformed HTTP request \"\""
level=error ts=2021-09-22T10:28:55.319Z caller=main.go:205 msg="Error reading request:" err="malformed HTTP request \"\""

And then probably because of that pushprox-proxy fails to retrieve data from the client:

caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=argon-u1910
caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"

Some more details:

The client gets successfully registered at proxy/clients page: [{"targets":["argon-u1910"],"labels":null}]

And Prometheus is able to see that client live, but fails to scrape.

Prometheus config files and dashboard screenshots:

If I run proxy-server locally - everything is fine. So the problem is probably somewhere in the stack on which I'm trying to host it on cloud my servers. Should there be some additional ports open or something? Might the problem be in some http/https redirect where the request body is lost?

Any help will be much appreciated!

Thank you!

I have a usecase where a retry every 5 seconds is excessive and would like to contribute customizations for the retries. The PushProx client currently hardcodes the timings for connection retries:

The github.com/cenkalti/backoff package (documentation) provides an exponential backoff algorithm. I'd use that instead of the custom implementation and provide flags:

• --retry.initial-wait=<duration>, default 50ms
• --retry.multiplier=<float>, default 1.5
• --retry.max-wait=<duration>, default 5s
• --retry.max-elapsed=<duration>, default 0 (keep retrying forever, otherwise give up and terminate after given duration)

The default behaviour would be comparable to what's currently implemented while permitting customization.

@SuperQ @brian-brazil Would you be happy with such flags?

I have been struggling to come up with a way to work with multiple exporter, but can not figure out how.

I can scrape the metrics from the client, based on the config in prometheus.yaml file.
But then I have to add the port no of node_exporter (manually) so prom can ask the pushprox proxy and goes to pushprox client and comes back with the metrics from the client node.

Now when I have couple of more exporters running on the client side, how do I configure prometheus to automatically figure out the port of the exporter which is running on the client side. so I dont have to enter the port no in the config file manually.

I'm not sure if this is possible or not.

Whats with the label, I see its always empty, how can I add labels

[{"targets":["0532c74f7e43"],"labels":null},{"targets":["dev-node-exporter"],"labels":null}]

Thanks.

Hi,

I have a use case that I believe would be fairly common for PushProx: My prometheus server is behind an https+basicauth endpoint over the internet, but the client part of PushProx doesn't have support for user/pass authentication. Is this something that would make sense to add to PushProx?

Is PushProx supporting gzip Content-Encoding, at least for the scrape request between the proxy and client?

Assuming so, it's perhaps worth noting in the README.

great job, this project does exactly what we need when the clients are unreachable. imagine that your clients are different companies and therefore no VPN or shenanigan like that is authorized. Client must push.

is this project still maintained?
will you add https for the pushprox-proxy one day?

has anyone yet measured how many connections a pushprox process can handle?

same for a typical load balancer like GLB which may be put in front of multiple pushprox instances-- how much in resources with the load balancer consume given the persistent connections?

I wonder if the pushprox client can use a heuristic to reduce the connection time of /poll. For example, don't initial /poll unless half of the recently observed scrape period has elapsed. That could reduce global connection count by 50%?

With #27, the HTTP Transport was configured as a custom transport to support the TLS values. A new Transport that is created does not maintain the values of DefaultTransport and instead sets multiple values like timeouts, keepalive, and connection counts to zero values.

Since this change, I have noticed that clients that may have temporary disconnects from the push-prox proxy may have their connections break and never reestablish.

PR #45 adds the values found in http.DefaultTransport so the client acts the same as it did before. I have tested this in my environment for several days now, and it has resolved the failure to reconnect issues I was seeing.

(Hi! Long time listener, first time caller. Great tool!)

For context: We monitor some of our customers' servers by installing a pushprox and a node exporter on their machine, and on our end we run a pushprox server (exposed to the internet through a reverse proxy) that can reach our internal prometheus server.

This is the use case: One of our customer's servers, in addition to not being exposed to the internet, is only able to get to the internet through a proxy, Squid I believe. Also, it is not a transparent proxy.

Is there a way for the pushprox client to explicitly specify an HTTP proxy for it to reach the outside world?

Some use cases may need to scrape multiple endpoints on the same FQDN, such as localhost.

For example, in a Kubernetes pod you might have multiple metrics endpoints to scrape, localhost:9100/metrics, localhost:9101/my-metrics, etc.

You can't just run another copy of the client and still get service discovery, since the client only forwards the FQDN.

What do you think about allowing the client to specify labels, ports, and metrics paths for discovery?

One way to do this might be changing --fqdn to --endpoint and maybe allowing it to be specified multiple times. An example --endpoint value might be http://localhost:9100/metrics?app=myapp, where app=myapp would be turned into labels for discovery.

I can work on a PR if you think this or similar is a good idea.

Additionally, the same values could optionally be used for security, as talked about in PR #41.

Hi,
Would it be possible to have an official release? or push a latest version on quay/docker hub (prom ?)?
I would like to integrate pushprox in my GKE monitoring cluster (prometheus stack) and i don't want to add build stage in my CI only for pushprox-client and proxy
Ty,

The /clients endpoint's labels field would be the ideal way to provide static target-based labels for relabeling metrics. Our use-case requires each scrape target to have an environment, region and component tags which are crucial when we're using these metrics.

If we could add a --labels argument to pushprox-client that would be perfect for us. What do others think? I haven't used Prometheus for long so maybe there's a better way to do this, though I'd be glad to raise a PR for this if others also would like it.

Would it be possible to have an official release? I created a PR on puppet-prometheus to install the pushprox_client, but I had to fork this project and do an unofficial release so that Puppet can deploy the binary, which I'd like to avoid : https://github.com/voxpupuli/puppet-prometheus/pull/333/files#diff-dfc467e95c8c22086eeeb7dfb5ba0114R20

This started out as a PoC, so there's no unittests yet. That should be fixed.

Hi all,

I am seeing several msg="Error WaitForScrapeInstruction messages in the server log, with corresponding msg="Error reading request:" err="invalid method "<!DOCTYPE"" messages on the client log.

As an example, see the below traces:

Server:
Aug 08 16:11:32 hosting.metersense.com proxy[13178]: level=info ts=2019-08-08T20:11:32.822Z caller=coordinator.go:123 msg=WaitForScrapeInstruction fqdn=myclient.harrissmartworks.com

Aug 08 16:11:32 hosting.metersense.com proxy[13178]: level=info ts=2019-08-08T20:11:32.822Z caller=proxy.go:129 msg="Error WaitForScrapeInstruction:" err="request is expired"

Client:
level=error ts=2019-08-08T20:11:31.805Z caller=client.go:167 msg="Error reading request:" err="invalid method \"<!DOCTYPE\""

(The client and server are synchronized to within a couple seconds of each other, with a 4-hour timezone difference.)

These errors are present every minute -- except on those minutes when the actual scrape request comes in, which is handled successfully. The current scrape interval is 10m.

I also noticed there is no "WaitForScrapeInstruction" method defined in client.go, but am unable to tell if that is the cause for the errors.

I logged #56 recently, and while I cannot discard the possibility that the two issues are connected, I thought it best to keep them separate for now. As mentioned there I am running the latest version of both the client and proxy software.

Hello,

Sorry if I missunderstand the code (new to go), but based on https://github.com/RobustPerception/PushProx/blob/master/client/client.go#L163 and https://github.com/RobustPerception/PushProx/blob/master/client/client.go#L235 am I right to understand that the client can only do one scrape per second?

So for example if you want to avoid NAT and have 10 equipements you want to monitor every 5 seconds, PushProx will not "work" right? I guess the simple solution is to run several clients (e.g 1 per equipment).

is it okay that the FQDN of client is written into ip+port? for example, localhost:9100，192.168.x.x:9100 and so on...

There are several places where we don't check error returns. These should be fixed if possible/necessary.

Hi guys,
After setup the proxy and the client, i ran them and it works as expected but i don' know why i am getting "server returned HTTP status 502 Bad Gateway".
Thanks in advance

Are there mechanisms in the client to add some kind of authorization headers for accessing the proxy already? E.g., BasicAuth or some custom header containing an API key or similar? The documentation on authentication/authorization is a little vague there.

Use case:

• Run PushProx e.g. inside a Kubernetes cluster, so that Prometheus can access it directly via Kubernetes FQDN (http://pushprox.default.svc.cluster.local:8080 or similar)
• Expose PushProx via APIm (Kong/glorified nginx) to the outside, so that you get routed to PushProx if you have the right set of credentials
• Pass on authorization parameters to the client call outside the cluster, so that it can access the proxy instance via the nginx --> This is where the authorization is required, e.g. a custom header (X-ApiKey) or some other type of credential, possibly OAuth2 Client Credentials flow.

Picking up the ball from my issue on pushgateway pointing to https://github.com/Haufe-Lexware/pushgateway-pruner ...

Would this work in principle, or did I once more think in the wrong direction?

I was thinking of something like

client --proxy-url=<...> --username=someuser --password=somepassword

or

client --proxy-url=<...> --custom-header=X-ApiKey=somesecretapikey

Hi, would like to have TLS encryption between the prox client and prox proxy. The recommendations so far has been using a reverse proxy. I'm wondering how this is possible with the proxy overwriting host when sending the response which breaks the routing in the reverse proxy. I assume I'm missing something so I'd appreciate if someone with experience setting this up nudging me into the right direction.

I was trying to Dockerize the proxy in my personal fork

Because I would like to use the file_sd_configs in Prometheus scrape config to get the scrape endpoints dynamically, I decided to send the fqdn along with the port. If I check the /clients endpoint, I now have valid scrape configs in there. So far so good.

I did some tests locally in Visual Studio Code, everything worked like a charm. As soon as I start the Docker Container for the proxy, the problems begin. The response for /poll sometimes works and sometimes not. If I do the exact same thing without Docker it's working reliable.

Does anyone have an idea? I stuck in this for more than one day. Probably someone with Go and Docker Know-How can support me here.

In meantime I will try to setup a remote debugging environment for Go and Docker to get a better overview.

Many Thanks
e-bits

Notice:
I'm in clear mind that I will loose the ability to scan different Ports if I inject the port already on client start.

In SpringBoot based uservices Prometheus endpoint is not necessary /metric but is is configured to something like /xx/prometheus. If health check use xx/status it is not possible to remove /xx prefix without breaking the health.
PushProx should provide command line parameter to configure non default metrics endpoint.

Hi!

ATM every response for a request to /clients on the proxy generate an error log entry.
I think the log level in this line should be changed from Error to Info.
https://github.com/RobustPerception/PushProx/blob/46f8f2fcf85f5b502b06e3f43e078f44c9d6b943/proxy/proxy.go#L97

It would be very nice, that after https has been implemented with:
params:
_scheme: [https]
that we could also add some tls_config, like ca_file or insecure_skip_verify: true in order to be able to pull some metrics from a ssl target that has a private certificate.
I'm getting this error:
caller=main.go:85 err="failed to scrape https://fqdn:port/metrics: Get \"https://fqdn:port/metrics\": x509: certificate signed by unknown authority"
Thanks

go get github.com/RobustPerception/PushProx/client
can't load package: package github.com/RobustPerception/PushProx/client: case-insensitive import collision: "github.com/RobustPerception/PushProx/vendor/github.com/alecthomas/template" and "github.com/robustperception/pushprox/vendor/github.com/alecthomas/template"

also

# github.com/robustperception/pushprox/client
src/github.com/robustperception/pushprox/client/client.go:78: undefined: time.Until

also

go get github.com/robustperception/pushprox/proxy
# github.com/robustperception/pushprox/proxy
src/github.com/robustperception/pushprox/proxy/coordinator.go:94: r.URL.Hostname undefined (type *url.URL has no field or method Hostname)

go version 1.7.6.

Some use cases require being able to protect against a bad actor on the proxy side. E.G. someone with access to the proxy could send valid yet malicious requests through the client, defeating the purpose of the network barrier.

For example, if I run node-exporter and mysql on the same machine, running the client on that machine would allow access to mysql by a compromised prometheus environment. I think many people's security requirements won't allow that.

How do you feel about a patch that adds a backwards compatible --override-url option to the client?

#41 has no feedback, thus creating this issue.

This crept in when MR 11 was merged. Needs to be unified, otherwise scraping always fails with "context deadline exceeded" timeouts.

Hello,

I want to know if it is possible to build the client for a Windows Server target?

I have a server stuck behind a NAT and I would like to use pushProx to recover metrics.
I do not know GO very well, I understand that it was possible.

Thank you all

Tried to make it work but on every scrape the client results in a runtime error:

feb 28 21:19:13 zbook pushprox-client[23421]: panic: runtime error: invalid memory address or nil pointer dereference
feb 28 21:19:13 zbook pushprox-client[23421]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x6c85ef]
feb 28 21:19:13 zbook pushprox-client[23421]: goroutine 1 [running]:
feb 28 21:19:13 zbook pushprox-client[23421]: main.loop(0x8f3ce0, 0xc420075ce0)
feb 28 21:19:13 zbook pushprox-client[23421]:         /home/yosh/go/src/github.com/robustperception/pushprox/client/client.go:127 +0xaef
feb 28 21:19:13 zbook pushprox-client[23421]: main.main()
feb 28 21:19:13 zbook pushprox-client[23421]:         /home/yosh/go/src/github.com/robustperception/pushprox/client/client.go:147 +0x381

Also get a runtime error when the proxy is not reachable.

(Sorry to open this "not so technical" issue, but...) Any interest of integrating the work done in adobe/PushProx? There hardcoding sounds like a good idea generally.

https://github.com/adobe/PushProx#security

I couldn't find in the documentation just how often the client polls the proxy.

I presume the answer is once per second, as per the below line in client.go?

time.Sleep(time.Second) // Don't pound the server. TODO: Randomised exponential backoff.

Thanks!

In some situations I have encoutered it would be very helpful to advertise a given FQDN while targeting a hostname different from such FQDN.

One situation in which I would find this quite helpful is having to scrape multiple environment through a common PushProx proxy instance. If those environments are basically clones (e.g.: they are created via docker-compose and PushProx client itself is running inside a container) I end up havin to expose via --fqdn the same hostname (i.e.: node-exporter) for many instances which, in turn, leads to clashes at proxy component level.

In this case it would be nice to be able to provide a --target=node-exporter flag at client level that along with a different --fqdn for each environment (e.g.: --fqdn=dev.foo.com, --fqdn=qa1.foo.com, --fqdn=qa2.foo.com) would allow to point to the same proxy.

I put together a PR for this: #80.

Is there a way to set the log level of pushprox?
At default it logs at level INFO, which is quite a log and spams the log directory :(

Would be nice to support the -log.level flag, like alot of other go programs do.

Hello Guys,

i'm trying to set up, a prometheus monitoring stack, that gathers information from different networks. Therefore i tried to installl one central proxy and a proxy client in each of the networks. That works fine so far, if i try to scrape exporters which run at the same machine as the client is running. Is it possible to target exporters on another machine in the same network via the agents, if yes how would i configure that. If no is the solution to host a proxy in each of these networks and a client on each machine i want to scrape exporters from?

Best regards
Andi

With Prometheus 2.28, there is now support for http_sd_configs. This would allow the proxy to not need to write out files to the Prometheus server.

Greetings,
First of all ,thank you for your effort and making this an open-source project.

Is it possible to group clients by "job" as explained here : https://prometheus.io/docs/concepts/jobs_instances/

This is because my clients belong to different services like "api-service" , "kafka-connect","spark"..etc

Hi all,

I've been chasing down an issue with gaps in my data series for the past few days, and I believe I've narrowed it down to scraping thru PushProx with an interval longer than 5m.

I'm attaching two screenshots showing the gap with the "up" metric, but the other metrics also show the gap. The scrape interval is 10m (which we cannot shorten, since we are collecting CPU-expensive metrics among other reasons.)

The tooltip text shows that the data seems to be quote-unquote "expired" after 5 minutes. I have tried the following to no avail:

• Downloaded the most recent proxy and client software
• Running the proxy software with flag "--registration.timeout=20m"

While I could work around this when visualizing with Grafana ("connect null values" does the trick nicely), the problem is that it affects alerting, as each gap seems to reset the alert period.

I wonder if this might have to do with the "WaitForScrapeInstruction" errors that I have been seeing in logs, which I'll log as a separate issue to keep things organized.

Thanks,
Saul

There's no metrics yet, it should really have them.

GC of client that was still running took place out of nowhere:

level=error ts=2017-10-03T09:21:09.446332474Z caller=proxy.go:97 msg="Responded to /clients" client_count=1
level=info ts=2017-10-03T09:21:28.653546534Z caller=coordinator.go:179 msg="GC of clients completed" deleted=0 remaining=1
level=info ts=2017-10-03T09:22:28.653595779Z caller=coordinator.go:179 msg="GC of clients completed" deleted=0 remaining=1
level=info ts=2017-10-03T09:23:28.653617421Z caller=coordinator.go:179 msg="GC of clients completed" deleted=0 remaining=1
level=info ts=2017-10-03T09:24:28.653431992Z caller=coordinator.go:179 msg="GC of clients completed" deleted=0 remaining=1
level=info ts=2017-10-03T09:25:28.653501368Z caller=coordinator.go:179 msg="GC of clients completed" deleted=0 remaining=1
level=info ts=2017-10-03T09:26:28.653597843Z caller=coordinator.go:179 msg="GC of clients completed" deleted=1 remaining=0
level=info ts=2017-10-03T09:27:28.653382959Z caller=coordinator.go:179 msg="GC of clients completed" deleted=0 remaining=0

Client is still running.

Restarting proxy re-registers the client as the client retries.