scrape: Add more tests validating HTTP requests made on scrape (and their headers) about prometheus HOT 3 OPEN

I'm sure I ran 2.49.0 and prometheus had no problem scraping itself. Maybe the content negotiation implementation used by golang client is broken/doesn't follow the RFC? (accepts q>1?).

It can also be useful to have some e2e tests where we scrape real targets (not just httptest's where a lot of stuff is mocked) of different prom clients...

EDIT:

Yep, the implementation doesn't seem to check the quality value: https://github.com/prometheus/common/blob/773d5664eb8d228d2c6ba235b1af12fbf7ee9020/internal/bitbucket.org/ww/goautoneg/autoneg.go#L84

from prometheus.

I'm sure I ran 2.49.0 and prometheus had no problem scraping itself.

Yes, Prometheus seems to be rather lenient on what requests it accepts.
We also were still successfully scraping node-exporter, but not our own service's /metrics endpoint.

from prometheus.

Yep, all targets based on the golang client should have a "broken" content negotiation implementation, we'll probably need to fix that or move to another library: https://github.com/elnormous/contenttype seems to better stick to the RFC and seems to be better tested.

(having a standard implementation will be even better golang/go#19307)

from prometheus.