Prometheus sends X-Prometheus-Scrape-Timeout-Seconds with a value like 10.000000, which is the scrape_timeout being used. Need to decide how defaults and overrides should work in this exporter; blackbox_exporter allows specifying an offset (buffer to subtract from this) as well as an overall timeout.

This is top priority for next metrics.

This is not currently possible with Bazel in its current form - the only way is to have these directly in WORKSPACE. Transitive dependencies must all be specified to the end of the chain in this way as well.

Need to rethink this, making the map write lock as tiny as possible. Is there a more efficient way to do a multi-delete? Kick off a goroutine to close each conn after map removal, then use a wait group to block until complete. Still need the lock in case a scrape is in progress. Deadlocks? Should never have a write lock on the mapper at the same time as a collector mutex.

With examples, e.g.

Power draw by manufacturer and model:

sum by (manufacturer, model) (chassis_power_draw_watts)

Number of machines by BMC firmware:

sum by (firmware) (bmc_info)

Number of machines with a cooling fault:

count(chassis_cooling_fault == 1)

Ratio of machines currently powered on:

count(chassis_powered_on == 1) / count(chassis_powered_on)

By default there are none; clients that open a conn and never send anything will cause goroutine leaks. Fix this entire class of bug. This is a good guide.

We definitely want a --collector.timeout; the question is whether we also want --scrape.timeout. Scrape timeout is the entire request; collector timeout constrains only the duration of the Collect() method.

We can get away with just a --collect.timeout, provided the user sets the scrape timeout to the collect timeout * number of scrapers. The exporter doesn't need the end-to-end timeout if it's guaranteed to enter Collect() by scrape timeout - collect timeout.

The only situation in which a scrape timeout would be useful is for slow (but responsive) BMCs, where a collect timeout alone could cause Prometheus to give up on the request before the exporter. E.g. collect timeout is 9s, Prometheus scrape timeout is 30s; collection starts at 25s and doesn't complete before 30s, meaning we lose all the metrics gathered in the 5s we were in the collect method. With the scrape timeout flag, we would've returned what we had after 3s in Collect().

It would be more defensive to have both, e.g. if a manual scrape came along by someone testing a BMC. So yes, we want both.

This should wait until bazelbuild/rules_go#2216 makes it into a release.

A single BMC returning a GCAC command failure will deadlock the entire exporter on next GC.

1. 2019-08-21 22:59: Get Channel Authentication Capabilities command fails
   • bmc_collector_session_expiries_total increments from 0 to 1, confirming the ValidateResponse() returned non-nil error.
   • Collector's prescrape() calls Close() which tries to reacquire the collector mutex, causing a deadlock, leaked scrape request goroutine and file descriptor.
   • No metrics from the /bmc endpoint recorded after this time, as the one collector in use is locked.
   • We are leaking 3 goroutines every 30s scrape due to requests waiting indefinitely on the mutex:
     • goroutine for HTTP request to /bmc, which also leaks the request fd
     • 2 goroutines inside the registry Gather() method somewhere, one of which is likely the uncancelled context creation at the beginning of Collect()
2. 2019-08-21 23:07: GC triggered
   • Acquires write lock on mapper.
   • Attempts to get LastCollection() from the collector which tries to acquire collector mutex and deadlocks.
     • This is why hits stop increasing here - we block all readers; queries keeps increasing as this is incremented before acquiring a read lock on the mapper.
     • The leak rate drops to one goroutine and fd every 30s - both for the request to /bmc. As GC has acquired the write lock, no more /bmc scrape requests can call Gather(), meaning the two additional goroutines no longer have a chance to leak (this backs up the theory that Gather() is responsible for the 8 minutes of leaking a goroutine every 10s).
3. 2019-08-22 07:26: We hit the max 1024 file descriptors
   • The web server can no longer accept requests, including to /metrics.

It doesn't make sense to expose a chassis_intrusion metric if the BMC says it doesn't support it. If the command fails, assume it is supported.

If we decode the command response successfully, we don't need to tear down the session, even if the completion code is non-normal, as the session is still in a good state. We will want to retry in the case of insufficient resources, otherwise return the error up, as we can't move past it. Doing this will invalidate this comment:

Perhaps modify that function to keep returning early iff the context has expired.

ipmitool has reversed the byte order of the first 3 components of the GUID. Who is correct?

When golang/go#36982 is closed.

Less encyclopedia, more helpful. It doesn't answer the questions people looking to run the damn thing will have - look at other exporter docs. Command line options, their defaults, the format of their input (you haven't actually provided the format of secrets.yml), exported metrics, hardware requirements.

The default grace period between SIGTERM and SIGKILL in K8s is 30s. We can wait for a few more retries.

Has been observed to stall; 0% CPU. pprof could help here.

Only some of this time is STW - the recorded duration includes firing off and waiting for the goroutines to finish.

• Firmware version changes (with a minor version >99, which is impossible for a 1 byte BCD value)
• GUID changes

Sometimes its the whole field, sometimes it's just a few bytes.
Definitely a bug, as if it were corruption, the checksum validation would fail (is the checksum definitely there for these packets?).

If a BMC shows strangeness, it's fine to treat it with care (e.g. new connection each scrape) until the process is killed, even between Close()s on the collector.

Could be an old socket full stop - not just the connection on it.
bmc_up and/or chassis_cooling_fault and/or chassis_powered_on flaps. You only need the first one to identify this. If bmc_up == 0, close the session before finishing collecting, so it is re-established next scrape.
The debug mode in #23 would help here. Particularly the error returned by the command exec attempt.

This is purely a probe, so a useful command can be used in its place.

The changes in cb27909 render the docs in 373c3ea invalid, and will likely change again when the collector and scrape timeouts are added in #35.

• IPMI v1.5 support.
• Temperatures.

There is known to be one when requests are abandoned (K8s ingress restart).

Another only reveals itself when one exporter is running.

For each BMC metric that the exporter exposes, say which command(s) it comes from, and any transformations (GUID). For metrics that have some logic behind them (Get Power Reading, temp sensors), say how the sensor is identified, what it means if it’s missing, what to do if it’s wrong (file issue).

Last scrape time is no longer exposed.

If we establish a session, but the context expires before Get Power Reading, we will incorrectly assume the BMC does not support power retrieval. There will be other similar situations.

This may also be useful in other providers, so maybe move up into session package.
Naming is file - can be aliased by importers.

Given a 8s timeout, requests can take >11s. This can be mitigated by setting a higher scrape_timeout, but should not be necessary.

$ tar xf bmc_exporter-1.0.0-linux_arm.tar.gz 
tar: .: Cannot utime: Operation not permitted
tar: .: Cannot change mode to rwxr-xr-t: Operation not permitted
tar: Exiting with failure status due to previous errors

The extracted binary still functions correctly.

This could clash with the overall machine manufacturer, which can be obtained from inventory. As the manufacturer does not change, this should be a target label.

The key this is for this not to interrupt throughput. A nice way to do it might be to drop something in each target's event loop for it to pick up when it's ready; need to ensure it cannot run parallel with GC and shutdown. The implementation of this is linked to mapping targets to http.Handlers.

Low priority as this is not a problem we currently have.

E.g. why is the mapper in a handler? Whatever is most intuitive.

SIGINT is Ctrl+C, SIGTERM is what K8s uses and is the more general way to request a process terminates.

load() it inside WORKSPACE and pass to bmc's deps() as well.

Useful for seeing the error messages produced by some BMCs, e.g. collect() []error. Adding and removing logging is doing it wrong.

Omit label?

This indicates scrapes are too far apart. Problem is what if establish fails... cannot use "does a session exist" as that may have expired.

It would be interesting, especially in the case of multiple scrapers, to expose the time between the request coming in and Collect() beginning. This might be difficult for the same reason as #13.

Each target had a goroutine (so by construction access to a BMC is serialised). Scrapes are done by looking up the goroutine’s scrape channel and sending it a struct containing the request and response writer args. The request goroutine blocks until the BMC selects that channel. This may also make lockless closing of connections easier in response to the latest outage.

Where we returned early because we ran out of time.

This will give inconsistent results when running exporters in a redundant pair.

We establish a session (using payloads that are not commands). The first thing executed is Get Power Reading, but that doesn't explain it.

In some cases, you may want to set the goos and goarch attributes of go_binary. This will cross-compile a binary for a specific platform. This is necessary when you need to produce multiple binaries for different platforms in a single build.