Failing HTTP targets result in a disproportionate amount of increase in pending TCP connections and goroutines about cloudprober HOT 2 CLOSED

Looking at http://<cloudprober_host>:9313/debug/pprof/goroutines, we noticed that:

For 72 failing targets:
1080 goroutines are in transport.dial and eventually in internal/poll.runtime_pollWait.
1080 goroutines in net.(*netFD).connect.func2+0xb9 net/fd_unix.go:129
1008 goroutines in net/http.(*Transport).getConn.func2.1+0x49 net/http/transport.go:965

from cloudprober.

I think client.Timeout is not having the desired effect on the underlying connection calls and we should switch to using request based contexts (when this probe was implemented HTTP requests didn't support contexts).

So this is what seems to be happening:

1. client.Timeout gets attached to the request in the form of a "Cancel" channel (req.Cancel) which gets closed at the deadline (set to time.Until(deadline)): src/net/http/client.go#L284

2. http.Transport's getConn() function is where most of HTTP level connection handling happens:
   net/http/transport.go:948

3. This function uses the req.Cancel to come out of itself when req.Cancel is closed, but AFAICT, it doesn't do much to close the pending connections. It calls handlePendingDial but all that seems to be doing is decreasing the outstanding connections count:
   http://google3/third_party/go/gc/src/net/http/transport.go?l=962&rcl=207119865

So even though, the HTTP request gets canceled at the timeout, underlying goroutines that handle TCP connections don't finish.

from cloudprober.