Comments (5)
That was our best guess :( Hmm, you're right that the case
re := <-resc
has to free the roundtrip method when the readLoop reaches its last select, there must be something else.The problem is that the situation with stuck goroutines is reproducible only in production. A gateway service (Golang, HTTP client) communicates with a Java service (HTTP server here) using PUT call. Both HA, initially 2->2 pods before scaling. When the gateway is bombed by requests generated by dozens of pods (simulated DDoS - hundreds of requests per second on each of them), the Java service starts to become slow and latencies rise, the number of concurrently processed requests in the gateway goes from ~tens to ~thousands. We let the load test go in such frequency for ~15-30 seconds and then turn it off (to reproduce the issue but being under k8s memory limits - not kill the pods and before the services scale up and spread the load handling). After this load test a metric watching the number of currently processed requests shows values around ~lower hundreds (and never returns to the normal level of ~tens). The same number of stuck goroutines is visible in pprof dump.
So the only information we have is the pprof dump pointing us to the roundtrip method. Still far from writing a test, searching for the exact reason :(
We set ResponseHeaderTimeout, there is deadline on context, we tried it without trace round tripper, disabled keep alive, with different golang versions - 1.21, 1.22 but nothing from this helped to stop the goroutines stuck.
Any ideas on how to proceed further to localize the bug? Thank you!
I'll keep investigating this. In the meantime, please inform us by updating this issue thread if any helpful info pops into your head, thanks!
from go.
CC @neild
from go.
This issue may have something to do with 212d385 and 854a2f8.
As we went deeper we saw a possible place in the net/http/transport.go that could cause the issue. When the readLoop is waiting when reading the body of the response and the pc.roundTrip waits until pgClosed/cancelChan/ctxDoneChan, it might happen, that the select in readLoop processes the rc.req.Cancel or rc.req.Context().Done(), calls and removes the cancelKeyFn from the transport.reqCanceler and the pc.roundTrip.pcClosed case will never have canceled=true or even pc.t.replaceReqCanceler()=true because the transport.reqCanceler won't contain the cancelKey anymore.
The prerequisites for this to happen are:
- the processing timeouted or is canceled
- the rc.reqCancel or rc.req.Context().Done() is processed by readLoop
Nonetheless, I'm not so convinced by this analysis because if the client has made its way to here:
Lines 2272 to 2277 in 065c5d2
, a response should have been sent out, which would set free the roundtrip method from blocking in
select{...}
:Lines 2708 to 2718 in 065c5d2
Any chance you can write a test that can reproduce this issue? Thanks!
@rschone
from go.
That was our best guess :( Hmm, you're right that the case re := <-resc
has to free the roundtrip method when the readLoop reaches its last select, there must be something else.
The problem is that the situation with stuck goroutines is reproducible only in production. A gateway service (Golang, HTTP client) communicates with a Java service (HTTP server here) using PUT call. Both HA, initially 2->2 pods before scaling. When the gateway is bombed by requests generated by dozens of pods (simulated DDoS - hundreds of requests per second on each of them), the Java service starts to become slow and latencies rise, the number of concurrently processed requests in the gateway goes from ~tens to ~thousands. We let the load test go in such frequency for ~15-30 seconds and then turn it off (to reproduce the issue but being under k8s memory limits - not kill the pods and before the services scale up and spread the load handling). After this load test a metric watching the number of currently processed requests shows values around ~lower hundreds (and never returns to the normal level of ~tens). The same number of stuck goroutines is visible in pprof dump.
So the only information we have is the pprof dump pointing us to the roundtrip method. Still far from writing a test, searching for the exact reason :(
We set ResponseHeaderTimeout, there is deadline on context, we tried it without trace round tripper, disabled keep alive, with different golang versions - 1.21, 1.22 but nothing from this helped to stop the goroutines stuck.
Any ideas on how to proceed further to localize the bug? Thank you!
from go.
Timed out in state WaitingForInfo. Closing.
(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)
from go.
Related Issues (20)
- cmd/go: script_test.go:156: FAIL: testdata/script/mod_install_versioned.txt:10: stdout v2: unexpected success HOT 2
- proposal: net: export listenerBacklog HOT 1
- proposal: net: export supportsIPv4, supportsIPv6, supportsIPv4map HOT 1
- crypto/x509: TestPlatformVerifierLegacy/self-signed_leaf failures HOT 1
- crypto/x509: TestPlatformVerifierLegacy/wrong_host_for_leaf failures HOT 2
- crypto/x509: TestPlatformVerifierLegacy/expired_leaf failures HOT 1
- x/tools/gopls: experiment with an option/capability for server-side file watching HOT 2
- proxy.golang.org: fails to fetch github.com/gmlewis/go-fonts HOT 8
- x/text/collate: CompareString(collate.Numeric) returns wrong result for "0.0" vs "1.0"
- syscall: missing Errno on openbsd/386, openbsd/arm and openbsd/amd64 HOT 1
- x/pkgsite: package update request for [github.com/grafana/k6build] HOT 3
- os/exec: document subtleties around Cmd.Dir, PWD, os.Getwd and C's getcwd HOT 1
- flag.FlagSet.UnixParse: add Unix-style flag parsing (needed for u-root) to Flagset HOT 1
- x/tools/gopls: improvements to loading behavior HOT 1
- x/pkgsite: package removal request for github.com/jeaguil/go-mesencryp HOT 2
- proposal: import/path: I hope there is an official golang GUI library and control library that is actively maintained and supports a similar set of standard controls as Delphi7's. It should also have the ability to support third-party custom control libraries. HOT 1
- cmd/go: adjust `PATH` when `go run`/`go generate` runs the built binary HOT 4
- x/build/cmd/golangbuild: generate instructions for reproducing test results HOT 1
- GOARCH=arm64 go build can't find imports HOT 3
- x/sys/unix: TestIoctlGetEthtoolDrvinfo failures HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go.