A panic occurred when used HTTP probe about cloudprober HOT 8 CLOSED

Fixed by: e086f8e.

from cloudprober.

@manugarg Sorry for the late reply.
Here is my sample configuration:

probe {
        name: "api-staging"
        type: HTTP
        targets {
                host_names: "host1,host2,host3"
        }

        interval_msec: 5000             # 5s
        timeout_msec: 2000              # 2s

        latency_unit: "ms"
        latency_distribution {
                explicit_buckets: "2,4,8,16,32,64,128,256,512"
        }

        http_probe {
                relative_url: "/health"
                port: 8080
        }
}

probe {
        name: "api-release"
        type: HTTP
        targets {
                host_names: "host1-prod,host2-prod,host3-prod"
        }

        interval_msec: 5000             # 5s
        timeout_msec: 2000              # 2s

        latency_unit: "ms"
        latency_distribution {
                explicit_buckets: "2,4,8,16,32,64,128,256,512"
        }

        http_probe {
                relative_url: "/health"
                port: 8080
        }
}
...

BTW, I found the cause.
Please see this code: https://github.com/google/cloudprober/blob/master/surfacers/prometheus/prometheus.go#L144-L159

If you look at the code, http.HandleFunc (ps.c.GetMetricsUrl (), ..) can be run multiple at the same time, and done will be passed to all currently running http.HandleFunc (ps.c.GetMetricsUrl (), ..). So maybe http.HandleFunc (ps.c.GetMetricsUrl (), ..) running ps.writeData(w) can receive done and be interrupted.
This causes the Writer to be reset internally and eventually panic occurs.

I have created PR(#229) to fix this so please take a look.
Thank you!

from cloudprober.

@mazing80 That's a great find. Excellent work :)

This bug is quite hard to reproduce because even if serving multiple requests in parallel, it depends a bit on chance -- which handler will read from the "done channel" first. Logic is clearly flawed though. I'll review your PR. Thank you!

from cloudprober.

@mazing80 Thanks a lot of reporting the issue.

This panic is very strange though. goroutine that you pasted above is running the prometheus surfacer and is in bufio.Writer.Flush(). I can't imagine a reason for a panic in that function as the data is already prepared by the time that function is called (unless there is a bug in net/http or bufio -- extremely unlikely). I think there must be other goroutines in the stack trace. Can you please paste them too. Panic may occurring in some other goroutine.

Thanks once again.

from cloudprober.

@manugarg Thank you for the quick reply!
Sadly, the stack trace is that's all. I also think it is a strange issue.
Anyway, I also will debug it.

Thank you!

from cloudprober.

@mazing80 If you could share your config (privately if you'd prefer that), I can keep a cloudprober running with that config, or even look at the code for venues for possible panics. Also, if you have any other info, for example if adding of a specific probe or surfacer causes it, that will also help.

from cloudprober.

Prefect! I also encountered the same problem. I will verify if it can be solved effectively.

from cloudprober.

@MrDragon1122 Thanks for reporting. Please do let us know what you find :)

from cloudprober.