Empty Stackdriver Traces about k8s-stackdriver HOT 4 CLOSED

@whatupdave Interesting, could you please add what's in the report (removed all sensitive information ofc)? What pod exactly is it coming from?

/cc @igorpeshansky

from k8s-stackdriver.

I'm not sure how to see which pod it's coming from. The gce/vm_ipv4 points to a kubernetes vm with nginx on it and this docker container, which looks like the most likely culprit: https://gist.github.com/whatupdave/16d3650266740f495d984aac41f71ba7

from k8s-stackdriver.

@whatupdave I had the hypothesis, but I confused trace and error reporting. This prometheus-to-sd container actually doesn't have the code to send anything to SD Trace, so it's not likely that it's the offender. I also haven't witnessed anything like this before with the prometheus-to-sd container

You can try to inspect the api manager and correlate the sources and e.g. some logs in the containers on the node. Also, you can try to set up a separate account for each container and check out the source service account for the calls

from k8s-stackdriver.

Ah, thanks for investigating. My new hypothesis is that google cloud load balancer is creating the span but not populating the URL or HTTP method fields.

from k8s-stackdriver.