gurpreet0610 / deploy-prometheus-grafana-on-kubernetes Goto Github PK

preface: im new to Kubernetes, Prometheus, and Grafana... be nice, I'm doing this to learn all the things... lol.

I have setup everything here, on K3s, (had to roll my own NFS storage class, but i jumped that hurdle), storage is working fine. Networking seems to be OK.

I log into my nodeportIP:32000, works, everything seems to check out. I see my Prometheus data source, great. I can even run the save/test option on the prom data source, it checks out. so great... moving on.

I go to import a new dashboard (using 8588 as everyone else uses it) and... bad gateway (see below screenshot). even if i copy/paste the json for the new dashboard, when it comes up finally, nothing populates and i just see bad gateway on all the dashboard widgets.
Not sure what the issue here is. My gut tells me its networking flowing from within the cluster (10.43.x.x subnet) out to my main network (192.16.1.x subnet) but I'm unsure where to look to prove this.

any help is appreciated (imma go dive down the traefik rabbit hole and learn how the internal networking works).