JMeter Master Stops Responding about jmeter-operator HOT 8 CLOSED

Noticed that it looks like all of the jmeter-slave pods also have 1 restart now logged on them. Perhaps it's those pods that are crashing....

from jmeter-operator.

Watching the logs on one of the slave pods
It seems to run fine, and then:
error: unexpected EOF
OR
killed

Pod then shows another restart

from jmeter-operator.

Looks like the influxdb is getting overwhelmed...seeing a lot of 204's in the logs, and then 500 timeouts after a while. Which is odd since kubernetes never reports it running out of memory or CPU. It must be network bandwidth.

from jmeter-operator.

You wrote: "and they usually run in 10 pods"
do you mean 10 slaves? which are running on 10 different nodes?
I'm running for smoke tests 3 slave pods with 10 threads and 10.000 loops "forever" and can reach 30.000 messages / second which writes 8 MB/s in Kafka running on different nodes where the slaves are running, and it works.

from jmeter-operator.

The 10 slave pods run on 5 nodes. These 5 nodes also run everything else including the kube-system pods and the grafana, influx, and jmeter-master pods. I'm trying to find some sort of performance tuning for influxdb, but everything I'm seeing says it should be able to handle 5-10k requests per second.

from jmeter-operator.

I'd suggest to use anti-affinity to run the slaves not on the same nodes where your app is running!

from jmeter-operator.

Sorry if I wasn't clear.
Only the JMeter deployment and required kubernetes pods are running on these nodes. The actual web application I am testing is far removed from this kubernetes cluster.

from jmeter-operator.

OK,
so adding wal-fsync-delay = "100ms" to the configmap seems to have solved the problem.
I had to edit the main.yml in my local \roles\jmeter\tasks and push that to the jmeter-operator pod running ansible at /opt/ansible/roles/jmeter/tasks/

from jmeter-operator.