Unable to send data to elasticsearch due to error - 'net/http: request canceled' about prometheusbeat HOT 7 CLOSED

Are you able to connect to Elasticsearch in general from within the prometheusbeat container (e.g. issuing curl -XGET http://<ip>:9200/_cluster/health?pretty)?

from prometheusbeat.

The prometheus beat is running as process.. ES is up and other beats are able to send data from same node

{
"cluster_name" : "my-application",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 192,
"active_shards" : 192,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 191,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.13054830287206
}

from prometheusbeat.

Maybe ES is overloaded, you can check the threadpools: curl -XGET localhost:9200/_cat/thread_pool?v
The write threadpool is the one who is used when prometheusbeat sends the metrics via the _bulk API.

Any suspicious logs in ES itself? Garbage collection logs?

from prometheusbeat.

Is there no prometheusbeat data at all in ES? curl -XGET localhost:9200/_cat/indices | grep prometheusbeat

from prometheusbeat.

I am to able to push data in same elasticsearch instance with same configs of prometheusbeat from another setup.
The is the only difference between prometheus instances of the 2 setups is that there is pushgateway enabled in case of non-working setup.

Can that cause issue?

from prometheusbeat.

I am not observation any suspicious logs on ES side

from prometheusbeat.

I am to able to push data in same elasticsearch instance with same configs of prometheusbeat from another setup.
The is the only difference between prometheus instances of the 2 setups is that there is pushgateway enabled in case of non-working setup.

Can that cause issue?

I'm not that familiar with the pushgateway but the prometheusbeat logs rather indicate that it cannot connect to ES and send data. So it looks like that the issue is on ES side or network between prometheusbeat and ES.

from prometheusbeat.