Want to understand that this spark matrics repo will work with prometheus in Hadoop cluster ? about spark-metrics HOT 4 OPEN

@pankajpkh24 I have the same question, you might look at this issue I just raised #84
I think there are problems with the pull and push based approaches on YARN clusters, see the linked issue

from spark-metrics.

Since Spark supports Prometheus metrics I'd advise to use rather that than this library. This library is recommended to be used for short lived spark jobs which terminate before Prometheus would scrape metrics. Note this library is pushing metrics to Pushgateway thus inherits the limitations of Pushgateway (e.g. metrics in Pushgateway lives forever or until its restarted)

from spark-metrics.

@stoader thanks for your replies. I don't think that the new PrometheusServlet can be used on YARN clusters without additional logic: The original Spark ticket (https://issues.apache.org/jira/browse/SPARK-29032) only mentions K8s and Standalone. I tried to set this up on EMR but the scrape URL on the driver is "dynamic" and contains the application id so I don't think there is an easy way to get this to work. I have a working solution using a different sink and will blog about it soon though. Did you manage to use the PrometheusServlet on YARN clusters yourself?

from spark-metrics.

never tried to use it on YARN only on K8s

from spark-metrics.