Metrics name pre-processing by custom Prometheus sink is working for only one component(driver/executor/applicationMaster) about spark-metrics HOT 10 CLOSED

@stoader I used new regex, but this issue is right. It only replaces driver metrics it doesn't considers others.

*.sink.prometheus.metrics-name-capture-regex=(.*(mgck_|application_))([0-9_]*)(.+)
*.sink.prometheus.metrics-name-replacement=$4

from spark-metrics.

Can you show examples of the original driver and executor metrics without name pre-processing enabled that are provided by Spark in your environment?

from spark-metrics.

@stoader Facing this same issue, I am running latest master on spark3 emr and using

*.sink.prometheus.metrics-name-capture-regex=(.+driver_)(.+)
*.sink.prometheus.metrics-name-replacement=$2
*.sink.prometheus.group-key=ns=stream-ns

and following in spark defaults.

spark.metrics.namespace          ${spark.app.name}

Is there any way to get rid application name & id from all metric names. They are in labels ( via job, number, role ) already.

Screenshot of application master group

Screenshot of driver group ( here the replacement has worked )

Screenshot of executor 2 ( Here instead of applicationId, job is coming in metric name)

Screenshot of executor 1

from spark-metrics.

from spark-metrics.

@t0il3ts0ap your config is incorrect:

*.sink.prometheus.metrics-name-capture-regex=(.+driver_)(.+)

This regex will replace only metric names that includes driver in the name.

In order to have the applicationMaster replaced you need something like:

*.sink.prometheus.metrics-name-capture-regex=(.+applicationMaster_)(.+)

from spark-metrics.

@stoader As you can see executor metrics are starting with appName delta_streamer_request_details_1.
Do you know how I can replace those as well ?

It will be difficult to provide dynamic metrics.properties based on application name.

from spark-metrics.

I'd suggest to check the executor metric names pushed by the executor first without any regex and than create the appropriate regex for extracting only the needed part from the name

from spark-metrics.

I'm closing this issue as it's not a spark-metrics issue but rather using the appropriate regex in the configuration.

from spark-metrics.

@stoader I used new regex, but this issue is right. It only replaces driver metrics it doesn't considers others.

*.sink.prometheus.metrics-name-capture-regex=(.*(mgck_|application_))([0-9_]*)(.+)
*.sink.prometheus.metrics-name-replacement=$4

yeah,I think you are right.It's so inconvenient

from spark-metrics.

@Kwafoor
It doesn't works for executor if you get the jar from s3.
Instead first store the jar in local filesystem ( ex-> /usr/lib/spark) in each node of your cluster ( In emr, can do via bootstrap action ), then it will work for both driver/executor.

from spark-metrics.