specify job name and pushgateway relevant labels via config about spark-metrics HOT 5 CLOSED

Have you considered using metrics namespaces (spark.metrics.namespace)?

If you have a PR please let me know and we'll review it.

from spark-metrics.

I am interested whenever my spark app crash it restart with new job name timestamp appended. I don`t know how i can clean up stale groups from pushgateway.

from spark-metrics.

@dahiyahimanshu pushgateway doesn't support adding a TTL to metrics (see prometheus/pushgateway#117) thus it keeps the received metrics until pushgateway is either restarted or not needed metrics are removed manually. Metrics can be removed either through the UI or using curl (see https://github.com/prometheus/pushgateway for examples)

from spark-metrics.

I think overriding just spark.metrics.namespace (job name) wont help here. I think common groupingKey is also required. However it is not possible to override it now:

      val groupingKey: Map[String, String] = (role, executorId) match {
        case ("driver", _) =>
          labelsMap match {
            case Some(m) => Map("role" -> role, "app_name" -> appName, "instance" -> instance) ++ m
            case _ => Map("role" -> role, "app_name" -> appName, "instance" -> instance)
          }

        case ("executor", Some(id)) =>
          labelsMap match {
            case Some(m) =>
              Map ("role" -> role,
                "number" -> id,
                "app_name" -> appName,
                "instance" -> instance) ++ m
            case _ =>
              Map ("role" -> role,
              "number" -> id,
              "app_name" -> appName,
              "instance" -> instance)
          }

        case _ => Map("role" -> role)
      }

from spark-metrics.

I think this article touch that topic - https://www.robustperception.io/common-pitfalls-when-using-the-pushgateway

This is particularly evident with metrics containing an instance label (which should not be going to the Pushgateway in the first place, as they are not service-level).

from spark-metrics.