Dockerfile based on the official Spark 2.3.0 image with spark-metrics and its dependencies about spark-metrics HOT 3 CLOSED

Hi @liyinan926 , great idea. Will add one to the repository.

from spark-metrics.

Hi @liyinan926, thinking on it more there is no need to create a new Dockerfile. You can use the official Spark 2.3.0 Dockerfile.

All you need is to copy the following jars:

• com.banzaicloud:spark-metrics_2.11:2.3-1.0.0

• io.prometheus:simpleclient:0.3.0

• io.prometheus:simpleclient_common:0.3.0

• io.prometheus:simpleclient_dropwizard:0.3.0

• io.prometheus:simpleclient_pushgateway:0.3.0

• io.dropwizard.metrics:metrics-core:3.1.2

  to assembly/target/scala-$SPARK_SCALA_VERSION/jars prior building the Spark 2.3 docker images. This way you'll get spark-metrics jars included into the built image at /opt/spark/jarspath inside the image which is part of the class path of the driver and executor.

Please also read the notes on Spark 2.3 here: https://github.com/banzaicloud/spark-metrics/blob/master/PrometheusSink.md

from spark-metrics.

The reason why I think providing a reference Docker image makes sense is it avoid the need for users to create a custom Spark image simply to add the jars. It's not hard but it's better if it can be completed avoided.

from spark-metrics.