[QUESTION] spark-submit not called, and other questions about spark-operator HOT 6 CLOSED

@RyanZotti glad it worked!

Yes, it seems we can provide two images here, one for running the pod that runs the operator itself, that has different service account and permissions too. Then we can have another image for the SparkApplication which is responsible for spawning driver and executor and uses a different service account with possibly different permissions. During helm install we specify the operator pod's image, but when we write the SparkApplication manifest we can supply a different image. This is quite handy as often we'd need to modify the Driver / Executor images with additional layers for various connectors, such as described in the GCP guide for example.

from spark-operator.

I was able to run your examples on my cluster (not minikube) using image: "spark:3.5.0" for the SparkApplication

from spark-operator.

@mereck Yes, that worked for me too. Thanks.

I'm new to Kubernetes and operators in general. Am I correct to assume that the driver and executors can use an image separate from the image of the operator? I assume that's the case, since that would explain why there are two places to specify images, one in the spark application yaml and another in the helm install command, but that wasn't clear to me from the docs.

from spark-operator.

@mereck That makes sense. As a quick follow-up, inspired by the external jars example you linked to, do you know if I'm doing anything wrong while adding third party jars?

For example, assume a driver/executor Dockerfile like so:

FROM spark:3.5.0

ADD https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/8.3.0/mysql-connector-j-8.3.0.jar /opt/spark/work-dir/

Where I've added the following line under spec to my original spark-py-pi.yaml:

 deps:
  jars:
    - local:///opt/spark/work-dir/mysql-connector-j-8.3.0.jar

Built the image like so:

eval $(minikube docker-env)
docker build -t spark-debug:latest -f Dockerfile

and where I've updated spark-py-pi.yaml to point to the spark-debug:latest image accordingly.

When I do all that I get a message like this:

Files local:///opt/spark/work-dir/mysql-connector-j-8.3.0.jar from /opt/spark/work-dir/mysql-connector-j-8.3.0.jar to /opt/spark/work-dir/mysql-connector-j-8.3.0.jar
Exception in thread "main" java.nio.file.NoSuchFileException: /opt/spark/work-dir/mysql-connector-j-8.3.0.jar

I can see the file at the specified location when I log into an interactive container. Do the jars need to be "local" to the Spark Operator image? I don't think that makes sense, but want to confirm.

from spark-operator.

It sounds like that jar would only need to be on the SparkApplication image. Have you tried adding the chmod 644 command to Dockerfile as well? Also I think it's better to place jars in the $SPARK_HOME/jars

Here's an example.

If permissions don't help, I would try spinning up a container with bash and explore the file system to check if the file you've added is there:

docker run --rm -it --entrypoint bash <image-name-or-id>

from spark-operator.

@mereck Thanks! That did the trick, so I'm marking this issue as closed.

from spark-operator.