Not able to launch a spark operator using standard docker about spark-operator HOT 11 CLOSED

BTW: I actually plan to add a command-line tool, say, sparkctl that supports staging local dependencies to, for example, the resource staging server.

from spark-operator.

It needs to run on a cluster with alpha features enabled to be able to use the initializer. See https://kubernetes.io/docs/admin/extensible-admission-controllers/#enable-initializers-alpha-feature.

from spark-operator.

Li, appreciate the quick response. Any other way we can run it if alpha feature is not turned on ?

from spark-operator.

I can add a flag to allow disabling the initializer so you can run on a non-alpha cluster.

from spark-operator.

perfect thanks. Appreciate it.

from spark-operator.

Ref: #28. enable-initializer is true by default, you can set it to false in spark-operator.yaml.

from spark-operator.

Thanks for the quick turn-around.

One more question, how can we run an application when the application jars are not packaged into the driver and/or executor.

eg : with the spark-k8s project I can specify,
init-container:
image: "path to my init container image"

In submission.go I can see a note to the effect that "// Note that when the controller submits the application, it expects that all dependencies are local"

What is the recommended approach here? eg : resource staging server, persistent volume claim with jars on it.(If yes then, how would this be mounted at runtime?)

Appreciate the help!

from spark-operator.

What we often do is to stage jars/files to, e.g., a GCS bucket, (optionally) make them public, and list the https urls in the deps.jars and deps.files sections. If you don't want to make the dependencies on GCS public, you will need a custom init-container image for downloading from GCS. See https://gist.github.com/liyinan926/f9e81f7b54d94c05171a663345eb58bf for an example init-container image.

from spark-operator.

Ref: #15 tracks work for supporting non-container-local dependencies.

from spark-operator.

@devtagare Can this be closed now?

from spark-operator.

Yes. Thanks for the quick turnaround.

from spark-operator.