Init action for Jupyter and JupyterHub about initialization-actions HOT 4 CLOSED

@nehalecky This sounds great! The current Jupyter init script leaves a lot to be desired, and needs to be fixed for each Spark release because it specifies the py4j path.

Thanks for working on this.

For a single jupyter instance, I think it's better to launch jupyter from pyspark (using PYSPARK_PYTHON=python and PYSPARK_DRIVER_PYTHON=jupyter) rather than launching spark from jupyter. It makes things much simpler. Not sure if this still applies with JupyterHub.

from initialization-actions.

@pquentin thanks for the note!

For a single jupyter instance, I think it's better to launch jupyter from pyspark (using PYSPARK_PYTHON=python and PYSPARK_DRIVER_PYTHON=jupyter) rather than launching spark from jupyter. It makes things much simpler.

This sounds like an interesting approach, and I'd be willing to implement, but I am unfamiliar the param. Could you explain more how it simplifies things (I'd like to understand better)?

Not sure if this still applies with JupyterHub.

OK, good point. I think whatever mechanism that we associate Jupyter instance with PySpark should be consistent between Jupyter and JupyterHub, do you agree? If so, how could we find out more about this?

Thanks!

from initialization-actions.

This is how we do it:

export PYSPARK_PYTHON=python
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook --no-browser --port 8123 --ip="*"'

pyspark \
    --conf spark.executorEnv.PYTHONHASHSEED=0 \
    ...

And sure, we need to use the same setup for both Jupyter and JupyterHub.

from initialization-actions.

closed with #31

from initialization-actions.