Python kernel is not initializing all PySpark modules (e.g. PySpark ML) about enterprise_gateway HOT 5 CLOSED

@lresende -- let me take a look at this. When I created the launcher script I tried to keep the initialization code to a minimum. There may be more we want to do here.

from enterprise_gateway.

took a quick look at the SystemML code which is (re-)raising the ImportError:

https://github.com/apache/systemml/blob/master/src/main/python/systemml/mlcontext.py#L31

try:
    import py4j.java_gateway
    from py4j.java_gateway import JavaObject
    from pyspark import SparkContext
    import pyspark.mllib.common
except ImportError:
    raise ImportError('Unable to import `pyspark`. Hint: Make sure you are running with PySpark.')

@lresende -- is that the code you ran?

the above imports work fine in my environment (without Anaconda)

from enterprise_gateway.

I am running this:
https://github.com/apache/systemml/blob/master/samples/jupyter-notebooks/SystemML-PySpark-Recommendation-Demo.ipynb

With the addition of a initial line to install SystemML:
!pip install systemml

And i only have anaconda on master, not on nodes where the iPython is installed

from enterprise_gateway.

@lresende -- can you still reproduce this issue?

from enterprise_gateway.

Properly updated Python and R kernelspec with required environment variables

from enterprise_gateway.