Spark APIs are not working in Elyra about enterprise_gateway HOT 4 CLOSED

I have a possible workaround to force sparkContext creation with proper configuration from inside Toree just when in Yarn cluster mode. Will cleanup code and provide a Toree PR tomorrow.

from enterprise_gateway.

Met a similar issue with the Toree Launcher when running Toree Magic with Spark SQL from tutorial:

Name: org.apache.spark.SparkException
Message: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: 
Lost task 0.3 in stage 1.0 (TID 7, nier4.fyre.ibm.com, executor 1): 
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy 
to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ 
of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD

from enterprise_gateway.

Jars deployed in our internal repositories. Waiting for PR to be merged on Toree master.

from enterprise_gateway.

Tried the same codes inside Spark-shell on my HDP cluster (spark version 2.1.1.2.6.1.0-129), everything is fine.

Don't know if this is relevant, but someone already asked this on JIRA two years ago. And one possible reason might be the default Spark version toree needs for build is 2.0.0 but on our cluster (at least in my case) the spark version is 2.1.1.

Link of comment and JIRA: https://issues.apache.org/jira/browse/SPARK-9219?focusedCommentId=14635088&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14635088

from enterprise_gateway.