Random segfaults (pointers to objects become invalid?) about javacall.jl HOT 8 CLOSED

Maybe we need to create GlobalRef for all objects? I didn't think we needed to... but maybe we do??

I'll try to replicate. mvn package will get me everything? I don't need to start a Spark process or anything?

from javacall.jl.

Good that you asked - I updated the branch so that now mvn package downloads all dependencies including Spark itself and packages them into a single "sparta-assembly.jar". This jar is then included in a classpath (see "init.jl"), and Spark context is created from Julia itself (see sc = SparkContext() above). So yes, now you only need to run mvn package to prepare all Java code.

from javacall.jl.

Regarding global references, yes, it looks pretty likely to be the cause. Would returning global refs from jcall and removing them in deleteref enough in this case?

from javacall.jl.

Yes, that is what I was thinking. Convert object references to global refs
in convert_result and remove them in deleteref. Can you try that and
see if that helps?

On Sun, Aug 9, 2015 at 12:18 PM, Andrei Zhabinski [email protected]
wrote:

Regarding global references, yes, it looks pretty likely to be the cause.
Would returning global refs from jcall and removing them in deleteref
enough in this case?

—
Reply to this email directly or view it on GitHub
#20 (comment).

from javacall.jl.

Just tried it in global-refs branch, but with no success. I wrapped all convert_result methods except for those for strings and arrays, so custom objects (like SparkContext or JavaRDD) should have been affected. Yet, I get same NullPointerException and segmentation fault.

from javacall.jl.

:( The changes in the branch look right...

Exception in thread "main" java.lang.NullPointerException at org.apache.spark.api.java.JavaRDD$.toRDD(JavaRDD.scala:197)

What is the object that is null in JavaRDD.scala:197? is it sc or java_rdd?

from javacall.jl.

It's java_rdd, or, more precisely, java_rdd.jrdd, because java_rdd is a Julia wrapper for a pointer to JavaRDD object in JVM. In source code referred method looks like this:

implicit def toRDD[T](rdd: JavaRDD[T]): RDD[T] = rdd.rdd

So basically it simply tries to get .rdd field of JavaRDD object.

Anyway, I believe it's not essential here - I've just got segfault on PipelineRDD creation and I think I had NullPointerException when creating SparkContext.

from javacall.jl.

fixed by #28

from javacall.jl.