Job Server going Out of Memory about spark-jobserver HOT 7 CLOSED

Hi @saif-17 ,

Spark 3.0.2 is not supported by Jobserver yet. There is an unreviewed PR with this feature #1359, which you may try to use. But so far the project supports only Spark 2.4.

Best,
Valentina

from spark-jobserver.

Hi @vglagoleva

Thanks for your reply, everything else is working fine with the Job Server with Spark 3. Only in the case of a little bigger dataset, it's causing out of memory. Any idea why that could be happening?

from spark-jobserver.

Hi @saif-17,
No, I guess then you just need to tune your Spark job and the memory settings for your context.

from spark-jobserver.

@vglagoleva - can you please update when is it planned to support Spark 3?

@saif-17 - did you manage to use it with spark3?

from spark-jobserver.

@yaelda
Yes, I'm able to use it with Spark 3.0.3 with scala 2.12.0 it's quite stable.

from spark-jobserver.

Thanks @saif-17 . I would really appreciate your help with this. Can you please share the Spark-jobserver version you used for it? Did you have to change something to support spark 3?

from spark-jobserver.

@yaelda
I'm using Spark-Jobserver version 0.11. I did not change anything to support Spark 3.

from spark-jobserver.