Support yarn-cluster mode about spark-jobserver HOT 9 CLOSED

For issue #2), it's pretty easy to solve, and I may have found someone willing to contribute a PR.

For #1), there is a separate thread about this. We can basically keep fixing issues in Spark or fork off separate processes. I'm leaning towards the latter, but not sure how well this would work with YARN, so the feedback from YARN folks would be great.

-Evan
"Never doubt that a small group of thoughtful, committed citizens can change the world" - M. Mead

On Mar 9, 2015, at 10:31 AM, Peter Rudenko [email protected] wrote:

Hi there's some issues regarding concurrency and multitenancy with jobserver & spark:

1. The issue that impossible to create multiple spark context inside one JVM (SPARK-2243)
2. There's no way to kill particular job when submitting several jobs within single spark context.

For yarn mode it's possible to submit job remotely from code (http://blog.sequenceiq.com/blog/2014/08/22/spark-submit-in-java/) - and kill job through YARN API. How difficult is to implement such functionality inside spark-jobserver?

—
Reply to this email directly or view it on GitHub.

from spark-jobserver.

Hm not sure issue 2 easy to solve when it runs on yarn. For each context it creates a yarn application. If you kill the whole app it will kill all the jobs from this context. For a standalone cluster it's also not clear how to kill jobs from a single context:

val nums = 1 to 1000
val jobs = nums.par.foreach{num => sc.parallelize(1 to num).count()}

And imagine you want to kill even jobs. Not sure how easily to do that.

from spark-jobserver.

Ah get it, it's possible to set jobgroup for each request and kill the whole group through sc.cancelJobGroup

from spark-jobserver.

Yes, that's right. Going to merge this issue with other similar ones.

On Mon, Mar 9, 2015 at 2:25 PM, Peter Rudenko [email protected]
wrote:

Ah get it, it's possible to set jobgroup for each request and kill the
whole group through sc.cancelJobGroup

—
Reply to this email directly or view it on GitHub
#83 (comment)
.

The fruit of silence is prayer;
the fruit of prayer is faith;
the fruit of faith is love;
the fruit of love is service;
the fruit of service is peace. -- Mother Teresa

from spark-jobserver.

Started to prototype cancel job functionality. Have some problems to run tests, but seems to work on manually testing. Please take a look whether i'm on a right direction. Will make PR when everything would work.

from spark-jobserver.

Hi Peter,

The changes look good to me! Thanks for updating the flow diagram as
well. Only thing is that the final cancelJobGroup on line 255 of
JobManagerActor, is that really necessary?

I'm not sure how to help you avoid the test issue. I think it's most
likely networking related..... :( I can do some research to see if we
could improve how Akka handles networking.

-Evan

On Thu, Mar 12, 2015 at 5:12 AM, Peter Rudenko [email protected]
wrote:

Started to prototype cancel job functionality
petro-rudenko@1303c83.
Have some problems to run tests, but seems to work on manually testing.
Please take a look whether i'm on a right direction. Will make PR when
everything would work.

—
Reply to this email directly or view it on GitHub
#83 (comment)
.

The fruit of silence is prayer;
the fruit of prayer is faith;
the fruit of faith is love;
the fruit of love is service;
the fruit of service is peace. -- Mother Teresa

from spark-jobserver.

I'm able to run tests now (run them on VM). Have a problem with this, not very familiar with spray yet:

it("should be able to kill job from /jobs/<id> route") {
      Delete("/jobs/foo") ~> sealRoute(routes) ~> check {
        status should be (OK)
        responseAs[Map[String, String]] should be (Map(
          StatusKey -> "KILLED"
        ))
      }
    }

[info] - should be able to kill job from /jobs/<id> route *** FAILED ***
[info]   Map("status" -> "OK", "result" -> "foo!!!") was not equal to Map("status" -> "KILLED") (WebApiSpec.scala:242)

from spark-jobserver.

@petro-rudenko have you modified WebApiSpec.scala? I mean the part where the response from the actors is stubbed out at the top. That's probably where the foo! is coming from.... sorry I didn't find hte link to the PR.

from spark-jobserver.

Fixed now PR #96. Take a look.

from spark-jobserver.