Output is sometimes suppressed about spylon-kernel HOT 8 CLOSED

Setting the Console out and err at the start of each cell execution leads to all sorts of problems when using anonymous functions with executors. The history in #31 should show many of the things I tried without success: marking everything transient, wrapping the stream setting in a object, marking the object properties transient, letting the object be serialized but error'ing quietly on executors, ...

Taking a step back, the correct way to solve this problem is still what I noted above: setting the console stdout/stderr before / around creation of the SparkContext so that all threads inherit the streams. The only reason I avoided that approach to start is because it's going to be difficult (and hopefully not impossible) to get initializing ordering right using spylon, pyspark, and py4j.

What we currently do:

1. Initialize pyspark.SparkContext which creates a py4j gateway for us
2. Initialize the Scala interpreter using the gateway
3. Set the Console / System streams to what we want (which doesn't work for all threads)

What we need to do:

1. Launch the py4j gateway ourselves using pyspark.java_gateway.launch_gateway
2. Initialize the Scala interpreter using the gateway
3. Set the Console / System streams to what we want
4. Initialize pyspark.SparkContext with the gateway instance

from spylon-kernel.

Saw this yesterday in a different case. Turned out it was a spark config setting that actually disabled output for explain. In your case, if you run the cell again, does it work the second time? Or is it that particular call always gives nothing.

from spylon-kernel.

No, in my case it refuses to return no matter the number of retries. It seems to be more about the command refusing to return output than the cell. That is, if I change the command within the same cell it will return output, but if I try the refused command in a new cell, it still refuses.

from spylon-kernel.

That's what i saw yesterday with explain but it was due to how Spark was configured.

from spylon-kernel.

After talking with @patrick-nicholson, his case is likely another form of #21. Leaving this open for the time being.

from spylon-kernel.

Got a report of at least one instance where the output started going into the notebook log instead of into the notebook, but then restored itself back to going into the notebook a few cells / calls later.

from spylon-kernel.

I can reproduce the above behavior by running cells quickly in succession. Not sure yet if it's coming from the scala or python process when it happens.

from spylon-kernel.

Problem is in scala. Console.setOut/setErr is thread specific, allows mutation (it's a setter!), and, for these reasons, deprecated in Scala 2.12. Tried using System.set but to no avail. Fix that works is to wrap all code to be interpreted in a Console.withOut/withErr(fileHandle) { } block. Will work this change in soon and see how it behaves with some interesting cases after some other code cleanup.

from spylon-kernel.