Using Deequ with zeppelin about deequ HOT 4 CLOSED

Gautam, thanks a lot for bringing this to our attention and apologies for the delay.

First of all, Deequ supports both Spark 2.4 as well as Scala 2.11.12.

However, I was able to reproduce the issue you just described. While I don't have a complete explanation for the observed behavior, it seems to come down to the way Zeppelin is handling external dependencies. I reproduced the error by adding Deequ as a dependency via the Zeppelin UI by adding the artifact com.amazon.deequ:deequ:1.0.1 as dependency in the UI via Settings - Interpreter - spark - edit. Subsequent import of com.amazon.deequ.VerificationSuite (or for what it's worth any class, for example also org.apache.spark.sql.DataFrame) leads to error message you posted previously.

The positive news is that I was able to run Deequ in an identical setting (EMR 5.23.0, Spark 2.4, Zeppelin 0.8.1, Scala 2.11.12) by declaring the dependency on Deequ within the notebook itself in code. To do this, start with a fresh Zeppelin instance (i.e. without adding the dependency via the UI), and in a new notebook execute the following code before anything else:

%dep
z.load("com.amazon.deequ:deequ:1.0.1")

followed by your code using Deequ, e.g.

case class Item(...)

val rdd = spark.sparkContext.parallelize(...)

val data = spark.createDataFrame(rdd)

import com.amazon.deequ.VerificationSuite
import com.amazon.deequ.checks.{Check, CheckLevel}

val verificationResult = VerificationSuite().onData(data).addCheck(....).run()

This gives you a working environment for Deequ. As this approach works, I suspect the original issue to be an issue with Zeppelin. Since we have a workaround using the %dep directive, I will close this issue. Please reopen it in case you encounter problems or the workaround does not work for you.

from deequ.

Hi Gautam, this looks more like an error related to Scala. Which Scala version are you using?

from deequ.

I'm using scala 2.11.12

from deequ.

It seems this is indeed a problem with Spark 2.4. Thanks for pointing this out, we will look into the problem.

from deequ.