Remove the hard dependency on the pyspark about chispa HOT 4 CLOSED

@alexott - thanks for reporting this & sorry for the delayed response.

I wasn't able to replicate your issue (was able to successfully attach chispa v0.6.0 to a Databricks cluster via PyPi & manually attaching the wheel), but I'm a Python n00b and I'm sure your point is valid.

In the Scala world, we add the Spark dependency like libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.4" % "provided" and the "provided" part is what makes Spark a "soft" dependency.

I updated PySpark to be a "dev-dependency" rather than a regular dependency:

[tool.poetry.dependencies]
python = ">2.7"

[tool.poetry.dev-dependencies]
pytest = "3.2.2"
pytest_describe = "^1.0.0"
pyspark = ">2.0.0"

I published chispa v0.7.0. Can you please try it out and let me know if it fixes your issue? Thanks!

from chispa.

Thank you very much Matthew! I'll check on Monday, when I get to my work laptop...

from chispa.

Thank you! I just checked, it works just fine now

from chispa.

@alexott - thanks for confirming!

If you ever have any additional recommendations for this library, just let me know, thanks!

from chispa.