Use eqNullSafe instead of collect about chispa HOT 2 OPEN

certainly this will be so much clearer and main maintenance will be on PySpark itself.

from chispa.

@rragundez - thanks for creating this issue.

I could see how eqNullSafe could be useful, especially for large column comparison operations. You could do something like df.withColumn("are_cols_equal", col1.eqNullSafe(col2)) and then run a filtering operation and make sure are_cols_equal is always equal to true. I did something similar to this in spark-fast-tests but don't really use this implementation of the method. Should do some more benchmarking and see if this is faster.

Is this what you're suggesting?

from chispa.