Spark-compatible CAST operation about arrow-datafusion HOT 4 OPEN

cc @viirya and @andygrove and @Omega359

You could always create a user defined function that does the cast 🤔 bit I am not sure about how much in DataFusion special cases the CAST

from arrow-datafusion.

I'm surprised cast is the thing that is catching you .. for me it was the differences in functions and their behaviour between the two systems. I wouldn't have a clue atm how to make cast in datafusion replaceable - I suspect it wouldn't be an easy task.

from arrow-datafusion.

There's definitely difference in some functions as well, but those have been mostly minor so far (I haven't gotten to regexps or date/time stuff yet), like returning null vs throwing on overflow.

I did confirm today I can use Comet's scalar functions by overriding the existing ScalarUDFs in DF, that's quite nice!

For Expr::Cast that doesn't work as it's part of the Expr enum rather than being a ScalarUDF - is that by choice, or just pending migration? But I plan to try if I can use FunctionRewrite to change the Expr::Cast into a ScalarUDF that uses the Comet cast.

from arrow-datafusion.

For Expr::Cast that doesn't work as it's part of the Expr enum rather than being a ScalarUDF - is that by choice, or just pending migration? But I plan to try if I can use FunctionRewrite to change the Expr::Cast into a ScalarUDF that uses the Comet cast.

I think it is largely historical, but realistically there is enough code that handles Cast specially in the core that switching it to use a UDF is likely a susbtantial project (we could certainly do it, but I wanted to offer my opinion on the scope)

from arrow-datafusion.