DataFusion weekly project plan (Andrew Lamb) - April 1, 2024 about arrow-datafusion HOT 14 CLOSED

DataFusion 37.0.0 is released: #9682 🌮

from arrow-datafusion.

Really interested to see your thoughts on custom indices, its something were currently working on to improve performance of listing table.

from arrow-datafusion.

aliases for character_length are defined in https://github.com/apache/arrow-datafusion/blob/4bd7c137e0e205140e273a7c25824c94b457c660/datafusion/functions/src/unicode/character_length.rs#L45 and is used during udf registration @ https://github.com/apache/arrow-datafusion/blob/4bd7c137e0e205140e273a7c25824c94b457c660/datafusion/core/src/execution/context/mod.rs#L2137

from arrow-datafusion.

BTW something cool for next week -- @milenkovicm WASM UDFs #9326 / #9326 (comment) / https://github.com/milenkovicm/wasaffi

from arrow-datafusion.

Really interested to see your thoughts on custom indices, its something were currently working on to improve performance of listing table.

In case anyone else is following along, here are some items related to indexing:

• #9963
• #9929

While I had this on the brain, @matthewmturner I also filed #9964 as I suspect others will be interested in helping make ListingTable faster too

from arrow-datafusion.

Review queue

Arrow

• apache/arrow-rs#5525
• apache/arrow-rs#5557 (review)
• apache/arrow-rs#5578
• apache/arrow-rs#5575 (review)
• apache/arrow-rs#5566
• apache/arrow-rs#5554

DataFusion

• #9874
• #9903

from arrow-datafusion.

Arrow:

• apache/arrow-rs#5557
• apache/arrow-rs#5576

DataFusion

• #9824
• #9876
• #9913
• #9906
• #9912
• #9813
• #9732
• #9655

from arrow-datafusion.

Sorry for the location and possibly dumb question, but shouldn't all functions be exported here: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/unicode/mod.rs#L138

I am probably using the library incorrectly but (for example) LENGTH will not be registered by the central context here: https://github.com/apache/arrow-datafusion/blob/2f550032140d42d1ee6d8ed86f7790766fa7302e/datafusion/core/src/execution/context/mod.rs#L1450

from arrow-datafusion.

Hi @seddonm1 👋

Sorry for the location and possibly dumb question, but shouldn't all functions be exported here: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/unicode/mod.rs#L138

Yes that is the intention

I am probably using the library incorrectly but (for example) LENGTH will not be registered by the central context here:

https://github.com/apache/arrow-datafusion/blob/2f550032140d42d1ee6d8ed86f7790766fa7302e/datafusion/core/src/execution/context/mod.rs#L1450

I think length is registered as an alias of character_length

from arrow-datafusion.

Thanks @alamb . If you look at the first link I cannot see anywhere where the length alias is actually registered. My queries which previously worked started failing after this update. I haven't investigated if more functions have also not been registered.

from arrow-datafusion.

Arrow:

• apache/arrow-rs#5557

DataFusion

• #9926 (review)
• #9934
• #9938
• #9936
• #9933
• #9947
• #9944
• #9942
• #9813
• #9732
• #9655

from arrow-datafusion.

Thanks @Omega359 . I was thrown as the aliases are not registered here https://github.com/apache/arrow-datafusion/blob/3ae029988754c3fd3eb000abd4b76e643b9cbc7b/datafusion/execution/src/registry.rs#L174

from arrow-datafusion.

Reviews:
@jayzhan211 is starting to pull aggregate functions out of the core: #9960

from arrow-datafusion.

Next week #10002

from arrow-datafusion.