Use `min_value` and `max_value` on statistics to avoid `ExecutionPlan.execute` about arrow-datafusion HOT 11 OPEN

For anyone else looking for this, the solution is to implement supports_filters_pushdown on the table, then use that to apply filters within scan.

Feel free to close this if you like.

from arrow-datafusion.

Relabed from bug to feature -- thanks @samuelcolvin

from arrow-datafusion.

I think using the reported statistics to prune using datafusion::physical_optimizer::pruning::PruningPredicate would be a nice improvement (so each table provider didn't have to apply the basic min/max filters themselves)

from arrow-datafusion.

@alamb using PruningPredicate makes sense, but please can you point me at where I need to make changes to add this functionality?

from arrow-datafusion.

@alamb using PruningPredicate makes sense, but please can you point me at where I need to make changes to add this functionality?

I recommend updating the existing though poorly named AggregateStatistics pass

And there you could potentially call ExecutionPlan::statistics() and use PruningPredicate to prove that some predicates can't be true and replace them with PlaceholderRowExec

from arrow-datafusion.

(Thank you for working on this, bTW)

from arrow-datafusion.

@samuelcolvin have you already started to write code?

from arrow-datafusion.

No code yet, if you'd like to work on this, feel free.

from arrow-datafusion.

Actually I'm on a flight today, so might have some time to work on this.

from arrow-datafusion.

Progress update, I've got min & max stats pruning "working" in our code, however I immediately ran into #10536, I'll let you know how I get on.

from arrow-datafusion.

You know it occurs to me that @dmitrybugakov / @jayzhan211 may be working on a similar feature with a different approach on #10456 🤔 I

from arrow-datafusion.