Examples of using `TreeNode` APIs to walk and manipulate LogicalPlans about datafusion HOT 6 OPEN

@alamb This is a great idea. As I learn more perhaps I can help.

from datafusion.

This looks very good, pretty much what I implemented. The only question I have remaining is:

If TableScan has filters why would that not catch all filters? What does LogicalPlan::Filter catch that the TableScan.filters would not? For my specific use case I only need filters on the tables that are present in the query.

from datafusion.

Got it, that makes total sense. I will only care about the accepted pushdown filters, so the TableScan will work for what we need.

from datafusion.

Thank you @cisaacson

from datafusion.

Moving out of slack into Github so it might be more easily found

If your usecase is to to get the list of filters and tables that appear in a query, one way to do this is:

• Get the list of tables by finding all LogicalPlan::TableScan (any Join will have an input from a TableScan)
• Get the list of filters/predicates, look in LogicalPlan::Filter and potentially the filters on the LogicalPlan::TableScan

So I think it could look something like (untested)

let mut referenced_tables = HashSet::new();
let mut filters = vec![];

// recursively visit all nodes in the tree (including subqueries)
logical_plan.apply_with_subqueries(|plan| {
  match plan {
   // record table names
    LogicalPlan::TableScan(table_scan) => { 
      referenced_tables.insert(table_scan.table_name);
   },
    // record filters
    LogicalPlan::Filter(filter) => {
      filters.push(filter.predicate);
     }
    // ignore other nodes
    _ => {}
     
      Ok(TreeNodeRecursion::Continue)
  }
 })?;

from datafusion.

If TableScan has filters why would that not catch all filters?

Depending on the value of https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.TableProviderFilterPushDown.html sometimes filters would not be pushed down into the provider

Also filters can be duplicated in the table scan and filter (again depending on the value of the FilterPushDown

from datafusion.