Comments (6)
@alamb This is a great idea. As I learn more perhaps I can help.
from arrow-datafusion.
This looks very good, pretty much what I implemented. The only question I have remaining is:
If TableScan
has filters
why would that not catch all filters? What does LogicalPlan::Filter
catch that the TableScan.filters
would not? For my specific use case I only need filters on the tables that are present in the query.
from arrow-datafusion.
Got it, that makes total sense. I will only care about the accepted pushdown filters, so the TableScan
will work for what we need.
from arrow-datafusion.
Thank you @cisaacson
from arrow-datafusion.
Moving out of slack into Github so it might be more easily found
If your usecase is to to get the list of filters and tables that appear in a query, one way to do this is:
- Get the list of tables by finding all
LogicalPlan::TableScan
(any Join will have an input from a TableScan) - Get the list of filters/predicates, look in
LogicalPlan::Filter
and potentially the filters on theLogicalPlan::TableScan
So I think it could look something like (untested)
let mut referenced_tables = HashSet::new();
let mut filters = vec![];
// recursively visit all nodes in the tree (including subqueries)
logical_plan.apply_with_subqueries(|plan| {
match plan {
// record table names
LogicalPlan::TableScan(table_scan) => {
referenced_tables.insert(table_scan.table_name);
},
// record filters
LogicalPlan::Filter(filter) => {
filters.push(filter.predicate);
}
// ignore other nodes
_ => {}
Ok(TreeNodeRecursion::Continue)
}
})?;
from arrow-datafusion.
If TableScan has filters why would that not catch all filters?
Depending on the value of https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.TableProviderFilterPushDown.html sometimes filters would not be pushed down into the provider
Also filters can be duplicated in the table scan and filter (again depending on the value of the FilterPushDown
from arrow-datafusion.
Related Issues (20)
- Minimize the dependency on `SessionState` HOT 2
- Handle Serde for Custom AggregateUDFImpl traits
- [EPIC] A collection of issues for supporting the `MAP` DataType HOT 2
- Preceding and Following (WindowFrameBound) are incorrectly handled when an unoptimized plan created via SQL is converted to a substrait Plan
- Add nullable in `StateFieldArgs` HOT 11
- Implement the rewrite from the Map literal to Map function HOT 2
- Document the Map function in the documentation
- Support Arrays for the Map scalar functions HOT 2
- Enforce the uniqueness of map key name for the map/make_map function HOT 1
- The struct value should not have duplicate and null name HOT 1
- Support `to_json` function on `struct` type HOT 1
- 2024 Q3-Q4 Roadmap? HOT 19
- Add some structure to capture result of write orchestration function
- Implement nested identifier access ( "Nested identifiers not yet supported" )
- Add `coop_budget` to `datafusion_common::config::ExecutionOptions` HOT 5
- Improve memory pool reservation `shrink` error handling
- Return scalar result when all inputs are constant for `map` and `make_map` udfs HOT 1
- Impl hash for ScalarValue::Map HOT 1
- `DFSchema::check_names` allows ambiguous references. HOT 1
- Feat: Support `GROUP BY unnest expr`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-datafusion.