Comments (15)
Oh yeah, we're going to be hitting this all the time. Definitely want to see this! 😁
from arrow-datafusion.
Thanks @alamb. I am happy to port ProgressiveEval
from InfluxDB to DB
from arrow-datafusion.
@alamb Just to be absolutely clear, if the plan consists entirely of Parquet files from a single table, then the SortPreservingMerge
will be eliminated. As you can see in this sqllogictest, a single file group is produced if none of the files overlap, and DataFusion correctly understands a SortPreservingMergeExec
is not required. In particular, after #9593, #6672 should be fixed.
That said, it's worth pointing out that ProgressiveEval
is more general (it can work for arbitrary plans e.g. heterogeneous unions).
from arrow-datafusion.
@alamb : I am starting porting
ProgresssiveEval
from IOx. Do you want me to use this ticket or #10433 for that work or you want me to open a new ticket for the porting?
How about we open a new ticketfor porting and link it to this epic?
from arrow-datafusion.
Here is the entire implementation in case anyone wants this: https://github.com/influxdata/influxdb3_core/blob/b546e7f86ee9adbff0dd3c5e687140848397604a/iox_query/src/provider/progressive_eval.rs
from arrow-datafusion.
I probably should have mentioned that in order to use this operator, one needs to be able to determine if the input streams to SortPreservingMerge
are non overlapping in value. Conveniently, that is basically the analysis that @suremarc has implemented in #9593
from arrow-datafusion.
Oh yeah, we're going to be hitting this all the time. Definitely want to see this! 😁
Also to be clear to everyone else -- we have this code in InfluxDB already but it would be great to have other people be able to use it (and help maintain it)
from arrow-datafusion.
FYI @ozankabak @mustafasrepo @matthewmturner
from arrow-datafusion.
I think we should both port ProgressiveEval as well as hook it up in the optimizer so it is is used (likely based on the analysis in #9593)
from arrow-datafusion.
If the hooking work is not that tricky (which I think the case), I am happy to do that, too
from arrow-datafusion.
What comes to my mind is that if we can successfully bring the statistics
to SortPreservingMerge
, it could handle this work without a separate operator (This will probably need a change in statistics
API to return statistics for each partition).
from arrow-datafusion.
What comes to my mind is that if we can successfully bring the
statistics
toSortPreservingMerge
, it could handle this work without a separate operator (This will probably need a change instatistics
API to return statistics for each partition).
I think this approach would work as well -- or maybe just a flag on the SortPreservingMerge
and a different stream rather than an entirely ExecutionPlan
🤔
Per partition statistics is an interesting idea 🤔 -- it certainly makes sense for things like data sources. I wonder how generally useful it could be
from arrow-datafusion.
@alamb : I am starting porting ProgresssiveEval
from IOx. Do you want me to use this ticket or #10433 for that work or you want me to open a new ticket for the porting?
from arrow-datafusion.
This is the ticket to port ProgressiveEval
from InfluxDB #10488
from arrow-datafusion.
Update here:
- @NGA-TRAN has a PR open to port this operator into DataFusion: #10490
- We don't want to merge it into DataFusion until at least one other user could get value from it.
So in my mind, what is required to move on with this ticket is:
- Update one of the existing optimizer passes to detect when the sort keys of a
SortPreservingMerge
don't overlap in value and replace withProgressiveEval
- Tests showing it working
For the "do key ranges overlap" detection code I think we can use what @suremarc added in #9593
from arrow-datafusion.
Related Issues (20)
- UserDefindedLogicalNode::from_template does not return a Result<...>. HOT 3
- Row groups are read out of order or with completely different values HOT 9
- DataFusion weekly project plan (Andrew Lamb) - May 20, 2024 HOT 1
- Advanced example for building an external index for Row Groups *within* parquet files
- DataFusion HashJoin LeftAnti doesn't support null aware anti join
- Incorrect statistics read for `i8` `i16` columns in parquet HOT 3
- DataFusion ignores "column order" parquet statistics specification
- DataFusion reads Date32 and Date64 parquet statistics in as Int32Array HOT 2
- Pass per-field BigQuery `OPTIONS` values to the LogicalPlan's Arrow Schema
- Expand Test Coverage for ScalarUDF's
- Make the configuration for `StreamTable` more generic to support more stream sources
- Support `date_bin` on timestamps with timezone, properly accounting for Daylight Savings Time HOT 26
- Incorrect statistics read for unsigned integer columns in parquet HOT 1
- Incorrect statistics read for binary columns in parquet
- Implement a benchmark for extracting arrow statistics from parquet HOT 1
- Incorrect statistics read for struct array in parquet HOT 3
- PlaceholderRowExec shown when select from union results. HOT 2
- Make it easier to register object stores HOT 2
- MySQL doesn't support the `NULLS FIRST/LAST` clause in `ORDER BY` statements
- Improve performance of extracting statistics from parquet files HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-datafusion.