Comments (5)
take
from arrow-datafusion.
...while looking into this I noticed, that there are no statistics written for an Interval
, which is also described here.
@alamb I guess we can't extract any statistics here? And writing any tests that check we have no statistics written, does not seem to be very helpful?
from arrow-datafusion.
@alamb I guess we can't extract any statistics here? And writing any tests that check we have no statistics written, does not seem to be very helpful?
I actually think these would be helpful then as soon as there are statistics we can hook them up to the tests. If you had time to write the tests that would be great. We can then perhaps file a ticket in parquet-rs for supporting writing statistics to interval types.
from arrow-datafusion.
sure I can do that; from the top of my mind - the fn run()
from the struct Test
panics if we can't extract any statistics, which is the case here. So I'd prepare as much as possible (creating record batches, adding a Scenario, writing those tests) but for now would assert should panic
- does this make any sense to you @alamb?
from arrow-datafusion.
I did some digging in order to find out why / or where the writing of those statistics is not supported (yet).
Since I'm not familiar with the parquet impl, here are my findings, which might be useful in a follow-up ticket in arrow-rs.
- When trying to
fn write_slice()
the min, max values are never updated due to a filter-condition; that checks if the type is INTERVAL - In order to support updating the min max values, we need to handle the comparison of INTERVAL here
I think this should be possible, or put differently, I don't see the reason yet, why this is not supported?
Somethin similar (comparing FixedLenByteArrays) is already done for DECIMAL here?
Perhaps, you have some more information on this @alamb - otherwise this might be enough information to file a ticket in arrow-rs?
from arrow-datafusion.
Related Issues (20)
- Building time for `cargo bench` takes quite a long time HOT 1
- Potential optimization for CASE WHEN for protecting against divide by zero HOT 1
- Add a sub-project for map udf functions HOT 5
- always failed test on datasource::file_format::csv::tests::test_csv_parallel_one_col::case_6 on windows machine
- Update Optimizer documentation HOT 1
- Implement `evaluate_bounds` for math unary functions HOT 2
- Why is there no content in `Extending DataFusion’s operators: custom LogicalPlan and Execution Plan` HOT 4
- Add NullState::is_null public method
- Typo in doc of datafusion::physical_plan::Partitioning HOT 2
- Expression Simplifier doesn't consider associativity (`(i + 1) + 2)` is not simplified to `i + 3`) HOT 5
- Serialization of UDF might lose aliases HOT 1
- Propagation of ordered `SortProperties` should consider `nulls_first`
- Rename `functions-array` to `functions-nested` to collect all nested-type functions HOT 2
- Consider having `SessionCatalog` also implement `FunctionRegistry` rather than providing access to the hash maps directly
- Review code that `downcast_ref` from `CatalogSession` to `SessionState`
- DataFusion weekly project plan (Andrew Lamb) - July 22, 2024 HOT 1
- Write a blog post about implementing StringView in DataFusion
- Optimize CASE expression to produce dictionary-encoded arrays in some cases HOT 1
- Decimal256 type is not supported in datafusion-proto
- Ability to chunk download from object store HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-datafusion.