Comments (1)
Here is a potential strawman implementation that I think would be fairly straightforward to implement:
- Use criterion and add a new
bench
benchmark (takes care of the warmup, measurements, etc) - Use the tpch / clickbench dataset (created via
./bench.sh data clickbench
, etc) - 2-3 queries (e.g. a TOP 10 query with predicates, and an aggregate with low cardinality)
- Use default target-partitions / threadpool (e.g. num cores on machine)
- The benchmark is how long it takes to complete some fixed number queries (e.g. 100)
Then we could run
# Run q1 100 times, start the next query as soon as the first is complete
concurrency_q1_concurrency1
# Run q1 100 times, running 2 queries at any time
concurrency_q1_concurrency2
# Run q1 100 times, running 4 queries at any time
concurrency_q1_concurrency4
ANd the same for one or two other queries
from arrow-datafusion.
Related Issues (20)
- Building time for `cargo bench` takes quite a long time HOT 1
- Potential optimization for CASE WHEN for protecting against divide by zero HOT 1
- Add a sub-project for map udf functions HOT 5
- always failed test on datasource::file_format::csv::tests::test_csv_parallel_one_col::case_6 on windows machine
- Update Optimizer documentation HOT 1
- Implement `evaluate_bounds` for math unary functions HOT 2
- Why is there no content in `Extending DataFusion’s operators: custom LogicalPlan and Execution Plan` HOT 4
- Add NullState::is_null public method
- Typo in doc of datafusion::physical_plan::Partitioning HOT 2
- Expression Simplifier doesn't consider associativity (`(i + 1) + 2)` is not simplified to `i + 3`) HOT 5
- Serialization of UDF might lose aliases HOT 1
- Propagation of ordered `SortProperties` should consider `nulls_first`
- Rename `functions-array` to `functions-nested` to collect all nested-type functions HOT 2
- Consider having `SessionCatalog` also implement `FunctionRegistry` rather than providing access to the hash maps directly
- Review code that `downcast_ref` from `CatalogSession` to `SessionState`
- DataFusion weekly project plan (Andrew Lamb) - July 22, 2024 HOT 1
- Write a blog post about implementing StringView in DataFusion
- Optimize CASE expression to produce dictionary-encoded arrays in some cases HOT 1
- Decimal256 type is not supported in datafusion-proto
- Ability to chunk download from object store HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-datafusion.