A research cloud OLAP query engine.
[1] Yifei Yang, Matt Youill, Matthew Woicik, Yizhou Liu, Xiangyao Yu, Marco Serafini, Ashraf Aboulnaga, Michael Stonebraker, FlexPushdownDB: Hybrid Pushdown and Caching in a Cloud DBMS, VLDB 2021.
[2] Yifei Yang, Hangdong Zhao, Xiangyao Yu, Paraschos Koutris, Predicate Transfer: Efficient Pre-Filtering on Multi-Join Queries, CIDR 2024.
git clone https://github.com/cloud-olap/FlexPushdownDB.git
cd FlexPushdownDB
Compiler needed:
- Linux: LLVM-12 or later versions.
- Mac OS: LLVM-13.
- Install required dependency. For Ubuntu,
./tools/project/bin/ubuntu-prerequisites.sh
. For other Linux OS or Mac OS, required dependencies listed intools/project/bin/ubuntu-prerequisites.sh
have to be manually installed. - Set
RESOURCE_PATH
offpdb-calcite/java/main/resources/config/exec.conf
to the path ofresources
. - Build the system
./resources/script/build.sh
.
- Create a cluster of EC2 nodes, within which one is the coordinator, the others are executors. The coordinator is not necessarily as powerful as the executor.
- Log in the coordinator. Put public IP of all nodes (including the coordinator) into
resources/config/cluster_ips
. Each line is one IP. - Set up the system
./resources/script/setup.sh
. If all required dependencies are already installed, the system can also be set up by./resources/script/build.sh
,./resources/script/deploy.sh
.
install_dependency
whether to install required dependencies when runningresources/script/setup.sh
.clean
whether to clean before the build starts.build_parallel
parallel degree used for CMake build.build_dir_name
name of the directory for the built files.deploy_dir_name
name of the directory used for deploying to and running on a cluster.temp_dir_name
name of the directory to store temp files after the system starts.pem_path
the pem file (AWS private key) used to log in created EC2 nodes.
- Set up AWS access credentials and config. Run
aws configure
or manually set~/.aws/credentials
and~/.aws/config
. - Organize data files as follows on one S3 bucket:
- Under
resources/metadata/
, create a directory of which the absolute path denotes the schema name, e.g. data files unders3://tpch-sf0.01/csv/
belong to schematpch-sf0.01/csv/
. - For each table
T
with a single partition, name the file asT.tbl
for CSV format orT.parquet
for Parquet format; for each tableT
with multiple partitions, create a directoryT_sharded/
and put partition files named asT.tbl.0/T.parquet.0
,T.tbl.1/T.parquet.1
... into it.
- Under
- Register data by creating metadata files under
resources/metadata/
. Create a directory named as the schema name, and organize metadata files as follows:schema.json
For each table, specifyname
,format
(delimiter
if CSV),numPartitions
,fields
.stats.json
For each table, specifyname
,rowCount
,apxColumnLength
(no need to be accurate, just an estimation, used by W-LFU).zoneMap.json
For each table, specifyname
, collected min_max values of each partition on some columns, leave it empty if not collected.
- Start the Calcite server
java -jar fpdb-calcite/java/target/flexpushdowndb.thrift.calcite-1.0-SNAPSHOT.jar &
. cd <build directory>/fpdb-main
.- Run tests
./fpdb-main-test -ts=<test-suite> -tc=<test-case>
, available single-node test suites aressb-sf1-single_node-no-parallel
,tpch-sf0.01-single_node-no-parallel
,tpch-sf0.01-single_node-parallel
. - When finished, stop the Calcite server.
- Make sure the security group of your nodes allows inbound and outbound traffic for TCP protocol at
CAF_SERVER_PORT
(4242 by default, can be changed by instructions below). cd ~/FPDB-build/
.- Start the system
./resources/script/start.sh
. cd fpdb-main
.- Run tests
./fpdb-main-test -ts=<test-suite> -tc=<test-case>
, available test suites aressb-sf1-single_node-no-parallel
,tpch-sf0.01-single_node-no-parallel
,tpch-sf0.01-single_node-parallel
,tpch-sf0.01-distributed
. - When finished, stop the system
~/FPDB-build/resources/script/stop.sh
.
- On large tests, check "ulimit -n", make sure the output number is high enough, otherwise it may throw "Too many open files" exception.
S3_BUCKET
s3 bucket name where the data is in.SCHEMA_NAME
the name of the schema to query.CACHE_SIZE
size of local cache.MODE
execution mode, can be one ofPULLUP
,PUSHDOWN_ONLY
,CACHING_ONLY
,HYBRID
. Currently SSB tests support all modes, TPC-H tests only supportPULLUP
andCACHING_ONLY
.CACHING_POLICY
cache replacement policy, can be one ofLRU
,LFU
,LFU-S
(size normalized LFU),W-LFU
(Weighted LFU).PARALLEL_DEGREE
execution parallel degree of the same kind of operator (e.g. join), independent of data parallel degree which corresponds to the number of partitions of a table.SHOW_OP_TIMES
whether to show execution times of each operator and each kind of operator.SHOW_SCAN_METRICS
whether to show scan metrics like data load speed, selectivity, data conversion speed...CAF_SERVER_PORT
port used by CAF for cross-node communications.IS_DISTRIBUTED
whether the system runs across a cluster or just locally.
S3_CLIENT_TYPE
type of S3 client, can be one ofS3
,AIRMETTLE
,MINIO
. ForAIRMETTLE
andMINIO
, access key and endpoint need to be set infpdb-aws/src/AWSClient.cpp
.NETWORK_LIMIT
used to throttle data loading from S3, set to 0 if not throttling.
SERVER_PORT
port for the Calcite server.JAR_NAME
built jar name of the Calcite server.
SERVER_PORT
port for the Calcite server, should be kept same as cpp side.RESOURCE_PATH
absolute path ofresources/
used for metadata fetching.exec.conf.ec2
is a fixed config for EC2 deployment and does not need to be changed.