-
关于脚本:路径为src/run/new_version_multi_doc.py
- 训练模式:在仓库目录下使用命令PYTHONPATH='src' python3 src/run/new_version_multi_doc.py config/ms_tpch_manydocs_masked.ini > train.log
- 测试模式:在仓库目录下使用命令PYTHONPATH='src' python3 src/run/new_version_multi_doc.py config/ms_tpch_manydocs.ini > test.log
- 使用脚本前,请确保mysqld未在运行
-
关于ini文件:
-
一些共有参数:
- device:cpu或cuda,建议在允许的情况下改成cuda加快模型更新速度
- nr_episodes:即iteration数
- timeout_s:运行时间限制
- mode:masked表示训练,unmasked表示测试
- recovery_cmd:请忽略这个参数,这是dbbert自带的用于作者pg数据库的重启命令
- workload_time:每个sysbench workload运行的时间
-
ms_tpch_manydocs_masked.ini:用于训练
- input:待训练的bert模型的路径(均为huggingface官网上下载的模型,我存在本地,如果网好,也可以直接连接到huggingface在线获取模型)
- output:训练完成后输出的bert模型路径,文件夹里面包括一个config。json和pytorch_model.bin模型
-
ms_tpch_manydocs.ini:用于测试 model_path:用于测试的bert模型的路径,可以与上面的output相同以使用刚才训练好的模型
-
DB-BERT is a database tuning tools that exploits natural language text as additional input. It extracts recommendations for database parameter settings from tuning-related text via natural language analysis. It optimizes parameter settings for a given workload and performance metric using reinforcement learning.
- DB-BERT was tested on an Amazon EC2 p3.2xlarge instance using Deep Learning AMI (Ubuntu 18.04) Version 43.0.
- Install Postgres 13.2 and the TPC-H benchmark database.
- Install required Python packages, including Huggingface Transformers, stable-baselines-3, psycopg2, Numpy, Pytorch, and streamlit.
- DB-BERT uses configuration files, an example file can be found under
config/pg_tpch_manydocs.ini
. - For a first try, update credentials in the
[DBMS]
section. You may adapt tuning time and other parameters in the[BENCHMARK]
section. - Execute
src/run/tuning_run.py
, passing the configuration file as first (and only) argument, e.g. (from main directory):PYTHONPATH='src' python3 src/run/tuning_run.py config/pg_tpch_manydocs.ini
- DB-BERT will parse input text (text from 100 Web documents), initialize the RL algorithm, and start tuning Postgres for TPC-H.
- Try
config/pg_tpch_onedoc.ini
for tuning using a single, TPC-H specific text document (forum discussion on TPC-H tuning).
- To start the GUI, run
streamlit run src/run/interface.py
from the root directory. - If accessing DB-BERT on a remote EC2 server, make sure to enable inbound traffic to port 8501.
- Enter the URL shown in the console into your Web browser to access the interface.
- You can select settings to read from configuration files in the
demo_configs
folder. - Select a collection of tuning text documents for extraction (e.g., from the
demo_docs
folder). - You may change parameter related to database access, learning, and tuning goals.
- Click on the
Start Tuning
button to start the tuning process.
- Switching language models used from BERT to BART (despite the project name).
- Refined text document pre-processing: extract most relevant passage for each parameter.
A video talk introducing the vision behind this project is available online.
@inproceedings{trummer2022dbbert,
title={DB-BERT: a database tuning tool that "reads the manual"},
author={Trummer, Immanuel},
booktitle={SIGMOD},
year={2022}
}
@article{trummer2021case,
title={The case for NLP-enhanced database tuning: towards tuning tools that" read the manual"},
author={Trummer, Immanuel},
journal={Proceedings of the VLDB Endowment},
volume={14},
number={7},
pages={1159--1165},
year={2021},
publisher={VLDB Endowment}
}