This repository contains datasets and source code for the paper "Boosting exact pattern matching with eXtreme Gradient Boosting (and more)".
The experiment consists of two parts.
-
The first part consists of measuring the times of exact pattern matching algorithms to create a dataset for machine learning.
- ๐
smart
- contains the source code for exact pattern matching, which is mostly taken from SMART tool (Nov 7 2018, 3b112e362b6ff9e3336b406cc8618164da91ef23). - ๐
data
- contains datasets and patterns excluding datasets that must be downloaded from Pizza&Chilli corpus (the tests were performed on 50MB texts). - ๐
run_tests.py
- executes exact pattern matching algorithms and saves the results todataset-raw.pickle
- ๐
utils.py
- contains helper functions - ๐
dataset-raw.pickle
- contains raw datasets with pattern searching times for all algorithms
- ๐
-
The second part is to prepare the dataset for machine learning and train machine learning models whose task is to predict the fastest algorithm for a given pattern.
- ๐
start_pred.py
- runs the tests (trainings and predictions) - ๐
datautils.py
- contains helper functions to process the data - ๐
dataset-full.pickle
- processed dataset, prepared for machine learning - ๐
dataset-full-notebook.ipynb
- jupyter notebook with results of exact pattern matching algorithms - ๐
train_datasets.pickle
- train dataset - ๐
test_datasets.pickle
- test dataset - ๐
trained_models.pickle
- contains trained models (make sure to have the same version of scikit-learn to load them, otherwise it will not be possible) - ๐
results-notebook.ipynb
- jupyter notebook with results of machine learning models
- ๐
Note: *.pickle
files are not included in the repository. They can be downloaded from:
Note 2: The text corpus is also not included in the repository. It can be downloaded from Pizza&Chilli (the tests were performed on 50MB texts) and then placed in the data
directory.
- Python 3.10.10 (other versions may work, but they were not tested)
- All required packages are listed in
requirements.txt
file.
-
Clone the repository
git clone https://github.com/rsusik/boosting-exact-pattern-matching.git
-
Install required packages by running
pip install -r requirements.txt
-
Download the
*.pickle
files from the links above and place them in the root directory of the repository. -
Run
start_pred.py
script.
The results may differ in prediction times slightly due to the hardware differences, but the impact on the overall results should be negligible.
Perform step 1 and 2 from the previous section and then:
-
Download the text corpus from Pizza&Chilli (the tests were performed on 50MB texts) and place it in the
data
directory. -
Execute searching algorithms to create the dataset:
cd smart python compile.py cd .. python run_tests.py
Running the searching algorithms may take a long time and lead to slightly different results due to the hardware differences.
- Robert Susik
- Szymon Grabowski