Repository for debugging large inserts via diesel
. This setup reads all of the rows into memory, chunks them, then pushes them into postgres
with rayon
, which takes 3-4 minutes.
Another setup tested pushed rows into the array as they were generated, in batches of 10922(Max parameter limit of 65535, divided by 6 parameters per row). This acheived a runtime of around 6 minutes.
To test this project, you should do the following:
- Download the dataset:
curl https://datasets.imdbws.com/title.principals.tsv.gz | gunzip - > data/title.principals.tsv
- Start/initialize the database:
$ docker-compose up -d
# ...
$ diesel setup
- Run the test:
$ cargo run