Download papers pdfs and other information from main AI conferences (when public available) and store it on ./data/, creating one directory per conference, and one per year. More specifically, it creates the following structure:

.
└── data
    └── conf
        └── year
            ├── abstracts.csv       # format `title|abstract`
            ├── authors.csv         # format `title;authors`
            ├── paper_info.csv      # format `title;abstract_url;pdf_url`
            └── papers
                ├── paper01.pdf     # pdf file from a paper
                ├── paper02.pdf
                ├── ...
                └── paperN.pdf

Based on CVPR_paper_search_tool by Jin Yamanaka. I decided to split the code into multiple projects:

• this project - Download papers pdfs and other information from main AI conferences
• AI Papers Cleaner - Extract text from papers PDFs and abstracts, and remove uninformative words
• AI Papers Search Tool - Automatic paper clustering
• AI Papers Searcher - Web app to search papers by keywords or similar papers

Currently supports the following conferences, from 2017 and on:

Docker or, for local installation:

• Python 3.10+
• Poetry

To make it easier to run the code, with or without Docker, I created a few helpers. Both ways use start_here.sh as an entry point. Since there are a few quirks when calling the scrappers, I created this file with all the necessary commands to run the code. All you need to do is to uncomment the relevant lines inside the conferences array and run the script.

You first need to install Python Poetry. Then, you can install the dependencies and run the code:

poetry install
bash start_here.sh

To help with the Docker setup, I created a Dockerfile and a Makefile. The Dockerfile contains all the instructions to create the Docker image. The Makefile contains the commands to build the image, run the container, and run the code inside the container. To build the image, simply run:

make

To call start_here.sh inside the container, run:

make run

To run the interactive scrapy shell inside a Docker container, run:

make RUN_STRING="scrapy shell 'https://your.site.com'" run