This repo consists of the data used for acl 2017 Submission

Learning Character-level Compositionality with Visual Features

This is an implementation for crawling the title and its corresponding categories of an Wikipedia page.

The crawler will crawl the categories according to category_list_lang.txt

The dataset is already crawled in the folder acl2017_data

with the following command:

python crawl.py -l zh -n 100000

python crawl.py -l ja -n 100000

python crawl.py -l ko -n 100000

Running the crawler again will cover the data in the folder.

python crawl.py [-h] -l LANG -n NUM

optional arguments:

-h, --help show this help message and exit

-l LANG, --lang LANG which language to crawl [zh, ja, ko]

-n NUM, --num NUM the minimun number of pages you wish to crawl for each category (may not be able to reach this number)