heraldry-crawler's Introduction

heraldry-crawler

Script to fill French armorial database, linked with heraldry-search-front project.

Features

Scrap data from Wikipedia armorials
Save data in SQLite database
Index data in Algolia
Index data in Elasticsearch

Get started

Create .env file from .env.sample and complete it with the Algolia app settings in your console. If you prefer, an Elasticsearch cluster can be used.

NODE_ENV=
ALGOLIA_INDEXING=true    # should index data in Algolia
ALGOLIA_APPLICATION_ID=  # Aloglia app ID
ALGOLIA_API_KEY=         # Algolia API Key
ALGOLIA_INDEX=           # Algolia index name
ELASTIC_INDEXING=false   # should index data in Elasticsearch
ELASTIC_URL=             # ES instance URL
ORIGIN_URL=              # heraldry-search-front origin url

To run this project locally, install the dependencies and run the script:

npm install
npm run init-dbs  # optional only before first crawl - (re)set database & elastic for emblems
npm run crawl     # crawl Wikipedia emblems and index data
npm start         # start server to search through ES

Alternatively, you may use Yarn:

yarn
yarn init-dbs  # optional only before first crawl - (re)set database & elastic for emblems
yarn crawl     # crawl wikipedia emblems and index data
yarn start     # start server to search through ES

Recommend Projects