The conglomeration of podcast crawlers.

Crawls podcast index from itunes using Scrapy and saves the results into MongoDB.

To start the crawler, run

scrapy crawl podcasts

To get the itunes podcast category list:

scrapy crawl categories

For each podcast crawled in podcrawler, show_crawler looks up its feed url and gets additional show information from its feed. To start the crawler, run

python show_crawler/show_crawler_main.py feed

To update feed url, then run

python show_crawler/show_crawler_main.py metadata

To update show metadata.

Pull episode information from individual podcast's feed url.

To start the crawler, run

python episode_crawler/episode_crawler_main.py

At project root level, use the following scripts to start the crawlers.

scripts/run_show_crawler.sh

scripts/run_episode_crawler.sh