Simple app for scrapping data from gumtree.
The project was created for learning purposes to know how to combine scrapy framework with TorIp changer.
- Docker desktop
.
โโโ docker-compose.yml
โโโ LICENSE
โโโ README.md
โโโ src
โโโ crawler
โ โโโ __init__.py
โ โโโ items.py
โ โโโ middlewares.py
โ โโโ pipelines.py
โ โโโ settings.py
โ โโโ spiders
โ โโโ __init__.py
โ โโโ mieszkania2.py
โ โโโ quotes_spider.py
โโโ Dockerfile
โโโ go_spider.py
โโโ scrapy.cfg
โโโ tests
โโโ ipchanger_works.py
Clone repository:
git clone https://github.com/Santhin/TorScrapy.git
To run the crawler type:
docker-compose up
Simple check if tor ip changer is working unmark commented test in dockerfile.
The exemplary output:
- add control startup for TorIpChanger container in docker-compose
- Scrapy - Crawler
- TorIpChanger - Privoxy + Tor
- Hat tip to DusanMadar for amazing framework and tutorial step by step https://github.com/DusanMadar/TorIpChanger https://gist.github.com/DusanMadar/8d11026b7ce0bce6a67f7dd87b999f6b