-
Clone this repo.
git clone https://github.com/NazikM/dataOX_test_task
-
Add your proxy to a list. In scraper/proxy/list.txt
Format for proxy list: http://username:pass@ip:port
The AUTOTHROTTLE options in settings.py are set for 100 proxies. If you'd like to use less or more you should change these params.
Note: You can get 10 free proxy with good speed on webshare.io or buy 100 proxies for 1$
- Run your Docker Desktop.
- Run git compose file. This will everything you need inside your docker.
docker-compose up