- Prepare venv
virtualenv -p python3 .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
- Run tests
- Run crawler locally
python crawler.py --urls https://hands.ru https://repetitors.info
- Get results in
result.json
file in your local directory
- Download and install docker
- Build image:
docker build -t crawler .
- Run container and pass urls for crawling
docker run --rm -it -v ${PWD}:/code crawler --urls https://hands.ru https://repetitors.info
- Get results in
result.json
file in your local directory