Crawl data from domain.com.au:
Install:
- python3 -m venv .venv
- pip3 install -r requirement.txt
Usage: run: scrapy crawl domainau -o output.csv -s DEPTH_LIMIT=1
Advanced:
https://scrapeops.io/python-scrapy-playbook/scrapy-403-unhandled-forbidden-error/