To install the python virtual environment follow the following instructions below.
Then to activate it so that any new modules that are installed are installed into this virtual environment:
source venv/bin/activate
(Linux/Mac OS)
./venv/bin/active
(Windows)
To install the required modules for this python project to run you need to install the required python modules using the following command:
pip install -r requirements.txt
Once the required python modules are installed you should be able to view/run the Python Scrapy Spider with the following command (from within the project folder):
Cd into the project spiders: cd lawscraper
View the project spiders: scrapy list
Run the project spider: scrapy crawl lawspider
If you have issues running the pip install -r requirements.txt
command this can be due to some things not being up to date on your computer.
Running the following may solve some of these issues:
pip install --upgrade pip
The following error: NotADirectoryError: [Errno 20] Not a directory: 'pkg-config'
might be solvable by running:
export PKG_CONFIG=/path/to/pkg-config