The project helps to parse the largest Kazakhstani marketplace: products and reviews using site API.
https://github.com/zhanymkanov/russian_reviews_dataset
- Partially cleaned (both raw and cleaned versions are available though)
- ~150k rows
- Crawl products JSON lists
- Crawl reviews based on products list
- Crawl laptops, PCs, smartphones specs based on products list
I had to do some stuff with my outgoing traffic to find out its endpoints.
Therefore, I think it is not tethical to put it online, but to say that is easy to get them
- Python 3.6+
- pip
- scrapy
- pydantic
- python-decouple
- orjson
- psycopg2-binary
- anosql
- Download the project
git clone https://github.com/zhanymkanov/reviews_parser
- Install the packages
pip install -r requirements/base.txt
To run the app you need to know API endpoints, which is currently not available to be shared, but discovered by yourselves.
If you have them, modify the main.py so that you parse only required categories.
For the first time, I recommend you to run the main.py
3 times:
- First, run for products
- Second and third times for either reviews or product specifications