ivn-svn / articles_api Goto Github PK

Python application for web crawling of dynamic web page, which generates content by doing asynchronous Javascript calls after page is loaded. The Scrapy spider implements inside a Selenuim WebDriver to handle the asynchronous JS calls and handles:

• collecting of NBS articles on multiple pages;
• validations of collected data;
• saving collected and valid articles to sqlite database;
• with FastAPI framework are build endpoints of collected data.

back to top

• Scrapy Framework
• JSON Schema
• sqlite3
• sqlachemy
• Selenium WebDriver
• Fast API

1. Clone the repo

   https://github.com/TanyaAng/Articles_API.git

2. Install all Python libraries

   pip install -r requirements.txt

3. Make nbs_articles root directory

back to top

1. Run Spider from terminal:

• (venv) ..\nbs_articles> scrapy crawl article

2. Run FastApi:

back to top

back to top

MIT License

back to top

Tanya Angelova - LinkedIn - [email protected]

Project Link: github link

back to top