siddharthaanand / ncbi-sars-cov2-data-crawler Goto Github PK

Modify the code to do incremental crawling based on date. This would improve the crawling code and make it easier to collect new changes on the website.

Try out an alternative way to scrape the data:
Use requests or an alternative to simulate the requests/or check if it's impossible to do it.

It would be helpful since once the requests are found to be independant of each other, scrapy can be used here to decrease the crawl time.

Improve code to use headless browser to avoid opening of the browsers on the screen.