mhenes / pythoncrawler Goto Github PK
View Code? Open in Web Editor NEWThis project forked from anoopbr/pythoncrawler
Spyder Crawler using python using multi threading. It fetches top 10 results from Google/Bing search engine for any user specified keyword. User can choose the number of results required and number of thread. Crawler also checks for robot.txt files and avoids restricted urls.