There is typos in this line of the README.md file:

Is there a crawler with an web GUI or rather a frontend such as "Octoparse" for Windows oder "Web Alert" for Android?

Hi,
Can we install this crawler using Kali Linux? Also, can we extract the web form field label names from a URL and the sub-links through the main domain, If not, can you suggest one?

Thank you!

Link to markdown:
https://gist.github.com/gustavorps/98a9b03143db24c5acedab343b7e7a55

Preview

A program to download photo and video from any site (e.g. Reddit, Twitter, Instagram).

https://github.com/AAndyProgram/SCrawler

https://github.com/weaming/picpicgo

Add Internet Archive Brozzler distributed crawler in Python

e.g. for R, HTML parsers:

• XML - XML and HTML scraper and parser. Also works via FTP.
• xml2 - R wrapper for the libxml2 C library.

e.g. for R, ghost browsers:

• RSelenium - R bindings for the Selenium 2.0 WebDriver.
• I think RStudio is working on something similar, but cannot find it any more

Could you add Add Bose Framework to Python Scrapers. It is a web scraping framework.

@BruceDone checkout #62 please, added reference to https://dev.to/spacenomad/mercator-url-frontier-in-golang-15o2

Your repo is so cool, so I want to contribute my crawler repo to your awesome list.

My repo is here : https://github.com/khanhicetea/distributed-webcrawler
It's built on-top Docker, Python, Redis and Gearman Job Server with a distributed architecture :)

Hope it will be helpful :)

rt.

Rust based crawler or indexer https://github.com/spider-rs/spider. Also the fastest and most efficient OSS crawler.

Hi, great repo!
So I'd like to add this website https://www.free-proxies.info
It's very helpful for crawling because it's grant an API for get free or premium proxy!

There are also some filters like transparent proxy or anonymous; google proxy and more..
It's always need a proxy for crawling the web!

Photon is an incredibly fast crawler which extracts URLs, website accounts, emails, files, endpoints and much more.

https://github.com/s0md3v/Photon

Here is my crawler made in Rust.
Supports seed database, adds external links to database, simple parsing and adding to elastic search.

https://github.com/tclaus/web_crawler

Jvppeteer is a Java library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Jvppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium
https://github.com/fanyong920/jvppeteer

Add https://github.com/LAW-Unimi/BUbiNG - The LAW next generation crawler.