Given a URL, it prints a simple site map, showing the links between pages.
- It's limited to one domain
- crawls recursively
- don't follow external links
$ go get github.com/rpip/crawl
$ crawl --help
Usage of crawl:
-d int
Depth of lookup within page (default 4)
-u string
URL to start crawl on
-v Verbose mode (default true)
$ crawl -u=https://golang.org -d=2 -v=false > sitemap.txt
$ make docker
$ docker run -i -t mz/crawl