A programming challenge. Features:

• Scrape websites for text and return the most frequently used words
• Cache results per website to not scrape unnecessarily
• Scrape several websites in parallel with goroutines

wordscrape [optional flags] [URLs to scrape]...
  -b    Add invalid and 404 URLs to the scrape list
  -e int
        How many pages of quotes.toscrape.com examples (max 10) to add to the scrape list
  -s    Do not read cache, always scrape sites
  -t int
        How many of the most frequently used words to display (default 5)
  -w int
        How many goroutines to use for scraping. Default is one for every URL (default -1)

Cache is written to a word-cache folder in the directory the program was ran in.

Replace go run . with the exec name (wordscrape by default) if you prefer to compile the program manually (go build).

Scrape 10 example pages without reading cache:

go run . -e 10 -s

Scrape 10 example pages with cache (if cache is available):

go run . -e 10

Scrape 2 example pages and include invalid URLs in scrape list

go run . -e 2 -b

Scrape 2 example pages and a list of custom pages:

go run . -e 2 https://go.dev/blog/loopvar-preview https://go.dev/blog/wasi