Hi @Vadim-Maletskii I found the command rcurl::getURL() thanks to which I can download the page, but I have not found the arguments in the function which could help us reveal our identity to be polite. For example, in httr::GET() we can write our email and user-agent, but in getURL() I have not found such an opportunity. Could you please check how we can do this?

The code now keeps the links to different parts of the page itself (e.g. jpg, pdf files, categories). I will try to fix it. Look it up in pull requests when it's ready

1. Download the page using rcurl: https://beppegrillo.it/un-mare-di-plastica-ci-sommergera/
2. Create a data frame with the links on the page
3. Keep the links that lead to other posts of the blog
4. Create the code for pages scraping: https://beppegrillo.it/category/archivio/2016/
5. Get links for each page and put them into a list
6. Download the page for each page (remember to Sys.sleep())
7. Scrape the text for each page
8. Sketch and explain the spider scraper