zambujo / editorials Goto Github PK

• improve logic: move editorials.R to data-raw/
• rename editorials.R to harvest_all.R
• create harvest_new.R
• create prepare_data.R to tidy, combine, and export datasets to data/

use polite methods

Ideas

• nejm notable articles of the year
• Scientific American Advances

GS list of top venues

• get funding information from crossref
• from Dimensions - free version (complement Crossref)

the following articles were found are missing:
https://www.nature.com/articles/474008a
https://www.nature.com/articles/474008b
https://www.nature.com/articles/474008c
https://www.nature.com/articles/474008d
https://www.nature.com/articles/474008e
https://www.nature.com/articles/474009a
https://www.nature.com/articles/474009b
https://www.nature.com/articles/474009c
https://www.nature.com/articles/474009d
https://www.nature.com/articles/474009e

cannot find the following articles listed in the table of contents:
https://www.nature.com/articles/d41586-018-00477-9
https://www.nature.com/articles/d41586-018-00494-8
https://www.nature.com/articles/d41586-018-02738-z
https://www.nature.com/articles/d41586-018-03094-8
https://www.nature.com/articles/d41586-018-03123-6
https://www.nature.com/articles/d41586-018-03124-5
https://www.nature.com/articles/d41586-018-03135-2
https://www.nature.com/articles/d41586-018-03136-1
https://www.nature.com/articles/d41586-018-03179-4
https://www.nature.com/articles/d41586-018-03300-7
https://www.nature.com/articles/d41586-018-03422-y
however, they appear in:
https://www.nature.com/nature/articles?type=research-highlight&year=2018

see: https://anystyle.io/
anystyle parse citations.txt > citations.json

see robots.txt

Research Highlights support ended in 2017: needs 3 sets of parsing rules: 2004-2007 issues (all in one page), 2008-2017 issues, and 2018-present issues.

no research highlights between vol. 544, 7648 and vol. 556, 7699!

ie.

10.1038/457360c
10.1038/457360e
10.1038/457361b
10.1038/457360a
10.1038/457360d
10.1038/457361a
10.1038/457361d
10.1038/458682f
10.1038/458682c
10.1038/458682e
10.1038/458683d

• re-scrape nature's data

see:
https://science.sciencemag.org/content/350/6265/1218.2
https://science.sciencemag.org/content/353/6302/883.4

https://github.com/CrossRef/rest-api-doc#etiquette

• rethink parsing logic of get_sciencemag_articles()
  filtering paragraphs by word count rule fails at times

• fix the regex before unnest() for data-raw/sicence_highlights.csv

  • allow reference to end with ; (see https://science.sciencemag.org/content/sci/333/6044/804.1)

• get data from explicit DOIs
• resolve shortened DOI links

scholar fallback?