Via https://ropensci.r-universe.dev/builds

We see

library("awardFindR")
nsf_and_nih <- search_awards(
  keywords = c("ethnography", "qualitative data"),
  sources = c("nsf", "nih"),
  from_date = "2018-01-01",
  to_date = "2018-05-01"
)
#> Warning in request(paste0(query_url, "&offset=", offset), "get", verbose): Bad
#> Request (HTTP 400).

#> Warning in request(paste0(query_url, "&offset=", offset), "get", verbose): Bad
#> Request (HTTP 400).
#> No results from NSF
#> Error in (function (..., row.names = NULL, check.rows = FALSE, check.names = TRUE, : arguments imply differing number of rows: 1, 0

^{Created on 2023-07-20 with reprex v2.0.2}

For NSF there are unique IDs, not sure for the others

We have some very basic guidelines in the Readme, but both for ourselves and future contributors, we'd want a full set of style guidelines including:

• structure of new packages (as per #14 )
• preferred dependencies
• anything else?

This is a longterm goal, but I think this would fit very nicely in their portfolio & would get us additional review

include full abstracts from sources where available

and geographic information where available, too

Gates foundation and carnegie modules seem to be malfunctoning, probably from changes in the web search engine

Running list:

Government funders

• https://federalreporter.nih.gov/
• https://ies.ed.gov/funding/grantsearch/index.asp?mode=1&sort=1&order=1&searchvals=data&SearchType=or&checktitle=on&checkaffiliation=on&checkprincipal=on&checkquestion=on&checkprogram=on&checkawardnumber=on&slctAffiliation=0&slctPrincipal=0&slctYear=0&slctProgram=0&slctGoal=0&slctCenter=0&FundType=1&FundType=2
• https://securegrants.neh.gov/publicquery/main.aspx
• Federal Reporter
• USA Spending

Private funders

• https://www.openphilanthropy.org/giving/grants
• https://sloan.org/grants-database
• Social Science Research Council
• Robert Wood Johnson Foundation
• John Templeton Foundation
• Bill and Melinda Gates Foundation
• Andrew Mellon Foundation
• MacArthur Foundation
• Carnegie Foundation
• Open Society Foundations
• Rockefeller Foundation
• Russell Sage Foundation
• Arnold Foundation

e.g. awardsBot(keywords = NULL, sources = NULL, from = NULL, to = NULL)

OK to leave keywords blank, but let's set sources to all currently available, from to reasonable start date (something covered by most APIs perhaps) and to to today's date).

Also, if we require keywords (which I think we would) not setting a default treats it as required which I think is right here. Apply similarly to other functions

Since different query terms have different precision in terms of finding what you want, we'll want the query terms that find a given grant included in the list.
This is complicated, a bit, by de-duplication. Looks like aggregate() via https://stackoverflow.com/a/14262917/1483360 should do the trick, though.
For the delimiter of the aggregated variable, let's do semicolon ;

awardFindR::search_awards("qualitative", "nih", "2022-01-01", "2022-01-15")

I changed the keyword argument from "ethnography" in my example to "data" (and even more specifically "data science" (e.g., awardFindR::search_awards(keywords="data")) and I get the following error:

Error: $ operator is invalid for atomic vectors
In addition: Warning message:
In request(url, "post", verbose, payload) : Bad Request (HTTP 400).

awardFindR::get_nsf("qualitative", "2019-01-01", "2019-02-25")
#> Error in curl::curl_fetch_memory(url, handle = handle): OpenSSL SSL_read: Connection reset by peer, errno 104

^{Created on 2022-09-12 with reprex v2.0.2}

Let's think about a more generic name for this -- especially since I think we'll focus on the search rather than the bot part.
How about
awardFindR?

Travis has gotten less reliable and gh actions very fast. Looks like there is nice integration with usethis:
https://ropenscilabs.github.io/actions_sandbox/

Other resources
https://github.com/r-lib/actions
https://www.rostrum.blog/2020/08/09/ghactions-pkgs/

I was removed from having write access to this repository - whoever has settings access please add me back

Here's the package for codecov: https://github.com/codecov/example-r the current example is for Travis but the gh_actions documentation looks like it'd be trivial to include

While looking at the code I see you use a loop to limit the rate.

If you were to switch from crul to httr2 (and from vcr to httptest2) you could use this function https://httr2.r-lib.org/reference/req_throttle.html

This is just for info :-)

ropensci/software-review#432 (comment)

• Fix all noted errors/unexpected output in examples
• Don't force stringAsFactor on user session
• consistent messages across all functions: include message when no results as in NEH
• don't show HTTP message unless a new verbose argument is set to TRUE
• Fix gates documentation and allow full dates in search
• Better error message for invalid arguments/organizations.

to recreate:

test <- awardFindR("fieldwork", "nsf", "2010-01-01", "2021-01-01")

result:

Error in (function (..., deparse.level = 1, make.row.names = TRUE, stringsAsFactors = default.stringsAsFactors(),  : 
  numbers of columns of arguments do not match 

👋 @mccallc! Via https://ropensci.r-universe.dev/builds we noticed

library("awardFindR")
nsf_and_nih <- search_awards(keywords=c("ontological", "audio recordings"), sources=c("nsf", "nih"), from_date="2018-01-01", to_date="2018-02-01")
#> Warning in request(url, "post", verbose, payload): Bad Request (HTTP 400).
#> Error in if (response$meta$total == 0) {: argument is of length zero
table(nsf_and_nih$source)
#> Error in table(nsf_and_nih$source): object 'nsf_and_nih' not found
unique(nsf_and_nih$keyword)
#> Error in unique(nsf_and_nih$keyword): object 'nsf_and_nih' not found

^{Created on 2024-02-02 with reprex v2.1.0}

I'm unable to run 64-bit on my work computer getting the failure:

** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
ERROR: loading failed for 'x64'
Warning message:
In i.p(...) :
  installation of package ‘C:/Users/ /AppData/Local/Temp/Rtmp4sRUUN/file9d50511c2763/awardFindR_1.0.1.tar.gz’ had non-zero exit status

Is there a way to make it run in 32-bit?

Package check runs all examples in function documentation by default. As a good pratice, let's make these examples as untaxing on the APIs as possible (i.e specific terms, short date ranges). Where that's not possible or desirable use dontrun{} or donttest{} as per https://stackoverflow.com/a/44997778/1483360

To reproduce:

> test <- awardFindR("fieldwork", source="rsf", from="2016-01-01")
GET https://www.russellsage.org/search/node/fieldwork ... OK (HTTP 200).
GET https://www.russellsage.org/search/node/fieldwork?page=1 ... OK (HTTP 200).
GET https://www.russellsage.org/search/node/fieldwork?page=2 ... OK (HTTP 200).
 Error in `$<-.data.frame`(`*tmp*`, "keyword", value = "fieldwork") : 
  replacement has 1 row, data has 0 

Encountered when searching NIH for "focus group" from 01/01/2000 to 01/01/2021

The NIH API throws a HTTP 400 bad request error when you set the offset to greater than 10,000

👋 here!

Via https://ropensci.r-universe.dev/ui#builds= and locally I see

Failure (test-ophil.R:6:3): Expected results
nrow(ophil) (`actual`) not equal to 4 (`expected`).

`actual` is NULL
`expected` is a double vector (4)

Happy to help if needed!

Awards prior to 2010 or so should have leading 0s in their ID numbers, which makes them unsearchable in the NSF web search. These are probably dropped in conversion to a numeric variable type, we should keep them as characters

Gates foundation no longer uses the implemented resource to query awards. Search function needs to be updated

seach_awards("data science", "nsf", "2000-01-01", "2021-12-31", verbose=F) returns nothing

but

seach_awards("data science", "nsf", "2000-01-01", "2021-12-31", verbose=T) returns a lot

In

a file is created but not deleted which is not optimal (especially for CRAN).

I'd suggest using withr::_localtemp_file() instead as it will be deleted automatically.

As per discussion on 2/8: Restructure the apis.R function to be a simple loop/lapply through all sources and move the code that standardizes output into a separate function in the same file as the api_get function. So, e.g., the arnold.R file would contain two functions: arnold_get() and arnold_standardize().

ropensci/software-review#432 (comment)

Definitely do:

• Use Roxygen inherit functionality for documentation
• Move installation instructions toward top of readme (I'd say right after the initial paragraph) + add heading for supported sites and maybe move those down?
• Check on use of read_csv and readr

Discuss

• Change main function name to not clash w package name (I'd be inclined to to this)
• Change verb/subject order of function from neh_get to get_neh. Check whether this is indeed more common. Practically, upside for using neh first is quicker auto-complete. Upside of using get_ first is that you see all get_ functions as part of autocomplete. I could go either way here. Can also ask on Twitter.

See comment here: https://github.com/PESData/sciscrape/blob/main/main.R#L25

Probably use readr::parse_date?

using the included keywords.txt

> data <- awardsBot(keywords = "~/keywords.csv")
Grabbing url: https://securegrants.neh.gov/Open/data/NEH_Grants2020s.csv
(snip)
NOTICE (non-fatal) no NEH results for: ethnographies
Error in data.frame(institution = Institution, pi_name = pi, pi_email = NA,  : 
  object 'AppNumber' not found

Carnegie function returns a maximum of 25 results, e.g. awardFindR::search_awards("method","carnegie",'1998-01-01','2025-01-01', verbose=TRUE) returns 25, wheras searching for "method" returns here returns 98 results.

The guide for submitting packages to be reviewed by Ropensci recommends publishing a pkgdown site for reviewers.

https://pkgdown.r-lib.org/ (note you need to install usethis first)

I compiled this locally and then pushed to a docs folder... Take a look at the site https://pesdata.github.io/awardFindR/ ... We can clean up by adjusting some of the vignettes metadata if we agree this is helpful for review.

Add @export to all functions so they're included in the awardsBot namespace (and thus can be called when installing the packet)

awardFindR::get_osociety("apply", from_year='2000',to_year='2024') returns 10 results, whereas the url https://www.opensocietyfoundations.org/grants/past?filter_keyword=apply returns 161.

As it seems it's not used?