Feature

• separate arguments for parsing the data and getting attachments in this function.

• odata_submission_get() returns all submissions, but the equivalent function for the RESTful workflow submission_get() returns one submission at a time. Turn this code chunk you have in the vignette into one function.

• Rename submission_get into get_one_submission

• Create submission_get from below code as seen in vignette:

submissions <- tibble::tibble(
  pid = ruODK::get_test_pid(),
  fid = ruODK::get_test_fid(),
  iid = sl$instance_id,  # this is a vector of multiple instance_ids
  url = ruODK::get_test_url(),
  un = ruODK::get_test_un(),
  pw = ruODK::get_test_pw()
) %>% 
  purrr::pmap(ruODK::submission_get)

source ropensci/software-review#335 (comment)

Problem

Columns with date type has missing values. See [date_full] variable in the main table - Submissions and [date_year] in the data_sub1 table generated via below code.

Reproducible example

url = "https://sandbox.central.opendatakit.org"
pid = "12"
fid = "date_test"
un = "[email protected]"
pw = "Test-central06"

srv <- odata_service_get(url=url, pid=pid, fid=fid, un=un, pw=pw)
data <- odata_submission_get(url=url, pid=pid, fid=fid, un=un, pw=pw)
View(data)
data_sub1 <- odata_submission_get(url=url, pid=pid, fid=fid, table=srv$url[2], un=un, pw=pw) 
View(data_sub1)

# utils::sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ruODK_0.6.6.9000 udaRuOdk_0.1.0  

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2        later_0.8.0       pillar_1.4.2      compiler_3.6.1    highr_0.8         tools_3.6.1       digest_0.6.21     zeallot_0.1.0    
 [9] lubridate_1.7.4   jsonlite_1.6      lifecycle_0.1.0   tibble_2.1.3      pkgconfig_2.0.3   rlang_0.4.0       shiny_1.3.2       rstudioapi_0.10  
[17] crosstalk_1.0.0   yaml_2.2.0        curl_4.2          xfun_0.9          httr_1.4.1        stringr_1.4.0     dplyr_0.8.3       knitr_1.25       
[25] janitor_1.2.0     xml2_1.2.2        fs_1.3.1          vctrs_0.2.0       htmlwidgets_1.3   DT_0.9            tidyselect_0.2.5  glue_1.3.1       
[33] snakecase_0.11.0  data.table_1.12.2 R6_2.4.0          purrr_0.3.2       tidyr_1.0.0       magrittr_1.5      listviewer_2.1.0  promises_1.0.1   
[41] htmltools_0.3.6   backports_1.1.4   ellipsis_0.3.0    rlist_0.4.6.1     assertthat_0.2.1  xtable_1.8-4      mime_0.7          httpuv_1.5.2     
[49] stringi_1.4.3     crayon_1.3.4  

Feature

• Acknowledge Karissa and Jason (pending his approval) as reviewers and Maelle as editor in DESC.
• Add a bit more description about ODK Central Sandbox in this doc to help with multiple points of entry. New prospective users may be looking for the quickest way to test and play around with ODK/ruODK to see if it fits the needs of their project, and it seems this would be via a Sandbox account. To this point, you could adjust the "ODK Central Setup" section to say something like “Create a web user account on an ODK Central instance, or request an account on Sandbox..."
• Put “(ruODK)” in backticks for consistency (in CONTRIBUTING doc as well)
• Typo in link “Publish the formto ODK Central”.
• Title of CONTRIBUTING is NA.
• Delete leftover language from the usethis contributing doc. For example: “Before you do a pull request, you should always file an issue and make sure someone from the tidyverse team…”
• "To run tests, you’ll need access to the ODK Central sandbox instance…" I think this should go right at the top or even in the “prerequisites” section of technical contributions.
• Review the need for utils-tidy-eval.R, and these rlang functions exported?
• Drop \code{} from @examples

I’m having some trouble distinguishing between the OData API and RESTful API functionalities. Do these workflows mirror each other? Does one workflow have functionality available that the other does not? I think some additional delineation between these could be helpful. Here are a few ideas on how to address this (suggestions only):
If these are parallel workflows, consider having the vignette structures mirror each other, and add the same blurb at the top of each (like “three ways to happiness") with links to the other vignette. If there are any large benefits or drawbacks of one method or the other, call them out there or in the README.
Does your preferred workflow consist of a mixture of both OData and RESTful calls? If so, consider creating just a typical workflow vignette that goes through a case study using both kinds of functions.

• Clarify purpose of different approaches:
  • REST API one by one: e.g. for incremental ETL to data warehouse. Most modular way to access and parse data.
  • REST API zip dump: for complete export. Easiest way to get all data.
  • OData: most comfortable way to access and parse data.
• Rename vignette file names to odata-api, and restful-api

Why provide a 'hard way' to pass credentials via a function call? My
sense is that this will encourage bad practices in writing code, as
this would give users a way of hard coding passwords in workflows,
which is typically discouraged.

• Clarified and warned in vignette "setup".

source ropensci/software-review#335 (comment)

Feature

Current tests download a 50MB zip file repeatedly. We should use a form with one (even in original file size very small) attachment and one submission for those tests.

Process:

• Use newer example form (flora quadrat 0.4)
• Set data collection device's ODK Collect to take smallest possible photos
• Capture a few test records (30 min)
• Update README, tests, vignettes to use new form
• If the submissions ZIP export is then small enough (say <10MB), tests will run way faster. This will fix #21 for now.

Add tests and increase test coverage to above 80%.

Problem

tidyr::unnest_wider trips over its own feet when trying to unnest a key, here photos_turtle, where all elements (all four possible photos) are NULL in the first list item.

Should ruODK maybe parse the form schema, unnest each existing column explicitly, and cast it into an appropriate type? This would take the tidyr::unnest_wider magic out of the picture entirely.

Reproducible example

  suppressMessages(library(tidyverse))
  library(ruODK)
  prod <- "https://odkcentral.dbca.wa.gov.au"
  uat <- "https://odkcentral-uat.dbca.wa.gov.au"

  ruODK::ru_setup(pid=1, fid="build_Marine-Wildlife-Incident-0-6_1559789189", url=prod)
  ft <- ruODK::odata_service_get()
  mwi_prod <- ruODK::odata_submission_get(table = ft$url[[1]], verbose = T, wkt=T)

The offending item is the first list element containing a nested list containing all NULLs.

The same error exists with other objects where the first record contains NULL values.

Error

<error/vctrs_error_incompatible_cast>
Can't cast `photos_turtle$...1` <logical> to `photos_turtle$...1` <vctrs_unspecified>.
Backtrace:
     █
  1. ├─mwi_prod %>% ruODK::odata_submission_parse()
  2. │ ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  3. │ └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  4. │   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  5. │     └─`_fseq`(`_lhs`)
  6. │       └─magrittr::freduce(value, `_function_list`)
  7. │         ├─base::withVisible(function_list[[k]](value))
  8. │         └─function_list[[k]](value)
  9. │           └─ruODK::odata_submission_parse(.)
 10. │             └─`%>%`(...) /home/florian/projects/ruODK/R/odata_submission_parse.R:98:2
 11. │               ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
 12. │               └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
 13. │                 └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
 14. │                   └─ruODK:::`_fseq`(`_lhs`)
 15. │                     └─magrittr::freduce(value, `_function_list`)
 16. │                       └─function_list[[i]](value)
 17. │                         └─ruODK::unnest_all(., names_repair = names_repair, verbose = verbose)
 18. │                           └─ruODK::unnest_all(nested_tbl, names_repair = names_repair, verbose = verbose) /home/florian/projects/ruODK/R/odata_submission_parse.R:59:4
 19. │                             ├─base::suppressMessages(...) /home/florian/projects/ruODK/R/odata_submission_parse.R:46:6
 20. │                             │ └─base::withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
 21. │                             └─tidyr::unnest_wider(nested_tbl, colname, names_repair = names_repair) /home/florian/projects/ruODK/R/odata_submission_parse.R:46:6
 22. │                               └─tidyr::unchop(data, !!col, keep_empty = TRUE)
 23. │                                 └─vctrs::vec_rbind(!!!x, .ptype = ptype)
 24. ├─vctrs:::vec_cast_dispatch(x = x, to = to, x_arg = x_arg, to_arg = to_arg)
 25. ├─vctrs::vec_cast.data.frame(x = x, to = to, x_arg = x_arg, to_arg = to_arg)
 26. ├─vctrs:::vec_cast.data.frame.data.frame(...)
 27. ├─vctrs:::vec_cast_dispatch(x = x, to = to, x_arg = x_arg, to_arg = to_arg)
 28. └─vctrs:::vec_cast.default(x = x, to = to, x_arg = x_arg, to_arg = to_arg)
 29.   └─vctrs::stop_incompatible_cast(x, to, x_arg = x_arg, to_arg = to_arg)
 30.     └─vctrs:::stop_incompatible(...)
 31.       └─vctrs:::stop_vctrs(...)
> 

Solution

Each form should have a first submission with completely filled in fields, which serve as tent poles to help tidyr::unnest_wider do its job. Who knew parsing structured data was that messy, huh.

> utils::sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 19.10

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C               LC_TIME=en_AU.UTF-8       
 [4] LC_COLLATE=en_AU.UTF-8     LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.4.0    stringr_1.4.0    dplyr_0.8.3      purrr_0.3.3.9000 readr_1.3.1     
 [6] tidyr_1.0.0      tibble_2.1.3     ggplot2_3.2.1    tidyverse_1.3.0  ruODK_0.6.6.9004

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5     xfun_0.11            janitor_1.2.0        haven_2.2.0         
 [5] lattice_0.20-38      snakecase_0.11.0     colorspace_1.4-1     vctrs_0.2.0.9007    
 [9] generics_0.0.2       htmltools_0.4.0      rlang_0.4.2.9000     pillar_1.4.2        
[13] glue_1.3.1           withr_2.1.2          DBI_1.0.0            dbplyr_1.4.2        
[17] modelr_0.1.5         readxl_1.3.1         lifecycle_0.1.0.9000 munsell_0.5.0       
[21] gtable_0.3.0         cellranger_1.1.0     rvest_0.3.5          htmlwidgets_1.5.1   
[25] evaluate_0.14        knitr_1.26           curl_4.2             highr_0.8           
[29] broom_0.5.2          Rcpp_1.0.3           backports_1.1.5      scales_1.1.0        
[33] jsonlite_1.6         fs_1.3.1             hms_0.5.2            packrat_0.5.0       
[37] digest_0.6.23        stringi_1.4.3        rlist_0.4.6.1        grid_3.6.1          
[41] cli_1.1.0            tools_3.6.1          magrittr_1.5         lazyeval_0.2.2      
[45] crayon_1.3.4         pkgconfig_2.0.3      ellipsis_0.3.0       data.table_1.12.6   
[49] xml2_1.2.2           reprex_0.3.0         lubridate_1.7.4      rmarkdown_1.18      
[53] assertthat_0.2.1     httr_1.4.1           rstudioapi_0.10      R6_2.4.1            
[57] nlme_3.1-141         compiler_3.6.1 

Problem

Karissa found:

• Installation succeeds as documented.
• remotes::install_github() works by default, but fails when build_vignettes = TRUE if you do not have your credentials stored prior to build.

Jason found same breakage with auth credentials not available in devtools::install(build_vignettes=TRUE).

Solution

• Have a squizzle at https://github.com/ropensci/rcites/blob/master/vignettes/a_get_started.Rmd for best practices: Vignettes decide whether or not to eval code chunks based on an environment variable (in this case NOT_CRAN) and both ways are tested in the Travis-CI testing grid. https://github.com/ropensci/rcites/blob/master/.travis.yml
• Include canned data for all results from all web requests sent by vignettes.
• This will enable vignettes to build from canned data without the need for ODK Central sandbox credentials.
• This will require extra tests to verify that canned data is identical to results from web requests.
• Add a clear delineation as to what can be done with and without test credentials.

source ropensci/software-review#335 (comment)

Feature

Implement, test, and show in vignette:

• form_schema_json

• form_xml

• submission_detail

• submission_export

• submission_list

• attachment_get

• attachment_list

• Add examples to all function docs. So far, we only have tests and vignettes. Boo.

Feature

Use lifecycle badges on each fuction.

https://lifecycle.r-lib.org/

This will assist reviewers of ropensci/software-review#335 to see where minor changes might come through and what's already on par with ODK Central 0.6.

Feature

Add diagram of ODK data flow to README.
Source ropensci/software-review#335 @maelle

Feature

Source ropensci/software-review#335 @maelle

The reference section of the pkgdown website could use some grouping.
https://devguide.ropensci.org/building.html#function-grouping
(ignore the CI advice, outdated, cf https://ropensci.org/technotes/2019/06/07/ropensci-docs/). 
2) For info here is our guidance for accepted packages: https://devguide.ropensci.org/collaboration.html#contributing-guide

Feature

Examples for all exported functions in R Help that run successfully locally – not all exported functions have examples, but some internal helper functions are exported that could potentially be removed from exports (e.g. prepend_uuid()).

Not all exported functions have examples, but some internal helper functions are exported that may not need to be (e.g. prepend_uuid(), yell_if_missing()).

source ropensci/software-review#335 (comment)

Problem

Audit results are overwritten and in the folder only the last submission's audit file is saved.

Reproducible example

library(ruODK)
    library(tidyverse)
    ruODK::ru_setup(
        svc = "https://sandbox.central.opendatakit.org/v1/projects/12/forms/simple_hh_question.svc",
        un = "[email protected]",
        pw = "Test-central06"
    )
    srv <- ruODK::odata_service_get()
    
    data <- ruODK::odata_submission_get(parse=TRUE, wkt=TRUE, table = srv$name[1], verbose = TRUE, local_dir = '/media', tz = "Australia/Perth")
    list.files(path = "/media")

# utils::sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 10 x64 (build 18362)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] gapminder_0.3.0  magrittr_1.5     forcats_0.4.0    stringr_1.4.0    purrr_0.3.3      readr_1.3.1      tidyr_1.0.0      tibble_2.1.3     ggplot2_3.2.1   
[10] tidyverse_1.2.1  dplyr_0.8.3      glue_1.3.1       ruODK_0.6.6.9001

loaded via a namespace (and not attached):
 [1] withr_2.1.2          rvest_0.3.4          tidyselect_0.2.5     lattice_0.20-38      pkgconfig_2.0.3      utf8_1.1.4           xml2_1.2.2          
 [8] compiler_3.6.1       htmlwidgets_1.5.1    fs_1.3.1             readxl_1.3.1         Rcpp_1.0.2           cli_1.1.0            cellranger_1.1.0    
[15] httr_1.4.1           tools_3.6.1          nlme_3.1-140         broom_0.5.2          R6_2.4.0             scales_1.0.0         assertthat_0.2.1    
[22] curl_4.2             digest_0.6.22        gtable_0.3.0         fansi_0.4.0          stringi_1.4.3        rstudioapi_0.10      janitor_1.2.0       
[29] hms_0.5.1            backports_1.1.5      htmltools_0.4.0      munsell_0.5.0        grid_3.6.1           lifecycle_0.1.0.9000 colorspace_1.4-1    
[36] data.table_1.12.6    lubridate_1.7.4      rlang_0.4.1.9000     ellipsis_0.3.0       generics_0.0.2       vctrs_0.2.0          lazyeval_0.2.2      
[43] zeallot_0.1.0        snakecase_0.11.0     haven_2.1.1          crayon_1.3.4         modelr_0.1.5         pillar_1.4.2         remotes_2.1.0       
[50] DT_0.9               rlist_0.4.6.1        jsonlite_1.6 

Feature

Source ropensci/software-review#335 (comment)
@maelle

Examples

• https://ropensci.github.io/CoordinateCleaner/articles/Background_comparison_other_software.html
• https://rtweet.info/#package-functionality

Candidates

ODK

• https://github.com/validmeasures/odkr
• https://cran.r-project.org/web/packages/odk/index.html
• https://github.com/nap2000/odkmeta and fork https://github.com/LloydOBanwart/odkmeta

OData

• https://github.com/HanOostdijk/odataR
• https://cran.r-project.org/web/packages/cbsodataR/
• https://cran.r-project.org/web/packages/OData/index.html
• https://www.cdata.com/kb/tech/odata-jdbc-r.rst

Note to self: chuck in a reference to https://www.odata.org/

Feature

{ruODK} instead of `ruODK'.

Goal: signify package names in prose.

ropensci/software-review#335

Feature

source ropensci/software-review#335 (comment)

Problem

Reproducible example

Reported at ropensci/software-review#335 (comment)
by @jmt2080ad

test-attachment_get.R:80: error: get_one_attachment handles repeat
download and NA filenames
unused argument (check = TRUE)
1: .handleSimpleError(function (e) 
   {
       handled <<- TRUE
       test_error <<- e
       options(expressions = expressions_opt_new)
       on.exit(options(expressions = expressions_opt), add = TRUE)
       e$expectation_calls <- frame_calls(11, 2)
       test_error <<- e
       register_expectation(e)
       e$handled <- TRUE
       test_error <<- e
   }, "unused argument (check = TRUE)", quote(tempdir(check = TRUE)))
   at /home/....../test-attachment_get.R:80

# utils::sessionInfo()

Current workflow, see vignette odata:

• odata_submission_get downloads a bunch of submissions
• odata_submission_parse parses them into tibbles
• parse_datetime parses columns matching a name into dttm (requires knowledge of form field types)
• attachment_get run on attachment columns (requires knowledge of form field types) downloads attachments and links to local paths

This workflow is a manually built, deliberate, and quite satisfying process to iteratively understand and parse the data. Since working examples are given in the vignette, the process is probably not too hard to replicate on own data.

On the other hand, we could optionally automate the above process with form introspection. Form introspection is the core idea of OData. Ironically, the plain JSON output of the form schema is way more manageable than the OData service XML with its field/attribute/property tornado.

fsp <- form_schema(parse=TRUE)
fsp %>% dplyr::filter(type=="binary") %>% dplyr::select(name) # attachment field names!
fsp %>% dplyr::filter(type=="dateTime") %>% dplyr::select(name) # dttm field names!

Idea: odata_submission_get(parse=TRUE) to internally call odata_submission_parse, form_schema(parse=TRUE), then run attachment_get on the columns identified as "binary" attachments by form_schema, and parse_datetime on columns identified as "dateTime".

It is still necessary to provide the steps individually in order to let users debug any possible breakages.

Timeline:
ODK Central 1.0

Feature

Rename the OData functions from get_OBJECT to odata_OBJECT_VERB.
This leaves the default OBJECT_VERB naming scheme for the more numerous RESTful API functions.

Support paginated OData submissions. Can we get away with reading an entire dataset into memory?
Add option to download only new or all submissions. Can we query the OData stream for submissions newer than the latest locally downloaded submission?

Status quo:

• Vignettes show working examples of smaller use cases.
• Man pages have info on pagination for odata_submission_get through skip (offset) and top (limit).

Remaining tasks:

• Demonstrate pagination through both OData and REST API in vignettes.

Timeline:

• Flesh out those bigger use cases through own dogfooding. ETA later 2019 once production data surpasses manageable size.

Feature

Source ropensci/software-review#335 @maelle

Task

I see a few open issues in your repo. 
Could you clarify here and in their description whether you're seeking advice 
on the functionalities implementation/usefulness? 
Why not solve some of them before the reviews of the package? 
It'd be better for the functionality to be implemented so the reviewers can review them too. 
It is fine if we put this issue on hold for a bit in order for you to have time to work on 
the enhancements if needed, and you could ask for help in rOpenSci channels.

The metadata contain a definition of the dataset. Form groups (ODK Build groups) result in nested data. Some other data types, such as location, also generate nested data.

For now, knowledge of the dataset is required to unnest the nested levels into a flat table (corresponding to what's exported from ODK Central as CSV).

Challenge:

• derive the nesting levels from the metadata, and
• automate the unnesting based on this information (repeated %>% tidyr::unnest_wider(.))

Feature

Support data retrieval from encrypted projects.
Docs: https://docs.getodk.org/central-encryption/

Caveats

• Encrypted data is not available via OData, only via CSV/ZIP export

In scope

• Project managed encryption (see docs)
• https://odkcentral.docs.apiary.io/#reference/forms-and-submissions/submissions/exporting-form-submissions-to-csv-via-post
  • ruODK::submission_export - change from GET to POST and add optional param for encryption key.
• Double check - does the API endpoint called by ruODK::submission_get docs work in encrypted projects?
• Double check - what happens with ruODK::odata_submission_get() in encrypted projects? Should ruODK emit a warning that only metadata are returned?

Nice to have

• https://odkcentral.docs.apiary.io/#reference/forms-and-submissions/submissions/listing-encryption-keys
• Which other functions would provide value to users of encrypted projects? Paging @dmenne @michaeledwardmarks @Thaliehln
• Vignette "Honey I've encrypted the kids" to demonstrate workflow with encrypted projects. e.g.:
  • Enable encryption through GUI: link to ODK docs https://docs.getodk.org/central-encryption/#using-project-managed-encryption
  • Collect some data (we'll use the test form)
  • Export submissions via all new ruODK::submission_export() with encryption keys and POST
  • Visualise the decrypted data
  • Mention out of scope encryption features of ODK Central and solicit user feedback on what's missing

Out of scope

• Enabling project managed encryption through ruODK
  • https://odkcentral.docs.apiary.io/#reference/project-management/projects/enabling-project-managed-encryption
  • best done through ODK Central's GUI, since this is a one-off per project it would spam the test server with new/encrypted/deleted projects.
• https://odkcentral.docs.apiary.io/#reference/forms-and-submissions/submissions/exporting-form-submissions-to-csv via GET

MVP

• https://odkcentral.docs.apiary.io/#reference/forms-and-submissions/submissions/exporting-form-submissions-to-csv-via-post with optional encryption passphrase
• One new encrypted test project on test server: https://odkc.dbca.wa.gov.au/#/projects/3/forms/Locations with Passphrase ThePassphrase

Feature

Prefixing is a good idea, as it reduces the risk of attachment filename collisions.
Not prefixing reduces the packages size, as the vignettes can share downloaded media files.

Make prefixing a default but opt-out feature. Test both ways, use non-prefixed in vignettes to reduce package size.

Relevant to ropensci/software-review#335

Feature

• add utils::globalVariables(".") to ruODK.R (check other global vars too)
• drop . <- NULL from functions
• enjoy the contented silence of devtools::check()
• owe Karissa a beer for the great tip

source ropensci/software-review#335 (comment)

Feature

Show a worked examples as asciicast in the README.

Thanks to @arestrom for the idea: #50 (comment)

• Add ru_setup(verbose = FALSE) to set env var RU_VERBOSE (default: FALSE)
• Add getter get_ru_verbose() as Sys.getenv("RU_VERBOSE", FALSE)
• Add to ru_settings(): verbose = get_ru_verbose()
• Amend all function params verbose to default to ruODK::get_ru_verbose()

Feature

ruODK fails attachment downloads on incomplete files (bad uploads) or server hiccups. Retrying the download fixes the problem. Deleting the locally downloaded bad attachment file (76kB "photos") re-creates the problem.
This problem shouldn't happen but has been observed with real-world data (WA turtle nesting census, 30k records/season). Restarting downloads of all the data sucks with large datasets.

We should make ruODK retry a download on failure (verbosely, and with opt-out param).

Feature

Clarify publication plans

Task

Source ropensci/software-review#335 @maelle

Reg "Note: I would like to submit a paper about the package in a few weeks, 
but haven't got the manuscript ready and approved for publication just yet." 
This is also a good reason to put the issue on hold since the reviewers 
would read the short JOSS paper. What do you mean by approved?

Feature

Since rOpenSci appears to host and theme pkgdown sites, although I can't really tell from https://devguide.ropensci.org/ci.html#even-more-ci-ropensci-docs what build process is used (Jenkins seems to be involved).

@maelle @sckott would it be helpful if ruODK were to build its site via usethis::use_pkgdown_travis()?
The main challenge might be the embedded media files. Solved example:

• make_data prepares files and generates vignette header
• vignette specifies its required media files for tic to host
• hosted vignette now shows media files in popups and reactable row details

Feature

Source ropensci/software-review#335 @maelle

I do not understand "Especially in these trying times, it is important to ask: “ruODK?”" 
and " u r ODK!" in the README, could you add footnotes or so there? 
Unless it is very obvious see_no_evil (play on words with OK?
 to my defence and the defence of my suggestion, 
my not being a native speaker might have made this less clear joy )

Likewise I do not understand the use of %<% when describing use cases.

There is a link missing " [ODK forum]". How should the future reviewers best ask 
for credentials if they're new to the service? I wasn't too sure about my own post.

In the example in the README you source .Rprofile. This should not be done, 
in your code in at least one function I see you use Sys.getenv("NAMEOFTHEVARIABLE") 
to access the credentials which is the best practice.

Feature

[x] Implement changes to API
[x] Adjust semver to reflect supported ODK Central version
[ ] Implement function stubs for all not (yet) implemented ODK Central API endpoints

Feature

HT @karissawhiting

• Make it possible to parse the data, but not download the attachments.
• What URL parameter should we add for this? Or default local_dir to NULL as toggle to not download?

When ODK Central allows hotlinking attachments (sometime in the future),it would be useful to store the fully qualified attachment URL if not downloading attachments.

Source: ropensci/software-review#335

Detect date formats and parse with lubridate. Add option to opt in or out.
Metadata property type Edm.DateTimeOffset are datetimes. Find out other property types for date and time.

Feature

Include Rmd templates for known workflows:

• Download form submissions as ZIP, extract, analyse
• Download OData schema, submissions, nested tables, attachments
• Download REST submissions, attachments

Examples see https://github.com/dr-harper/example-rmd-templates

Include form templates

Already included, see inst/, mention in Rmd templates how to access and use on ODK Central.

This would be immensely useful for participants of a hands-on demo workshop, or to any users of ruODK.

Feature

https://devguide.ropensci.org/guide-for-authors.html
http://joss.theoj.org/about#author_guidelines

Feature

A very recent addition to the ODK Forum, https://forum.opendatakit.org/t/automating-data-delivery-using-the-odata-endpoint-in-odk-central/22010 is too good to leave out from the software comparison.

Feature

All functions using settings should fail with a meaningful error message.

Source ropensci/software-review#335 @maelle

The current error messages are not informative in the absence of credentials, 
i.e. I get Error in curl::curl_fetch_memory(url, handle = handle) : <url> malformed 
if I run project_list() now without credentials. 
If there's no system variables set and no credentials passed either, 
the functions should fail with an informative error message.

Feature

• Add ru_settings(pw=FALSE) to hide or show pw
• ru_setup
  • Print ru_settings(pw=FALSE) as last step of ru_setup
  • add parameter write=FALSE to write to .Renviron (steal from ggmaps::register_google())
• add ru_soundcheck to test:
  • default settings: set? working? warn if not
  • test settings: set? working? warn if not
  • return TRUE if both work

source ropensci/software-review#335 (comment)

Feature

Clarify which accounts and settings to request and use to run tests and build the vignettes.

Source ropensci/software-review#335 @maelle

Feature

Logging this idea for future Florian:
https://devguide.ropensci.org/building.html#recommended-scaffolding

This will make sense once ODK Central has reached 1.0 and development slows down.
Until then, ODK Central will add features to its API, and ruODK targets the latest ODK Central version (as indicated by mirroring the major/minor version numbers).

Suggestion: the best approach until 1.0 is to test the real deal and run tests uncached against the latest ODK Central instance run by ODK themselves. If those tests work, ruODK will be O(D)K.

Feature

The example shows parsing a point location into lat, lon, alt, acc. Now do lines and polys!

odata_submission_parse breaks Geopoints into their components (lat, lon, alt, acc). Being an unnamed list, these numbers will be named by tidyr::unnest_wider(.names_repair="universal") as ...1, ...2 etc.

We should provide a function to .names_repair which uses the full path, e.g. mylocation_1, mylocation_2, mylocation_3. We can't name them latitude etc as a form may have several geopoints. The field name however is unique per form and would be a great prefix.

Current status: Vignette "odata" demonstrates how to rename fields manually. This is good enough to be used in production.

Problem

In testing the majority of ruODK functions all seem to work well so far...with the exception of the get_submission() functions. However, when I edited the get_one_submission() function and removed magrittr::extract2("data") it all worked and submissions downloaded as expected.

Reproducible example

# Amended function works...but get_one_submission() returned NULLs
get_single_submission <- function(
  iid,
  pid = get_default_pid(),
  fid = get_default_fid(),
  url = get_default_url(),
  un = get_default_un(),
  pw = get_default_pw()) {
  glue::glue("{url}/v1/projects/{pid}/forms/{fid}/submissions/{iid}.xml") %>%
    httr::GET(httr::authenticate(un, pw)) %>%
    httr::content(.) %>%
    xml2::as_list(.)
}

# utils::sessionInfo()

Feature

Reported at ropensci/software-review#335 (comment)

Feature

Tidyr turned 1.0.0!
This means that {ruODK} can depend on stable CRAN tidyr>=1.0.0 instead of dev version 0.8.3.9000 to have access to tidyr::unnest_wider().

This will make installation of latest master of tidyr and its dependencies less frequent and save reviewers of ropensci/software-review#335 some time.

Problem

Image field and audit field have the type in the dataframe generated using "odata_submission_get" function. This creates problem for saving the dataset as csv file.

Reproducible example

library(ruODK)
    library(tidyverse)
    ruODK::ru_setup(
        svc = "https://sandbox.central.opendatakit.org/v1/projects/12/forms/simple_hh_question.svc",
        un = "[email protected]",
        pw = "Test-central06"
    )
    srv <- ruODK::odata_service_get()
    
    data <- ruODK::odata_submission_get(parse=TRUE, wkt=TRUE, table = srv$name[1], verbose = TRUE, local_dir = '.', tz = "Australia/Perth")
    write.csv(data, "mydata.csv", row.names = TRUE, fileEncoding = "UTF-8")

# utils::sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 10 x64 (build 18362)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] gapminder_0.3.0  magrittr_1.5     forcats_0.4.0    stringr_1.4.0    purrr_0.3.3      readr_1.3.1      tidyr_1.0.0      tibble_2.1.3     ggplot2_3.2.1   
[10] tidyverse_1.2.1  dplyr_0.8.3      glue_1.3.1       ruODK_0.6.6.9001

loaded via a namespace (and not attached):
 [1] withr_2.1.2          rvest_0.3.4          tidyselect_0.2.5     lattice_0.20-38      pkgconfig_2.0.3      utf8_1.1.4           xml2_1.2.2          
 [8] compiler_3.6.1       htmlwidgets_1.5.1    fs_1.3.1             readxl_1.3.1         Rcpp_1.0.2           cli_1.1.0            cellranger_1.1.0    
[15] httr_1.4.1           tools_3.6.1          nlme_3.1-140         broom_0.5.2          R6_2.4.0             scales_1.0.0         assertthat_0.2.1    
[22] curl_4.2             digest_0.6.22        gtable_0.3.0         fansi_0.4.0          stringi_1.4.3        rstudioapi_0.10      janitor_1.2.0       
[29] hms_0.5.1            backports_1.1.5      htmltools_0.4.0      munsell_0.5.0        grid_3.6.1           lifecycle_0.1.0.9000 colorspace_1.4-1    
[36] data.table_1.12.6    lubridate_1.7.4      rlang_0.4.1.9000     ellipsis_0.3.0       generics_0.0.2       vctrs_0.2.0          lazyeval_0.2.2      
[43] zeallot_0.1.0        snakecase_0.11.0     haven_2.1.1          crayon_1.3.4         modelr_0.1.5         pillar_1.4.2         remotes_2.1.0       
[50] DT_0.9               rlist_0.4.6.1        jsonlite_1.6  

Problem

Source ropensci/software-review#335 @maelle

Do not use the Author and Maintainer fields in DESCRIPTION, they are automatically generated from Authors@R.

Problem

While trying to get submissions using ruODK::odata_submission_get
I'm getting: Error: No common type for `..1$coordinates$...1` <list> and `..4$coordinates$...1` <logical>.
When I have only submission without GPS information I had no problem using the above command
Full command:

  fq_data <- ruODK::odata_submission_get(
    table = fq_svc$name[1], verbose = TRUE, tz = tz, local_dir = loc
  ) %>%
    dplyr::rename(
      longitude = x12,
      latitude = x13,
      altitude = x14
    )

EDIT
First I tested with my gps variables

  fq_data <- ruODK::odata_submission_get(
    table = fq_svc$name[1], verbose = TRUE, tz = tz, local_dir = loc
  ) %>% 
    dplyr::rename(
      longitude = gps-Latitude,
      latitude = gps-Longitude,
      altitude = gps-Altitude
    )

Backtrace with: rlang::last_error()

<error>
message: No common type for `..1$coordinates$...1` <list> and `..4$coordinates$...1` <logical>.
class:   `vctrs_error_incompatible_type`
backtrace:
  1. `%>%`(...)
 32. vctrs:::vec_ptype2.list.default(...)
 33. vctrs::stop_incompatible_type(x, y, x_arg = x_arg, y_arg = y_arg)
 34. vctrs:::stop_incompatible(...)
 35. vctrs:::stop_vctrs(...)

     █
  1. ├─`%>%`(...)
  2. │ └─base::eval(lhs, parent, parent)
  3. │   └─base::eval(lhs, parent, parent)
  4. ├─ruODK::odata_submission_get(...)
  5. │ └─sub %>% odata_submission_parse(verbose = verbose)
  6. │   ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  7. │   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  8. │     └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  9. │       └─ruODK:::`_fseq`(`_lhs`)
 10. │         └─magrittr::freduce(value, `_function_list`)
 11. │           ├─base::withVisible(function_list[[k]](value))
 12. │           └─function_list[[k]](value)
 13. │             └─ruODK::odata_submission_parse(., verbose = verbose)
 14. │               └─`%>%`(...)
 15. │                 ├─base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
 16. │                 └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
 17. │                   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
 18. │                     └─ruODK:::`_fseq`(`_lhs`)
 19. │                       └─magrittr::freduce(value, `_function_list`)
 20. │                         └─function_list[[i]](value)
 21. │                           └─ruODK::unnest_all(., names_repair = names_repair, verbose = verbose)
 22. │                             └─ruODK::unnest_all(nested_tbl, names_repair = names_repair, verbose = verbose)
 23. │                               └─ruODK::unnest_all(nested_tbl, names_repair = names_repair, verbose = verbose)
 24. │                                 └─tidyr::unnest_wider(nested_tbl, colname, names_repair = names_repair)
 25. │                                   └─tidyr::unchop(data, !!col, keep_empty = TRUE)
 26. │                                     └─vctrs::vec_rbind(!!!x, .ptype = ptype)
 27. ├─vctrs:::vec_type2_dispatch(x = x, y = y, x_arg = x_arg, y_arg = y_arg)
 28. ├─vctrs:::vec_ptype2.tbl_df(x = x, y = y, x_arg = x_arg, y_arg = y_arg)
 29. ├─vctrs:::vec_ptype2.tbl_df.data.frame(...)
 30. ├─vctrs:::vec_type2_dispatch(x = x, y = y, x_arg = x_arg, y_arg = y_arg)
 31. ├─vctrs::vec_ptype2.list(x = x, y = y, x_arg = x_arg, y_arg = y_arg)
 32. └─vctrs:::vec_ptype2.list.default(...)
 33.   └─vctrs::stop_incompatible_type(x, y, x_arg = x_arg, y_arg = y_arg)
 34.     └─vctrs:::stop_incompatible(...)
 35.       └─vctrs:::stop_vctrs(...)

Reproducible example

Upload the attached Islands.xml form to Central. Send several submissions without GPS and some with GPS coordinates.
Try to run ruODK::odata_submission_get in R

Islands.zip

# utils::sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ruODK_0.6.6.9000 udaRuOdk_0.1.0  

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2        later_0.8.0       pillar_1.4.2      compiler_3.6.1    highr_0.8         tools_3.6.1       digest_0.6.21     zeallot_0.1.0    
 [9] lubridate_1.7.4   jsonlite_1.6      lifecycle_0.1.0   tibble_2.1.3      pkgconfig_2.0.3   rlang_0.4.0       shiny_1.3.2       rstudioapi_0.10  
[17] crosstalk_1.0.0   yaml_2.2.0        curl_4.2          xfun_0.9          httr_1.4.1        stringr_1.4.0     dplyr_0.8.3       knitr_1.25       
[25] janitor_1.2.0     xml2_1.2.2        fs_1.3.1          vctrs_0.2.0       htmlwidgets_1.3   DT_0.9            tidyselect_0.2.5  glue_1.3.1       
[33] snakecase_0.11.0  data.table_1.12.2 R6_2.4.0          purrr_0.3.2       tidyr_1.0.0       magrittr_1.5      listviewer_2.1.0  promises_1.0.1   
[41] htmltools_0.3.6   backports_1.1.4   ellipsis_0.3.0    rlist_0.4.6.1     assertthat_0.2.1  xtable_1.8-4      mime_0.7          httpuv_1.5.2     
[49] stringi_1.4.3     crayon_1.3.4