Git Product home page Git Product logo

patrick's Introduction

R-CMD-check CRAN

Introducing patrick

This package is an extension to testthat that enables parameterized unit testing in R.

Installing

The release version of patrick is available on CRAN. Install it in the usual manner:

install.packages("patrick")

The development version of patrick is currently only available on GitHub. Install it using devtools.

devtools::install_github("google/patrick")

To use patrick as a testing tool within your package, add it to your list of Suggests within your package's DESCRIPTION.

Suggests:
    patrick

Use

Many packages within R employ the following pattern when writing tests:

test_that("Data is a successfully converted: numeric", {
  input <- convert(numeric_data)
  expect_type(input, "double")
})

test_that("Data is a successfully converted: character", {
  input <- convert(character_data)
  expect_type(input, "character")
})

While explicit, recycling a test pattern like this is prone to user error and other issues, as it is a violation of the classic DNRY rule (do not repeat yourself). patrick eliminates this problem by creating test parameters.

with_parameters_test_that("Data is successfully converted:", {
    input <- convert(test_data)
    expect_type(input, type)
  },
  test_data = list(numeric_data, character_data),
  type = c("double", "character"),
  .test_name = type
)

Parameterized tests behave exactly the same as standard testthat tests. Per usual, you call all of your tests with devtools::test, and they'll also run during package checks. Each executes independently and then your test report will produce a single report. A complete name for each test will be formed using the initial test description and the strings in the .test_name parameter.

Small sets of cases can be reasonably passed as parameters to with_parameters_test_that. This becomes less readable when the number of cases increases. To help mitigate this issue, patrick provides a case generator helper function.

with_parameters_test_that("Data is successfully converted:", {
    input <- convert(test_data)
    expect_type(input, type)
  },
  cases(
    double = list(test_data = numeric_data, type = "double"),
    character = list(test_data = character_data, type = "character")
  )
)

More complicated testing cases can be constructed using data frames. This is usually best handled within a helper function and in a helper-<test>.R file.

make_cases <- function() {
  tibble::tribble(
    ~ .test_name, ~ expr,      ~ numeric_value,
    "sin",       sin(pi / 4),     1 / sqrt(2),
    "cos",       cos(pi / 4),     1 / sqrt(2),
    "tan",       tan(pi / 4),               1
  )
}

with_parameters_test_that(
  "trigonometric functions match identities",
  {
    testthat::expect_equal(expr, numeric_value)
  },
  .cases = make_cases()
)

If you don't provide test names when generating cases, patrick will generate them automatically from the test data.

Inspiration

This package is inspired by parameterized testing packages in other languages, notably the parameterized library in Python.

Contributing

Please read the CONTRIBUTING.md for details on how to contribute to this project.

Disclaimer

This is not an officially supported Google product.

patrick's People

Contributors

amrrs avatar michaelchirico avatar michaelquinn32 avatar mirca avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

patrick's Issues

Can we only use one expectation inside the `code` argument?

I was wondering if patrick is designed to only allow one expectation, expect_*(), inside with_parameters_test_that(). The examples only have one expectation, and I've only been able to include one expectation in my testing code:

fixed_names = c("index", "original_text", "text", "wav",
                "file", "audio_type",
                "duration",
                "service")

with_parameters_test_that("Google Cloud Text-to-Speech / Amazon Polly Translation",
                          {
                            if (tts_auth) {
                              response_df = tts("Algorithmic complexity is a key consideration
                                                when designing efficient solutions for large-scale data processing",
                                                service = company)
                              testthat::expect_s3_class(response_df, char_value)
                            }
                          },
                          tts_auth = c(tts_amazon_auth(), tts_google_auth()),
                          company  = c("amazon", "google"),
                          char_value = "data.frame"
)

with_parameters_test_that("Google Cloud Text-to-Speech / Amazon Polly Translation",
                          {
                            if (tts_auth) {
                              response_df = tts("Algorithmic complexity is a key consideration
                                                when designing efficient solutions for large-scale data processing",
                                                service = company)
                              testthat::expect_equal(response_df$service, char_value)
                            }
                          },
                          tts_auth = c(tts_amazon_auth(), tts_google_auth()),
                          company  = c("amazon", "google"),
                          char_value = c("amazon", "google")
)

with_parameters_test_that("Google Cloud Text-to-Speech / Amazon Polly Translation",
                          {
                            if (tts_auth) {
                              response_df = tts("Algorithmic complexity is a key consideration
                                                when designing efficient solutions for large-scale data processing",
                                                service = company)
                              audio_value = response_df$wav[[1]]
                              testthat::expect_s4_class(audio_value, char_value)
                            }
                          },
                          tts_auth = c(tts_amazon_auth(), tts_google_auth()),
                          company  = c("amazon", "google"),
                          char_value = "Wave"
)

Could not find function "with_parameters_test_that"

I'm working on a R package for converting text to speech, text2speech. I added patrick to Suggests field within DESCRIPTION, and tested this file:

fixed_names = c("index", "original_text", "text", "wav",
                "file", "audio_type",
                "duration",
                "service")

with_parameters_test_that("Google Cloud Text-to-Speech / Amazon Polly Translation",
                          {
                            if (tts_auth) {
                              response_df = tts("Algorithmic complexity is a key consideration
                                                when designing efficient solutions for large-scale data processing",
                                                service = company)
                              testthat::expect_s3_class(response_df, char_value)
                            }
                          },
                          tts_auth = c(tts_amazon_auth(), tts_google_auth()),
                          company  = c("amazon", "google"),
                          char_value = "data.frame"
)

with_parameters_test_that("Google Cloud Text-to-Speech / Amazon Polly Translation",
                          {
                            if (tts_auth) {
                              response_df = tts("Algorithmic complexity is a key consideration
                                                when designing efficient solutions for large-scale data processing",
                                                service = company)
                              testthat::expect_equal(response_df$service, char_value)
                            }
                          },
                          tts_auth = c(tts_amazon_auth(), tts_google_auth()),
                          company  = c("amazon", "google"),
                          char_value = c("amazon", "google")
)

with_parameters_test_that("Google Cloud Text-to-Speech / Amazon Polly Translation",
                          {
                            if (tts_auth) {
                              response_df = tts("Algorithmic complexity is a key consideration
                                                when designing efficient solutions for large-scale data processing",
                                                service = company)
                              audio_value = response_df$wav[[1]]
                              testthat::expect_s4_class(audio_value, char_value)
                            }
                          },
                          tts_auth = c(tts_amazon_auth(), tts_google_auth()),
                          company  = c("amazon", "google"),
                          char_value = "Wave"
)

However, I get the Error message: could not find function "with_parameters_test_that" . I tried to add library(patrick) in testthat.R, but still got the same message.

Here is my testthat.R

# This file is part of the standard setup for testthat.
# It is recommended that you do not modify it.
#
# Where should you do additional test configuration?
# Learn more about the roles of various files in:
# * https://r-pkgs.org/tests.html
# * https://testthat.r-lib.org/reference/test_package.html#special-files

library(testthat)
library(text2speech)

test_check("text2speech")

.setup= argument to allow re-usable (but hermetic) local variables?

In plain {testthat} we can re-use local variables "hermetically" by running some code before expectations:

test_that("a test", {
  var1 <- 1
  var2 <- 2

  expect_equal(var1, var2 - 1)
  # ... more tests using var1,var2 ...
})

After "a test" runs, we don't have to worry about other tests "seeing" var1/var2, their scope is restricted.

It's not so easy to do this with {patrick}, because we want to declare just once variables used across all the .cases. We might also want to declare the .cases themselves.

The workaround that I've landed on is to wrap with_parameters_test_that() with local({}), but this is a tad clumsy and usage of local({}) is not so prevalent in R code, so it slightly harms readability IMO.

One solution would be to offer a .setup argument that encapsulates what local() is doing:

with_parameters_test_that(
  "a parametrized test",
  .setup = {
    var1 <- 1
    var2 <- 2

   cases <- expand.grid(p1 = 1:10, p2 = 10:1)
  },
  expect_equal(p1 + p2, 10 * var1 + var2 - 1),
  .cases = cases
)

.setup is run first, so var1 and var2 are available to code but also to the other arguments in the with_parameters_test_that call.

WDYT?

Do I add patrick to the Remotes field inside DESCRIPTION?

In the README, you suggest adding patrick to Suggests and Remotes within a package's DESCRIPTION. However, in the code chunk, you only add it to Suggests.

I'm a bit confused as to why patrick should be added to Remotes. According to the R packages book, the Remotes field can be used when you need to install a dependency from a nonstandard place, i.e. from somewhere besides CRAN or Bioconductor.

patrick seems to be in a standard place, i.e. CRAN

Support automatic test case naming

This deserves a bit of thought before going forward

  • right now the fallback is to have no name, which obviously isn't ideal
  • deparsing might be the right next step
  • i also need something like snapshot testing to actually look at the testcase output to be sure that it's sensible

Supply a glue-formatted string to `.test_names=` for building the name from `.cases`

          WDYT about an extension here where `.test_names=` can be a glue-formatted string?
with_parameters_test_that("str", { ... },
  .test_names = "{x} case where {y}, {z}",
  x = ...
  y = ...
  z = ...
)

The actual test name is filled out with x[i], y[i], z[i].

Not fully back-compatible in the case that (1) .test_names were previously supplied with { and (2) the symbol inside {...} resolves to a variable in the parameters. But I don't think it'll break anything, just weird test names maybe, and seems rare anyway.

Originally posted by @MichaelChirico in #11 (comment)

Filing as an issue to make sure it's not lost in the PR comments.

Update for dev testthat

Current test failures:

── FAILURE (test-with_parameters.R:21:7): Running tests: fail ──────────────────
exp\$message does not match "`case` isn't true".
Actual value: "`case` is not TRUE\\n\\n`actual`:   FALSE\\n`expected`: TRUE "

── FAILURE (test-with_parameters.R:21:7): Running tests: null ──────────────────
exp\$message does not match "`case` isn't true".
Actual value: "`case` is not TRUE\\n\\n`actual` is NULL\\n`expected`
is a logical vector \(TRUE\)"

The recommended solution is to use expect_failure() or similar.

Reported by @hadley

Warning for code outside `test_that()`

This is a great package and really worthwhile extension to testthat. It works great and is really intuitive to use!

One thing I noticed is that test_that throws the following warning and I just wanted to know if that's because of a mistake I made, or possibly a known issue.

Code

with_parameters_test_that(
  "distance_result_matches_test_data",
  {
    expect_equal(group_distance(m1,m2), result, tolerance=1e-6)
  },
  cases(
    case1 = list(m1=test_data$A1, m2=test_data$A2, result=test_data$result[1]),
    case2 = list(m1=test_data$B1, m2=test_data$B2, result=test_data$result[2]),
    case3 = list(m1=test_data$C1, m2=test_data$C2, result=test_data$result[3]),
    case4 = list(m1=test_data$D1, m2=test_data$D2, result=test_data$result[4]),
    case5 = list(m1=test_data$E1, m2=test_data$E2, result=test_data$result[5])
  )
)

Warning

── Warning (test_distance.R:19:1): (code run outside of `test_that()`) ─────────
The `code` argument to `test_that()` must be a braced expression to get accurate file-line information for failures.
Backtrace:
 1. patrick::with_parameters_test_that(...) test_distance.R:19:0
 2. purrr::pmap(all_pars, build_and_run_test, desc = desc_stub, code = captured)
 3. patrick:::.f(...)
 4. testthat::test_that(completed_desc, rlang::eval_tidy(code, args))

Thanks,

Fritjof

API for handling `skip()` of some cases?

See r-lib/lintr#2504. I didn't like having to run skip_if() multiple times in every test case. It made more sense to me to exclude certain cases up front.

The downside is that the final {testthat} report doesn't list out which tests were skipped, though.

It would be nice if {patrick} could offer some way to conditionally subset .cases, but also report which cases were skipped. Maybe just a subset= argument a la lm() would work?

Make package available via conda

Hi, we are using your package in one of our projects (and plan to use it for further ones, the parameterized tests are very helpful), but handle our dependencies mostly via conda - would it be possible for you to make the package available via a public conda channel such as conda-forge? This would also improve the visibility of your package beyond cran!

If you'd need any help with that I'm happy to assist!

Creating factorial tests

I'd like to run some factorial tests where the parameters are expand.grid(A, B). Is there an easier way to accomplish this in patrick than:

param_df <- expand.grid(A, B)
patrick::with_parameters_test_that(
  "factorial test",
  expect_passes(foo(A, B))
  .test_name = "test",
  A = param_df$A,
  B = param_df$B
)

If not, could we build this into the API? e.g. .factorial = list(A, B) auto-expands as do.call(expand.grid, .factorial) (perhaps with also setting stringsAsFactors=FALSE?)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.