https://metacpan.org/dist/Test-Simple/view/lib/Test/Tutorial.pod is very relevant, see if there's parts worth incorporating into our vignettes.

When the unevaluated call to ok is to == (or another known function), redirect it to a ut_cmp_eq() that can:

• Implicitly do all(x == y) (possibly this is something ok() should be doing anyway, and accepting all(isTRUE(result)) as success)
• Generate useful failure output by evaluating each half of the == on it's own, in the appropriate environment

See 8953e16 for an implementation.

However, this doesn't work with Rscript / CMD check. option(keep.source) gives you a number within the first code-block (i.e. ok_group()).

#10 would make this a bit less of a problem.

\dontrun should only be used as last-resort. I don't think that's the case here, we're just attempting to format examples in a non-standard fashion.

In TAP 14, plans are non-optional, but can be at the end.

Count tests as we go, and output the plan at the end.

Put the following somewhere in the suggested boilerplate:

if (!interactive()) options(warn=2, error = function() { sink(stderr()) ; traceback(3) ; q(status = 1) })

...we could have a helper to do this, but I'm guessing there's too much that will anger CRAN checks to make it worthwhile.

There's currently a test failure: https://cran.r-project.org/web/checks/check_results_unittest.html

 not ok - Environments get converted to lists
 # Test returned non-TRUE value:
 # Length mismatch: comparison on first 2 components
 # Component "a": Mean relative difference: 0.6666667
 # <1b>[1m--- as.environment(list(a = 3, b = 4))<1b>[m
 # <1b>[1m+++ as.environment(list(a = 5, b = 4, c = 9))<1b>[m
 # <1b>[32m{+$c
 +}<1b>[m
 # <1b>[32m{+[1] 9+}<1b>[m
 #
 # $b<1b>[m
 # [1] 4<1b>[m
 #
 # $a<1b>[m
 # [1] <1b>[31m[-3-]<1b>[m<1b>[32m{+5+}<1b>[m

...without escapes (which should be filtered before comparsion) this looks like:

# Test returned non-TRUE value:
# Length mismatch: comparison on first 2 components
# Component "a": Mean relative difference: 0.6666667
# --- as.environment(list(a = 3, b = 4))
# +++ as.environment(list(a = 5, b = 4, c = 9))
# {+$c+}
# {+[1] 9+}
#
# $b
# [1] 4
#
# $a
# [1] [-3-]{+5+}

The current version on CRAN will show the raw output from the test that failed. The only difference to the expected output is the carriage return after $c, but I have no idea where it comes from.

No way of replicating this, other than re-releasing and seeing if it fails again.

ok_group("moo", { ok(..) }) will result in all tests clumped together in CRAN output, instead of per-line output.

As a result it might be better to write ok_group("moo") ; ok(...).

I've just put this together:

ut_cmp_warnings <- function(code, expected_regexes, ignore.case = FALSE, perl = FALSE, fixed = FALSE) {
    stopifnot(is.character(expected_regexes) || length(expected_regexes) == 0)
    all_warnings <- list()
    rv <- withCallingHandlers(code, warning = function (w) {
        all_warnings <<- c(all_warnings, list(w))
        invokeRestart("muffleWarning")
    })

    if (length(expected_regexes) != length(all_warnings)) {
        return(c(
            paste0("Expected ", length(expected_regexes), " warnings, ", "not ", length(all_warnings), ":"),
            vapply(all_warnings, function (w) w$message, character(1)) ))
    }
    out <- c()
    for (i in seq_along(expected_regexes)) {
        if (!grepl(expected_regexes[[i]], all_warnings[[i]]$message,
                ignore.case = ignore.case, perl = perl, fixed = fixed)) {
            out <- c(all_warnings[[i]]$message, "Did not match:-", expected_regexes[[i]])
        }
    }
    return(if (length(out) == 0) TRUE else out)
}

It expects a character vector of regexes, one per warning, and they should be in order.

Insisting that we provide one regex per-warning is probably reasonable, if they all match you ought still care about the number of warnings. You could do something like rep(regex, count) to say you're expecting count warnings matching regex.

What's more awkward is if you want to check for your warnings, but ignore third-party warnings out of your control. We could instead make sure there is at least one warning matching each regex, but I'm more tempted to say "if you really want to ignore extra warnings, then you're on your own, this isn't something we encourage".

EDIT: Insisting they're in order is also debatable, but it's pretty unlikely you're testing multi-threaded code.

The filter options flip through a string representation of an object until it finds one that doesn't match:

The theory was it should end with multiple str calls, increasing length / accuracy until it generates output that is different. Something like:-

    function (x) utils::str(x, vec.len = 1000, digits.d = 5, nchar.max =
1000),
   function (x) utils::str(x, vec.len = 1e10, digits.d = 10, nchar.max =
1e10),
   function (x) utils::str(x, vec.len = 1e20, digits.d =
.Machine$double.digits, nchar.max = 1e20),

Move package documentation out of faq into it's own vignette, and include stuff on testing vignettes, basically:

{r, message=FALSE, echo=FALSE}
logging::setLevel('WARN')
library(unittest)
# Redirect ok() output to stderr
ok <- function(...) capture.output(unittest::ok(...), file = stderr())

Otherwise the behaviour will depend on locale, and you will have a bad day.

Currently we tell people to run tests with CMD check or Rscript & a for loop, either is a bit ropey.

Include a test runner, borrowing ideas from tinytest that:

• Allows for parallel tests without Makefile magic
• Is compatible with the wip-line-numbers branch

I can make a helper to compare environments, e.g.:

cmp_environment <- function (a, b) {
    a <- as.list(a)
    b <- as.list(b)
    if (length(a) == 0) {
        # Can't order an empty list
        ut_cmp_identical(a, b)
    } else {
        ut_cmp_identical(a[order(names(a))], b[order(names(b))])
    }
}
ok(cmp_environment( as.environment(list(a=1, b=2)), as.environment(list(a=1, b=2))   )  )

...but then the diff output doesn't include the previous expressions as.environment(list(a=1, b=2)), just +++ a[order(names(a))], which doesn't help. Need to either:-

• Tell cmp_inner that we're a helper function, and the -1 should be upped to -2
• Supply a "comparison filter" to convert to (ordered) list (which makes the helper shorter, but doesn't really solve the problem)
• Expose cmp_inner, allow us to write our own comparison function.