ReferenceTests.jl is a Julia package that adds a couple of additional macros to your testing toolbox. In particular, it focuses on functionality for testing values against reference files, which in turn the package can help create and update if need be. ReferenceTests.jl is build on top of FileIO.jl and designed to be used alongside Base.Test.

Package Status	Package Evaluator	Build Status

It is very common for Julia packages to test the functionality of their exported functions against known input-to-output combinations. We will refer to such kind of tests as reference tests. In most cases these will be quite simple; something along the line of @test f(x) == y, where f is a function of the user package and x is some interesting input value for which the desired output y is known.

For testing the output of more complex functions, for which the expected output is more complicated (e.g. anything image processing related), using @test can be a little cumbersome to work with. To that end this package provides the @test_reference macro, which expects a filename (relative to the file that invokes the macro) and an expression that evalutes to the value of interest.

using ReferenceTests
@test_reference "stringtest1.txt" string(collect(1:20))

If you put the above code into your test/runtests.jl and execute the file in an interactive julia session (i.e. with include), then it will trigger an interactive dialog if the results don't match or the file does not yet exist. This dialog allows the user to create and/or update the reference files.

The given file stringtest1.txt is assumed to be the relative path to the file that contains the macro invocation. This likely means that the path is relative to the test/ folder of your package.

The file-extension of the filename (here txt), as well as the type of the result of evaluating the expression (here String), determine how the actual value is compared to the reference value. The default implementation will do a simple equality check with the result of FileIO.load. This means that it is the user's responsibility to have the required IO package installed.

Colorant arrays (i.e.) receive special treatment. If the extension of the filename is txt then the package ImageInTerminal.jl will be used to create a string-based crude approximation of the image. This will have low storage requirements and also allows to view the reference file in a simple terminal using cat.

using ReferenceTests, TestImages
@test_reference "imagetest1.txt" testimage("cameraman")

Note that while a text-based storage of reference images can be convenient, proper image formats (e.g. png) are also supported by the package. Those, however, will require the proper FileIO backends to be installed.

Another special file extension is sha256 which will cause the hash of the result of the given expression to be stored and compared as plain text. This is useful for a convenient low-storage way of making sure that the return value doesn't change for selected test cases.

Check out the latest documentation

Additionally, you can make use of Julia's native docsystem. The following example shows how to get additional information on @test_reference within Julia's REPL:

?@test_reference

This code is free to use under the terms of the MIT license.