"Take Julia code coverage and memory allocation results, do useful things with them"
Code coverage: Julia can track how many times, if any, each line of your code is run. This is useful for measuring how much of your code base your tests actually test, and can reveal the parts of your code that are not tested and might be hiding a bug. You can use Coverage.jl to summarize the results of this tracking, or to send them to a service like Coveralls.io or Codecov.io.
Memory allocation: Julia can track how much memory is allocated by each line of your code. This can reveal problems like type instability, or operations that you might have thought were cheap (in terms of memory allocated) but aren't (i.e. accidental copying).
- Coverage.jl (this package): allows you to take coverage results and submit them to online web services such as Codecov.io and Coveralls.io
- CoverageTools.jl: core functionality for processing code coverage and memory allocation results
Most users will want to use Coverage.jl.
Step 1: Navigate to your test directory, and run julia with the --code-coverage
option:
julia --code-coverage=user
julia --code-coverage=tracefile-%p.info --code-coverage=user # available in Julia v1.1+
Step 2: Run your tests (e.g., include("runtests.jl")
) and quit Julia. (If you use Pkg.test
to run your tests, set the coverage
keyword argument to true
, i.e. Pkg.test("MyPkg"; coverage=true)
.)
Step 3: Navigate to the top-level directory of your package, restart Julia (with no special flags) and analyze your code coverage:
using Coverage
# process '*.cov' files
coverage = process_folder() # defaults to src/; alternatively, supply the folder name as argument
coverage = append!(coverage, process_folder("deps"))
# process '*.info' files
coverage = merge_coverage_counts(coverage, filter!(
let prefixes = (joinpath(pwd(), "src", ""),
joinpath(pwd(), "deps", ""))
c -> any(p -> startswith(c.filename, p), prefixes)
end,
LCOV.readfolder("test")))
# Get total coverage for all Julia files
covered_lines, total_lines = get_summary(coverage)
# Or process a single file
@show get_summary(process_file(joinpath("src", "MyPkg.jl")))
The fraction of total coverage is equal to covered_lines/total_lines
.
To discover which functions lack testing, browse through the *.cov
files in your src/
directory and look for lines starting with -
or 0
- those lines were never executed.
Numbers larger than 0 are counts of the number of times the respective line was executed.
Start julia with
julia --track-allocation=user
Then:
- Run whatever commands you wish to test. This first run is to ensure that everything is compiled (because compilation allocates memory).
- Call
Profile.clear_malloc_data()
- Run your commands again
- Quit julia
Finally, navigate to the directory holding your source code. Start julia (without command-line flags), and analyze the results using
using Coverage
analyze_malloc(dirnames) # could be "." for the current directory, or "src", etc.
This will return a vector of MallocInfo
objects, specifying the number of bytes allocated, the file name, and the line number.
These are sorted in increasing order of allocation size.
There are many tools to work with LCOV info-format files as generated by the geninfo
tool. Coverage.jl can generate these files:
coverage = process_folder()
LCOV.writefile("coverage-lcov.info", coverage)
When using Coverage.jl locally, over time a lot of .cov
files can accumulate. Coverage.jl provides the clean_folder
and clean_file
methods to either clean up all .cov
files in a directory (and subdirectories) or only clean the .cov
files associated with a specific source file.
Tracking Coverage with Codecov.io
Codecov.io is a test coverage tracking tool that integrates with your continuous integration servers (e.g. TravisCI) or with HTTP POSTs from your very own computer at home.
-
Enable Codecov.io for your repository.
- If it is public on GitHub and you are using using Travis, CircleCI or Appveyor, this is all you need to do. You can sign into Codecov using your Github identity.
- Otherwise you will need to define a
CODECOV_TOKEN
environment variable with the Repository Upload Token available under the Codecov settings.
-
Use the command line option when you run your tests:
- Either with something like
julia --code-coverage test/runtests.jl
, or - with something like
julia -e 'Pkg.test("MyPkg", coverage=true)'
- Either with something like
-
Configure your CI service to upload coverage data:
-
If you are using Travis with
language: julia
, simply addcodecov: true
to your.travis.yml
. -
You can also add the following to the end of your
.travis.yml
. This line downloads this package, collects the per-file coverage data, then bundles it up and submits to Codecov. Coverage.jl assumes that the working directory is the package directory, so it changes to that first (so don't forget to replaceMyPkg
with your package's name! -
On Travis CI:
after_success: - julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Codecov.submit(process_folder())'
-
On AppVeyor:
after_test: - C:\projects\julia\bin\julia -e "using Pkg; Pkg.add(\"Coverage\"); using Coverage; Codecov.submit(process_folder())"
-
If you're running coverage on your own machine and want to upload results to Codecov, make a bash script like the following:
#!/bin/bash CODECOV_TOKEN=$YOUR_TOKEN_HERE julia -e 'using Pkg; using Coverage; Codecov.submit_local(process_folder())'
-
Tracking Coverage with Coveralls.io
Coveralls.io is a test coverage tracking tool that integrates with your continuous integration solution (e.g. TravisCI).
-
Enable Coveralls.io for your repository. If it is public on GitHub and you are using TravisCI, this is all you need to do. If you are using AppVeyor, you need to add a secure environment variable called
COVERALLS_TOKEN
to your.appveyor.yml
(see here). Your repo token can be found in your Coveralls repo settings. If neither of these are true, please submit an issue, and we can work on adding additional functionality for your use case. -
Activate the
--code-coverage
command line option when you run your tests- Either with something like
julia --code-coverage test/runtests.jl
, or - with something like
julia -e 'Pkg.test("MyPkg", coverage=true)'
- Either with something like
-
Configure your CI service to upload coverage data:
-
If you are using Travis with
language: julia
, simply addcoveralls: true
to your.travis.yml
. -
You can also add the following to the end of your
.travis.yml
. This line downloads this package, collects the per-file coverage data, then bundles it up and submits to Coveralls. Coverage.jl assumes that the working directory is the package directory, so it changes to that first (so don't forget to replaceMyPkg
with your package's name! -
On Travis CI:
after_success: - julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Coveralls.submit(process_folder())'
-
On AppVeyor:
after_test: - C:\projects\julia\bin\julia -e "using Pkg; Pkg.add(\"Coverage\"); using Coverage; Coveralls.submit(process_folder())"
-
Coverage tracking in Julia is not yet quite perfect. One problem is that (at least in certain scenarios), the coverage data emitted by Julia does not mark functions which are never called (and thus are not covered) as code. Thus, they end up being treated like comments, and are not counted as uncovered code, even though they clearly are. This can arbitrarily inflate coverage scores, and in the extreme case could even result in a project showing 100% coverage even though it contains not a single test.
To overcome this, Coverage.jl applies a workaround which ensures that all lines of code in all functions of your project are properly marked as "this is code". This resolves the problem of over reporting coverage.
Unfortunately, this workaround itself can have negative consequences, and lead to under reporting coverage, for the following reason: when Julia compiles code with inlining and optimizations, it can happen that some lines of Julia code do not correspond to any generated machine code; in that case, Julia's code coverage tracking will never mark these lines as executed, and also won't mark them as code. One may now argue whether this is a bug in itself or not, but that's how it is, and normally would be fine -- except that our workaround now does mark these lines as code, but code which now never has a chance as being marked as executed.
We may be able to improve our workaround to deal with this better in the
future (see also JuliaCI#188), but this
has not yet been done and it is unclear whether it will take care of all
instances. Even better would be if Julia improved the coverage information it
produces to be on par with what e.g. C compilers like GCC and clang produce.
Since it is unclear when or if any of these will happen, we have added an
expert option which allows Julia module owners to disable our workaround code,
by setting the environment variable DISABLE_AMEND_COVERAGE_FROM_SRC
to
yes
.
For Travis, this can be achieved by adding the following to .travis.yml
:
env:
global:
- DISABLE_AMEND_COVERAGE_FROM_SRC=yes
For AppVeyor, add this to .appveyor.yml
:
environment:
DISABLE_AMEND_COVERAGE_FROM_SRC: yes
Pull requests to add your package welcome (or open an issue)
- ArgParse.jl
- AstroLib.jl
- AudioIO.jl
- Augur.jl
- Bootstrap.jl
- CAIRS.jl
- ClimateTools.jl
- DASSL.jl
- DataFrames.jl
- Decimals.jl
- Distributions.jl
- DSP.jl
- ExtractMacro.jl
- FastaIO.jl
- FiniteStateMachine.jl
- FourierFlows.jl
- Gadfly.jl
- GeometricalPredicates.jl
- Glob.jl
- GradientBoost.jl
- GraphCentrality.jl
- GraphLayout.jl
- Homebrew.jl
- HttpParser.jl
- IntervalTrees.jl
- IPNets.jl
- JointMoments.jl
- JuMP.jl
- LibGit2.jl
- LightGraphs.jl
- LinearExpressions.jl
- Orchestra.jl
- ODE.jl
- OnlineStats.jl
- OpenCL.jl
- OpenStreetMap.jl
- PValueAdjust.jl
- QuantEcon.jl
- QuantileRegression.jl
- RationalSimplex.jl
- RDF.jl
- Requests.jl
- Restful.jl
- Robotics.jl
- RouletteWheels.jl
- SASLib.jl
- SimJulia.jl
- SIUnits.jl
- StatsBase.jl
- TaylorIntegration.jl
- TaylorSeries.jl
- TextWrap.jl
- TimeData.jl
- TypeCheck.jl
- Unitful.jl
- URIParser.jl
- URITemplate.jl
- Voting.jl
- WAV.jl
- Weave.jl
- WeightedStats.jl
- YAML.jl