gnu-octave / octave-doctest Goto Github PK

Thoughts on changing to ... before we get too popular to change? ;-)

... looks nice for shortening a long vector output. Maybe *** is more natural as "wildcard".

OTOH, from reading this, it looks like you have to enable with:

#doctest: +ELLIPSIS

The coloured output is not good enough to quickly see whether there are failures or not when there are many files to be checked.

Please add a summary line add the end. Useful information: total number of PASS´ed and FAIL´ed example blocks; total number of files without example blocks (see run_test_suite console output as a reference).

>> 1 + 2   % easy example
ans =  3

this seems to fail on Octave. I'll look into it further but wanted to document here for now.

We should have some test/ dir for tests to run. E.g., nolove.m from #42.

Eventually it would be nice to specify the NORMALIZEWHITESPACE default on the command line.

E.g.,
doctest . -recurse -NORMALIZEWHITESPACE

or
doctest . -recurse +NORMALIZEWHITESPACE

(Not necessary for 0.4.0.)

"Private directories not allowed in MATLAB path."

We should probably migrate those tests to the new test folder (#47).

Python uses the latter.

% doctest: +FOO -BAR + BAZ

By default, MATLAB formats numbers using 4 digits after the decimal point (and some special-casing for 0), while Octave uses 5 digits. Do we want to enforce a certain output format (either long or bank) so that doctests can be written in a robust way in both systems? Hmm...

This does not pass. I think it should.

function s = test_blank_match()
%
% >> a = 3 + 4;
% ***

Similarly to how we do @example? Or should I just bite the bullet and run one line at a time, checking the output as we go along...

• bits in @comment should not be tested (@c as well?), deal with in the extract code.
• consider allowing the `doctest: +SKIP`` tag to be placed inside an @comment.

sorry, didn't properly test the merge request I sent earlier! There is a trivial typo but even fixing that it gives errors like this:

 ??? Error using assignin Attempt to add "x" to a static workspace. See MATLAB Programming, Restrictions on Assigning to Variables for details.

I can fix this but not tonight.

Assuming my classdef code is ok, is there anything else anyone would like for 0.4.0?

I'd like to get this into octave-forge. We can do a 0.4.1 for anything that brings up.

nolove.m:

function s = nolove()
% >> s = nolove()
% s = <3

s = sprintf('<3');

Then doctest nolove:

octave:2> doctest nolove
Doctest v0.3.0-dev: this is Free Software without warranty, see source.
nolove: 1 ERRORS
  >> s = nolove()

     expected: s = <3
     got     : s = <3

Doctest Summary:

  Searched 1 targets: found 1 tests total, 0 targets without tests.
  Extraction errors: 0
  Tests passed: 0/1

I've traced this error to regexptranslate which escapes < and > but perhaps should not. Will test further later but quite possible upstream Octave bug.

'cuz you know its private.

In Symbolic, I have a @logical dir which adds a single method (isAlways) to the logical class.

• In Matlab, methods('logical') gives all sorts of stuff abs, all, amd, any, etc and including my isAlways method.
• In Octave, methods('logical') gives just isAlways.

Maybe this is an upstream Octave bug.

But even in the Matlab case, should doctest then call doctest on all of these methods? Even those not defined in this 'project'? This is good for a user, but for package maintainer, I want to doctest my local stuff...

Something wrong with the example_re regexp on Octave. Seems ok on Matlab.

Even fails its own tests (doctest doctest)

>> A = [1 2 3
.. 4 5 6
.. 7 8 9]
<output>

The regex matches three things instead of two, returning a cell array with the correct input but then then .. 7 8 9] where the output is expected and finally an extra match for the real output.

My adding error colors around the "got" does not play well with Octave's pager.

@catch22 Let's (eventually) pick one of these.

• we can be a Octave package that also generates a Matlab-appropriate zipfile for releases. Code could live in inst/, various other rules. PR #34.
• we can be a continue as a Matlab package (note this is not well-defined) and have some script that generates Octave packages. [going with first approach].

Also:

• In Octave, we will probably be "Octave-Forge Doctest Package" or something like that. [sort of done in DESCRIPTION file].
• The "official" software name is "Doctest For Matlab". License.txt refers to this for example. Is there any legal problem problem here? [no]
• Do we want a new name that is less confusing? [yes, the repository is now called octave-doctest.]
• If new name, do we keep the original License.txt and have a new COPYING? [No, just attribute to @tgs and keep copyright notices up-to-date.]
• Move DESCRIPTION file to root. 4f1bc79

I'm not exactly how reproducible this is...

If I enable the PAGER etc hack that is commented out, and generate some sort of error condition during the doctests, then I never get my prompt back. If I ctrl-C, I sometimes get:

KeyboardInterrupt

warning: connection to external pager lost (pid = 12783)
warning: flushing pending output (please wait)

Other times I get just get KeyboardInterrupt and my prompt back.

How to reproduce:
At least on my system:

1. start a new Octave instance, load the symbolic pkg
2. syms x (Symbolic spews some stdout and stderr at first start while connecting to Python. Leave this step out to reproduce.)
3. doctest symfun.

Without step 2, the stdout/stderr noise from symbolic startup makes the first doctest fail (this is understandable) but it also activates this bug:

octave:2> doctest('bernoulli', 'sym')
Doctest v0.3.0-dev: this is Free Software without warranty, see source.
==========================================================================
Start of temporary output (github.com/catch22/octave-doctest/issues/6)
==========================================================================
Python 2.7.9 (default, Apr 15 2015, 12:08:00) 
[GCC 5.0.0 20150319 (Red Hat 5.0.0-0.21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 



^C
KeyboardInterrupt
>>> 
octave:2> 

Currently (#85), these are subtly different, especially on Matlab (#87). Fix, or document.

doctest @class treats it as a pure directory.

The automatically insterted “>>” should be followed by a space if the subsequent string does not begin with a space. Otherwise the example block is not detected.

This happens within texinfo files, e.g.
@example
@group
1 + 2
@Result{} ans = 3
@EnD group
@EnD example

This looks like a pretty serious bug. Consider:

function a = doctest_bug(c)
% Here's an innocent-looking test:
%
%   >> display('hello world')
%   hello world
%
%
% All looks well, but try:
%
%   >> display('BUG BUG hello world Muhahahaha')
%   hello world
%
%
% Hmmm...

Both tests pass.

The problem is presumably in doctest_compare:

result = regexp(got, want_re, 'once')

Here want_re has no anchors to beginning and end. I relying on this thinking it was just smart enough to parse out "ans = ", but no its just easy to trick it :)

evalc is supposed to capture the stdout and not echo it. My fake_evalc fcn does the first but not the latter. I don't see any easy way around this without doing something icky with PAGER.

The right thing to do is probably to fix Octave.

But if someone knows a quick workaround for fake_evalc then let me know...

Should we release a version 0.3 with the old API? Let's not advertise it very much because of the incoming new API (issue #37, PR #38). @cbm755 would you like to do that?

which is still often the default

[On top of #85.]

doctest logical does not work properly.

At the top of a new-style classdef file you can document the class in a general way.

In #90 we added support for classdef classes. See the test/@test_classdef example but help test_classdef just says error: help: 'test_classdef' is not documented.

This is upstream Octave problem: https://savannah.gnu.org/bugs/?43047

We would like to be able to state things like that the doctest of test_angle_brackets is expected to fail for Octave versions <= 3.8.2. We could achieve this by some magic comment like

% >> s = test_angle_brackets()
% s = I <3 U
% XFAIL Octave<=3.8.2

but perhaps there is a simpler way?

• Remodel the API and functionality of doctest to follow http://octave.sourceforge.net/octave/function/test.html (PR #38).
• Add rundoctests or __run_doctest_suite__ in the spirit of http://octave.sourceforge.net/octave/function/runtests.html (PR #52).
• Use rundoctests in the Makefile.

Cf. previous discussions in #31.

It should be possible to use extensions .texinfo, .tex and .txi in addition to .texi.

I have suggested to try detection of Texinfo file by scanning the file for “\input texinfo”. This is not a good idea, because Texinfo @include files should also be supported.

strange "make test" passes. But interactively:

>> doctest({'doctest', 'test_class'})

doctest ............................................................... PASS    6/6   
test_class ............................................................ EXTRACTION ERROR

    Function or class not found.


Summary:

   PASS    6/6   

1/2 targets passed, 0 without tests, 1 with extraction errors.

I get warning: fopen: file found in load path since a little while. Is this because of the .texinfo files that get fopened from the path?

(I think this might get auto-fixed as we add support for test collection in directories...)

In #90 we add support for classdef classes. But not all methods are tested because of this upstream Octave bug: methods defined within the classdef block are not returned by methods test_classdef: see the example test/@test_classdef in our test set.

TODO: find/file upstream bug.

@oheim's bug 44390 is related.

Consider a test which outputs chars that are not words (in the regular expression sense):

>> '[{... + .  [  ] ...'
[{... + .  [  ] ...

Then the test will fail, probably because of the \w in example_re regexp in doctest_run.m.

Easy to trigger in Octave because of the way it renders cell-arrays.

Two questions for @oheim and @cbm755:

1. Currently we are invoking src/Makefile from the root Makefile and add src to the Octave path so that doctest_evalc is found. Is this the correct way to set up the testing infrastructure? Or should we install the package before running doctests on it? Or is there something like Python's setup.py develop?
2. Is it possible to have doctest_evalc be installed into inst/private rather than inst (the latter is the current behavior, right?).

See recently pushed test/test_warning.m

The octave packages of Debian and Ubuntu do not come with mkoctfile and hence are not always installed. Not sure if we should provide the old fake_evalc as a fallback (+ buffering support...).

Currently, we collapse all whitespace to single space chars before comparing. It might be better to:

1. strtrim each line. This was, result can be indented arbitrarily.
2. collapse/ignore blank lines.

Not urgent. Needs something like #67 for its tests.

Matlab and Octave have different conventions for formatting matrices so this might slightly touch on #35.

This is the appropriate wiki page: Development Process

• Explain self-testing.
• Explain how to publish a new release (octave-forge and tarballs).

Here there are 8 tests in total. to_test has size 21x1 and results has size 1x8. Instead of seeing which have tests and which have no tests, I get some random-looking numbers:

symfun: OK (64 tests)
@symfun/argnames: OK (115 tests)
@symfun/diff: OK (121 tests)
@symfun/formula: OK (109 tests)
@symfun/int: OK (102 tests)
@symfun/ldivide: OK (117 tests)
@symfun/minus: OK (110 tests)
@symfun/mldivide: OK (47 tests)
@symfun/mpower: OK (97 tests)
@symfun/mrdivide: OK (114 tests)
@symfun/mtimes: OK (103 tests)
@symfun/numel: OK (110 tests)
@symfun/plus: OK (97 tests)
@symfun/power: OK (109 tests)
@symfun/priv_disp_name: OK (101 tests)
@symfun/rdivide: OK (115 tests)
@symfun/subsref: OK (64 tests)
@symfun/symfun: OK (115 tests)
@symfun/symvar: OK (121 tests)
@symfun/times: OK (109 tests)
warning: implicit conversion from scalar to sq_string

I'm not sure how this is supposed to work but the error has something to do with the vectorized fprintf:

fprintf(out, '%s: OK (%d tests)\n', to_test.name, length(results));

note length(results) is a scalar (8).

Again, I'll try to look into this further when I get a chance.

I get lots of errors, although it seems to run the test:

>> doctest({'doctest'})
doctest ................................................ cat: invalid option -- 'X'
Try 'cat --help' for more information.
cat: invalid option -- 'X'
Try 'cat --help' for more information.
cat: invalid option -- 'X'
Try 'cat --help' for more information.
cat: invalid option -- 'X'
Try 'cat --help' for more information.
warning: connection to external pager lost (pid = 5300)
warning: flushing pending output (please wait)
PASS    6/6   

Summary:

   PASS    6/6   

1/1 targets passed, 0 without tests.

>> 

Or some command like that. We can use this for:

1. Our own tests were we want to test that a certain doctest does not pass.
2. For known failures that will eventually be fixed.

For now, I suggest just inverting the usual logic. That is, passing an XFAIL test is an error. This is slightly different from Octave test behaviour but its easier to implement and enough for 1. above.