Currently, multiline strings are not shown properly in failure messages. For example,

Expect "foo\nbar\nbaz" == 'foo'

echos the following failure message:

# Expected "foo\nbar\nbaz" == 'foo'
#       Actual value: 'foo
bar
baz'
#     Expected value: 'foo'

And prove without --verbose drops lines which are invalid as TAP format. --verbose outputs too many lines, so that I usually use prove --comments to run tests. So that what I usually see is as follows:

# Expected "foo\nbar\nbaz" == 'foo'
#       Actual value: 'foo
#     Expected value: 'foo'

It's hard to guess what is wrong with messages like this. It's better to escape special characters including line feeds.

I ran into this problem today.

describe 'Increment'
  it '2000 times succeeds'
    let a = Z()
    for i in range(2000)
      let a = a.Increment()
    endfor
    " ...
  end
end

# ...
# Vim(function):E475: Invalid argument: 2000_20times_20succeeds
1..0

I haven't looked into it any further.

Hi,
thank you for writing a great plugin.

I tried naming a test directory 'test' instead of 't', but in this case, specs aren't found.
Is there a way to make this work?
(btw. I got everything working with a test dir 't', but I find that dir name too obscure).

Thanks for the help!

Im running a simple test:

describe 'vspec'
  it 'test'
    Expect 1 == 1 
  end
end

and getting the following error:

❯ rake test                                                                                                                                                                                                                                             vim-fortify/git/master
bundle exec vim-flavor test
-------- Preparing dependencies
Checking versions...
  Use kana/vim-vspec ... 1.6.1
Deploying plugins...
  kana/vim-vspec 1.6.1 ... skipped (already deployed)
Completed.
-------- Testing a Vim plugin
t/test.vim .. No subtests run

Test Summary Report
-------------------
t/test.vim (Wstat: 0 Tests: 0 Failed: 0)
  Parse errors: No plan found in TAP output
Files=1, Tests=0,  0 wallclock secs ( 0.02 usr +  0.01 sys =  0.03 CPU)
Result: FAIL
rake aborted!
Command failed with status (1): [bundle exec vim-flavor test...]
/Users/alvaro/Development/vim-test/Rakefile:10:in `block in <top (required)>'
Tasks: TOP => test
(See full trace by running task with --trace)

Im using:
ruby 2.2.3
vim-flavor 2.2.1
vim-vspec 1.6.1

Any ideas why Im getting this error?

Thanks!

i sometimes want to write some tests for common behaviour.
but i don't know how to write common behaviour using vspec.
rspec has `Shared Example Object', for writing that.

Shared example group
https://www.relishapp.com/rspec/rspec-core/v/2-0/docs/example-groups/shared-example-group

is vspec supporting that?
or, do you have a plan to support it?

Neovim was introduced several useful features (like job control) we can't test with vim -u... call.

So it might be worth to add the opportunity to choose a vim binary with an environment variable or an option?

We can simply reach this with changing bin/vspec file by only adding a generic vim binary name (all other options will be the same).

If an error happens outside :it, vspec shows the error as a diagnostic message like:

# function A..B..C, line 1
# Vim:E492: Not an editor command:   ThisLineIsNotAValidVimScriptStatement
1..0

This is just a diagnostic message. TAP harness does not treat it as a failure. Since a test script is not executed properly, a failure message should be generated instead.

• Replace cryptic a:i.expr_expected != '' and others with readable ones.
• ...

Derived from gh-14.

Adding to this, I think it should also be documented that the arguments are implicitly a list. Consider this test:

function! ToBeInteger(actual, expected)
  return a:actual == a:expected
endfunction

call vspec#customize_matcher('to_be_int', {'match': function('ToBeInteger')})

describe 'vspec default failure message'
  it 'has undocumented representation'
    let fortytwo = 42
    Expect fortytwo to_be_int 43
  end
end

This yields a message stating 42 does not equal [43], which may be misleading.

not ok 1 - vspec default failure message has undocumented representation
# Expected fortytwo to_be_int 43
#       Actual value: 42
#     Expected value: [43]

It seems to be hard to support arbitrary number of arguments for custom matchers. Limit the number of arguments to 0 or 1.

This affects all output generated by the vspec executable.

Given a file cr.vim:

describe 'vspec'
  it 'introduces carriage return characters'
    Expect 1 + 1 == 2
  end
end

Do the following in the shell:

$ cd /path/to/vspec
$ ./bin/vspec . /path/to/cr.vim > temp
$ vim -c 'e ++ff=unix temp'

You will see carriage return characters at the end of every line, pointing to a mistaken "CRLF" line ending.

It could be a Vim bug, I'm not sure.

Try this test. It wil fail:

describe 'the read command'
    it 'inserts text after the current line'
        Expect getline(1) ==# ''
        " read any file with text in the first line
        read ~/.vimrc
        Expect getline(1) ==# ''
    end
end

Vim normally leaves an empty line if you read a file. For some reason vim-vspec does not seem to do this.

This can be an inconvenience in larger test suites that use

1. fixtures (e.g. :tabedit fixtures/foo.rb), or
2. testing utility scripts (e.g. :source util/foo.vim).

In my opinion it would be nice to be able to use relative paths inside the test script, and run bin/vspec on the tests from anywhere. Currently such tests can only be run from a certain directory. (I don't use rake at the moment.)

I am not sure if this is by design. I would be curious to hear your thoughts on this.

For jedi-vim I'm implementing some kind of testing. Because of it's Python nature, I wanted to lower dependencies and came up with this py.test script: https://github.com/davidhalter/jedi-vim/blob/dev/conftest.py

The only important thing that it does is actually invoking vspec like this:

vspec-folder/bin/vspec vspec-folder . test/goto.vim

Where test/goto.vim is this file: https://github.com/davidhalter/jedi-vim/blob/dev/test/goto.vim

Testing autoload files works fine, but using plugins, doesn't. Therefore I had to add

source plugin/jedi.vim

That's of course possible, but a little bit annoying. Is this intentional? Or did I call vspec the wrong way? IMHO vspec should automatically source plugin files, without me sourcing them.

Thanks for your work! It really seems to be a nice project for testing vim scripts.

@kana, I find it is quite difficult to test exceptions with vspec.

Perhaps a new built-in matcher to_throw or similar could be useful?

Expect mylib#SomeFunction('bad_input') to_throw 'mylib error: bad input'

I believe it is currently not possible for users to implement this using a custom matcher. Instead the user has to write something like

it 'throws BadValue exception'
  for s in ['',
          \ 'a',
          \ '  134',
          \ '0x283a',
          \ '0723 ',
          \ '62.382',
          \ '-0.3',
          \ ]
    let thrown = 0
    try
      call bases#ParseNumber(s, 10)
    catch /ERROR(BadValue)/
      let thrown = 1
    endtry
    Expect thrown to_be_true
  endfor
end

This is cumbersome, or at least it could be more elegant, don't you think? Especially for APIs where exception conditions constitute a documented part of the API better support for exceptions would be useful.

Derived from gh-13:

Every {example} is used as a function name. But they are not used as is. Characters which are invalid as a function name are encoded. Thus, I think nothing will be changed by accepting double-quoted strings. Why did you think that it is problematic?

I know. I remember that I wrote this because I had found an issue – now I see that it is a different issue. It affects all {example} strings (single- and double-quoted).

Every string that has combining characters will break generate_example_function_name(). Two examples:

describe "combining characters in {example}"
  it "make vspec fail: 'a\u0300'"
    Expect 1 + 1 == 2
  end
end

# /private/var/folders/Sa/SafiUtyBG3Wmwja17AWRyE+++TI/-Tmp-/vZd4XYt/0, line 6
# Vim(function):E475: Invalid argument: make_20vspec_20fail_3a_20_27à_27
1..0

describe "combining characters in {example}"
  it "collide with different example '\u0935\u093e'"
    Expect 1 + 1 == 12345 " ok!
  end
  it "collide with different example '\u0935\u093f'"
    Expect 1 + 1 == 2
  end
end

ok 1 - combining characters in {example} collide with different example 'वा'
ok 2 - combining characters in {example} collide with different example 'वि'
1..2

I haven't investigated further. This is probably really outside the scope of vspec. I'm not sure it is worth fixing ...

I didn't find any clues, is this possible? Preferably rspec-style specification would be great.

:before/:after blocks are not in :try blocks. If an exception thrown from one of :before/:after blocks is not caught, it causes "Bail out". But it would be better to continue running further examples.

On the topic of the "match" Funcref in {matcher}, the help :h vspec#customize_matcher() only states:

A |Funcref| to determine whether {actual} value matches to {expected}
value. It takes 1 or more arguments.

The only examples of custom matchers in the help do not take any arguments. This leaves the syntax to be used undocumented.

• Is it :Expect x to_have_args [1, 2, 3], like for the built-in
  call()?
• Is it :Expect x to_have_args 1 2 3, like for :command, or in the
  shell?
• Is it :Expect x to_have_args 1, 2, 3? (Yes.)

Adding to this, I think it should also be documented that the arguments are implicitly a list. Consider this test:

function! ToBeInteger(actual, expected)
  return a:actual == a:expected
endfunction

call vspec#customize_matcher('to_be_int', {'match': function('ToBeInteger')})

describe 'vspec default failure message'
  it 'has undocumented representation'
    let fortytwo = 42
    Expect fortytwo to_be_int 43
  end
end

This yields a message stating 42 does not equal [43], which may be misleading.

not ok 1 - vspec default failure message has undocumented representation
# Expected fortytwo to_be_int 43
#       Actual value: 42
#     Expected value: [43]

When an expectation is failed, the default "Expected: XXX, Actual: YYY" message sometimes doesn't help much to understand the cause of the failure. So that it would be useful to output more messages about the failure by inspecting the current state, such as buffers, key mappings, and so on.

Currently, vim-vspec doesn't handle :it blocks with the same description properly. All but the last one will never be executed. Only the last one will be executed many times.
Though it's better to write a distinct description for each :it block, the current behavior is surprising for users, and it's annoying when writing one-time tests to check nontrivial behavior.

Test cases to check failure messages contains stack traces like b:foo..b:bar..b:baz, line 1. Anonymous functions are written as magic numbers like 42..777..23, line 1. And these magic numbers are not consistent. Whenever the internal of vim-vspec is changed, these magic numbers are also changed randomly.

It's a boring task to modify the test cases for the changes of the magic numbers. So that it's better to exclude anonymous function numbers from test cases.

Possible approaches:

• Ignore anonymous function numbers while comparing actual results and expected results.
• Use only named functions to get consistent results.

Currently it's not possible to use quickfix from vspec's output, because there is no assistive information to jump to failed examples.

In the solution for #15 a final filter sed 's/\r$//' was added in bin/vspec.

Unfortunately, by default sed buffers its output, so it is no longer possible to add another filter and see the output live.

vspec . t/test.vim | grep '# TODO'
# have to wait until whole test is finished!

Making sed output unbuffered (sed -u) or line-buffered (sed -l) fixes the problem. Unfortunately there are many different sed flavours and not all of them have these flags.

The reason this is a problem for me is that I use a small benchmark wrapper around vspec, and the benchmark tests I run with it can take quite long. For one benchmark, for example, I have to wait 20 seconds until the whole thing is finished and I can see the benchmark results.

Vspec usage is

Usage: /path/to/bin/vspec [{non-standard-runtimepath} ...] {input-script}

Problem 1: $ vspec test.vim

{non-standard-runtimepath} is optional (for example, {non-standard-runtimepath} is not required when a Vim distribution includes vspec in the system runtime files). But vspec adds an empty runtime path: rtp=,/opt/share/vim/vimfiles,/opt/share/vim/vim73,/opt/share/vim/vimfiles/after,/after.

&rtp should not contain empty elements. /after is not a proper runtime path.

Problem 2: $ vspec "./my dir" test.vim

Paths with spaces don't work with echo "${args[@]}" | sed.

In line https://github.com/kana/vim-vspec/blob/master/bin/vspec#L31

vim ... -e ...

we start load test script using -e ex-mode

Can we use

vim ... -E ....

Improve Ex-mode to do that? maybe it'll be better?

ref: What is the difference between Ex mode and improved Ex mode? http://vi.stackexchange.com/a/2695

Hi,

I am having trouble getting feedkeys to work. Not sure if it's an issue with the vspec plugin or if I am doing something wrong (My vimscript is pretty weak)

Here's an example :-

  it 'completes abcde'
    put! = 'abc'  
    call feedkeys("Ade\<esc>")
        sleep
    Expect getline(1) == 'abcde'
  end

Output :-

# Expected getline(1) == 'abcde'
#       Actual value: 'abc'
#     Expected value: 'abcde'

Tests about vspec#pretty_string are failed on Travis CI.

t/failure-messages-with-special-characters.t .. 1/?
not ok 1
# --- /dev/fd/62  2014-03-16 09:40:41.995094427 +0000
# +++ /dev/fd/61  2014-03-16 09:40:41.995094427 +0000
# @@ -20,6 +20,6 @@
#  #     Expected value: "foo\\\\bar"
#  not ok 6 - vspec shows strings with other special characters in a decodable style
#  # Expected "foo\x80\X7fbar" ==# 'foo\x80\X7fbar'
# -#       Actual value: "foo\x80\x7Fbar"
# +#       Actual value: "foo<80>\x7Fbar"
#  #     Expected value: "foo\\x80\\X7fbar"
#  1..6

But the tests are not failed on my machine. It might be caused by 'isprint' (which has a different default value based on the current platform) and/or 'encoding'. And I suspect both options have different values to the ones on my machine.

• Firstly, 0x80 is treated as printable on Travis CI. So that 'isprint' is configured to some value other than the default.
• Secondly, 'encoding' on Travis CI seems to be set to utf-8 or other Unicode encodings, because 0x80 is shown as <80> in the result.

• Remove the last path in the default 'runtimepath' (such as $HOME/.vim/after). It is unexpectedly included.
• Use an absolute path instead of . to avoid unexpected behavior with :cd in examples.

From gh-41:

In this case it was because of redraw being invoked in one of the tested functions. Can be easily reproduced by calling redraw in the spec directly.

Steps to reproduce

" t/redraw.vim
describe ':redraw'
  it 'does not affect TAP format (1)'
    redraw
  end

  it 'does not affect TAP format (2)'
    redraw
  end
end

./bin/vspec . t/redraw.vim

Expected result

ok 1 - :redraw does not affect TAP format (1)
ok 2 - :redraw does not affect TAP format (2)
1..2

Actual result

ok 1 - :redraw does not affect TAP format (1)ok 2 - :redraw does not affect TAP format (2)
1..2

These changes 1.5.0...revise-skip-todo-messages are actually problematic. prove treats text after # as a directive. If description contains # like describe 'autoload#function', prove treats function ... as a directive. So that :SKIP/:TODO messages are not correctly recognized by prove. Especially :TODO causes a failure of tests.

I am trying to write a test that uses an autoloaded variable. The variable is part of my API and I can use it everywhere without problems, but I can't seem to use it in vspec it ... end blocks.

path/to/bundles/autovar/autoload/autovar.vim:

function! autovar#Number()
  return 12
endfunction

let autovar#NUMBER = 12

path/to/bundles/autovar/t/test.vim:

describe 'autovar#Number'
  it 'returns 12'
    Expect autovar#Number() == 12
  end
end

describe 'autovar#NUMBER'
  it 'fails'
    Expect autovar#NUMBER == 12
  end
end

Run it:

$ vspec path/to/bundles/{vspec,autovar} t/test.vim
ok 1 - autovar#Number returns 12
not ok 2 - autovar#NUMBER fails
# function <SNR>1_main..vspec#test..6, line 1
# Vim(call):E121: Undefined variable: autovar#NUMBER
1..2

There is no way to customize failure messages for expectations with custom matchers. So that some failure messages are not natural and usable to debug.

# Expected 0 to_be_true
#       Actual value: 0
#     Expected value: []

[] is strange as an example of truthy value. It'd be better to omit expected value in this case.

Hello,

I had to reinvent assert_raise and friends in vspec for some test cases.

Would you be open to providing functions like those if I submit a PR? 🎅

Thanks for your consideration.

P.S. Thanks for the excellent tool. 🏆 I've already used it for two plugins. 😻

If I want to use e.g. nvim instead of vim I would currently have to add a vim script to the path that starts nvim.

Maybe something like VSPEC_VIM could be used from the environment?

After #33, "Bail out!" is generated for some exceptions even if they were thrown in :it blocks. "Bail out!" should not be generated because exceptions in :it blocks are fully under control of vim-vspec.

Is there a way to type text into a :call input("question") and hit <cr>?

Derived from gh-13.

:SKIP has the same problem, but that might be more difficult to solve, because of command! -bar.