kana / vim-vspec Goto Github PK
View Code? Open in Web Editor NEWVim plugin: Testing framework for Vim script
Home Page: http://www.vim.org/scripts/script.php?script_id=3012
Vim plugin: Testing framework for Vim script
Home Page: http://www.vim.org/scripts/script.php?script_id=3012
These changes 1.5.0...revise-skip-todo-messages are actually problematic. prove
treats text after #
as a directive. If description contains #
like describe 'autoload#function'
, prove
treats function ...
as a directive. So that :SKIP
/:TODO
messages are not correctly recognized by prove
. Especially :TODO
causes a failure of tests.
From gh-41:
In this case it was because of
redraw
being invoked in one of the tested functions. Can be easily reproduced by callingredraw
in the spec directly.
" t/redraw.vim
describe ':redraw'
it 'does not affect TAP format (1)'
redraw
end
it 'does not affect TAP format (2)'
redraw
end
end
./bin/vspec . t/redraw.vim
ok 1 - :redraw does not affect TAP format (1)
ok 2 - :redraw does not affect TAP format (2)
1..2
ok 1 - :redraw does not affect TAP format (1)ok 2 - :redraw does not affect TAP format (2)
1..2
In the solution for #15 a final filter sed 's/\r$//'
was added in bin/vspec
.
Unfortunately, by default sed buffers its output, so it is no longer possible to add another filter and see the output live.
vspec . t/test.vim | grep '# TODO'
# have to wait until whole test is finished!
Making sed output unbuffered (sed -u
) or line-buffered (sed -l
) fixes the problem. Unfortunately there are many different sed flavours and not all of them have these flags.
The reason this is a problem for me is that I use a small benchmark wrapper around vspec
, and the benchmark tests I run with it can take quite long. For one benchmark, for example, I have to wait 20 seconds until the whole thing is finished and I can see the benchmark results.
It seems to be hard to support arbitrary number of arguments for custom matchers. Limit the number of arguments to 0 or 1.
This can be an inconvenience in larger test suites that use
:tabedit fixtures/foo.rb
), or:source util/foo.vim
).In my opinion it would be nice to be able to use relative paths inside the test script, and run bin/vspec
on the tests from anywhere. Currently such tests can only be run from a certain directory. (I don't use rake at the moment.)
I am not sure if this is by design. I would be curious to hear your thoughts on this.
:before
/:after
blocks are not in :try
blocks. If an exception thrown from one of :before
/:after
blocks is not caught, it causes "Bail out". But it would be better to continue running further examples.
# Expected 0 to_be_true
# Actual value: 0
# Expected value: []
[]
is strange as an example of truthy value. It'd be better to omit expected value in this case.
Hi,
I am having trouble getting feedkeys to work. Not sure if it's an issue with the vspec plugin or if I am doing something wrong (My vimscript is pretty weak)
Here's an example :-
it 'completes abcde'
put! = 'abc'
call feedkeys("Ade\<esc>")
sleep
Expect getline(1) == 'abcde'
end
Output :-
# Expected getline(1) == 'abcde'
# Actual value: 'abc'
# Expected value: 'abcde'
Derived from gh-14.
Adding to this, I think it should also be documented that the arguments are implicitly a list. Consider this test:
function! ToBeInteger(actual, expected) return a:actual == a:expected endfunction call vspec#customize_matcher('to_be_int', {'match': function('ToBeInteger')}) describe 'vspec default failure message' it 'has undocumented representation' let fortytwo = 42 Expect fortytwo to_be_int 43 end endThis yields a message stating
42
does not equal[43]
, which may be misleading.not ok 1 - vspec default failure message has undocumented representation # Expected fortytwo to_be_int 43 # Actual value: 42 # Expected value: [43]
Currently it's not possible to use quickfix from vspec's output, because there is no assistive information to jump to failed examples.
Is there a way to type text into a :call input("question")
and hit <cr>
?
'runtimepath'
(such as $HOME/.vim/after
). It is unexpectedly included..
to avoid unexpected behavior with :cd
in examples.Tests about vspec#pretty_string
are failed on Travis CI.
t/failure-messages-with-special-characters.t .. 1/? not ok 1 # --- /dev/fd/62 2014-03-16 09:40:41.995094427 +0000 # +++ /dev/fd/61 2014-03-16 09:40:41.995094427 +0000 # @@ -20,6 +20,6 @@ # # Expected value: "foo\\\\bar" # not ok 6 - vspec shows strings with other special characters in a decodable style # # Expected "foo\x80\X7fbar" ==# 'foo\x80\X7fbar' # -# Actual value: "foo\x80\x7Fbar" # +# Actual value: "foo<80>\x7Fbar" # # Expected value: "foo\\x80\\X7fbar" # 1..6
But the tests are not failed on my machine. It might be caused by 'isprint'
(which has a different default value based on the current platform) and/or 'encoding'
. And I suspect both options have different values to the ones on my machine.
'isprint'
is configured to some value other than the default.'encoding'
on Travis CI seems to be set to utf-8
or other Unicode encodings, because 0x80 is shown as <80>
in the result.I ran into this problem today.
describe 'Increment'
it '2000 times succeeds'
let a = Z()
for i in range(2000)
let a = a.Increment()
endfor
" ...
end
end
# ...
# Vim(function):E475: Invalid argument: 2000_20times_20succeeds
1..0
I haven't looked into it any further.
Neovim was introduced several useful features (like job control) we can't test with vim -u...
call.
So it might be worth to add the opportunity to choose a vim binary with an environment variable or an option?
We can simply reach this with changing bin/vspec file by only adding a generic vim binary name (all other options will be the same).
Im running a simple test:
describe 'vspec'
it 'test'
Expect 1 == 1
end
end
and getting the following error:
❯ rake test vim-fortify/git/master
bundle exec vim-flavor test
-------- Preparing dependencies
Checking versions...
Use kana/vim-vspec ... 1.6.1
Deploying plugins...
kana/vim-vspec 1.6.1 ... skipped (already deployed)
Completed.
-------- Testing a Vim plugin
t/test.vim .. No subtests run
Test Summary Report
-------------------
t/test.vim (Wstat: 0 Tests: 0 Failed: 0)
Parse errors: No plan found in TAP output
Files=1, Tests=0, 0 wallclock secs ( 0.02 usr + 0.01 sys = 0.03 CPU)
Result: FAIL
rake aborted!
Command failed with status (1): [bundle exec vim-flavor test...]
/Users/alvaro/Development/vim-test/Rakefile:10:in `block in <top (required)>'
Tasks: TOP => test
(See full trace by running task with --trace)
Im using:
ruby 2.2.3
vim-flavor 2.2.1
vim-vspec 1.6.1
Any ideas why Im getting this error?
Thanks!
There is no way to customize failure messages for expectations with custom matchers. So that some failure messages are not natural and usable to debug.
i sometimes want to write some tests for common behaviour.
but i don't know how to write common behaviour using vspec.
rspec has `Shared Example Object', for writing that.
Shared example group
https://www.relishapp.com/rspec/rspec-core/v/2-0/docs/example-groups/shared-example-group
is vspec supporting that?
or, do you have a plan to support it?
Test cases to check failure messages contains stack traces like b:foo..b:bar..b:baz, line 1
. Anonymous functions are written as magic numbers like 42..777..23, line 1
. And these magic numbers are not consistent. Whenever the internal of vim-vspec is changed, these magic numbers are also changed randomly.
It's a boring task to modify the test cases for the changes of the magic numbers. So that it's better to exclude anonymous function numbers from test cases.
Possible approaches:
Hi,
thank you for writing a great plugin.
I tried naming a test directory 'test' instead of 't', but in this case, specs aren't found.
Is there a way to make this work?
(btw. I got everything working with a test dir 't', but I find that dir name too obscure).
Thanks for the help!
Derived from gh-13:
Every
{example}
is used as a function name. But they are not used as is. Characters which are invalid as a function name are encoded. Thus, I think nothing will be changed by accepting double-quoted strings. Why did you think that it is problematic?I know. I remember that I wrote this because I had found an issue – now I see that it is a different issue. It affects all
{example}
strings (single- and double-quoted).Every string that has combining characters will break
generate_example_function_name()
. Two examples:describe "combining characters in {example}" it "make vspec fail: 'a\u0300'" Expect 1 + 1 == 2 end end# /private/var/folders/Sa/SafiUtyBG3Wmwja17AWRyE+++TI/-Tmp-/vZd4XYt/0, line 6 # Vim(function):E475: Invalid argument: make_20vspec_20fail_3a_20_27à_27 1..0
describe "combining characters in {example}" it "collide with different example '\u0935\u093e'" Expect 1 + 1 == 12345 " ok! end it "collide with different example '\u0935\u093f'" Expect 1 + 1 == 2 end endok 1 - combining characters in {example} collide with different example 'वा' ok 2 - combining characters in {example} collide with different example 'वि' 1..2
I haven't investigated further. This is probably really outside the scope of vspec. I'm not sure it is worth fixing ...
On the topic of the "match" Funcref in {matcher}
, the help :h vspec#customize_matcher()
only states:
A |Funcref| to determine whether {actual} value matches to {expected}
value. It takes 1 or more arguments.
The only examples of custom matchers in the help do not take any arguments. This leaves the syntax to be used undocumented.
:Expect x to_have_args [1, 2, 3]
, like for the built-incall()
?:Expect x to_have_args 1 2 3
, like for :command
, or in the:Expect x to_have_args 1, 2, 3
? (Yes.)Adding to this, I think it should also be documented that the arguments are implicitly a list. Consider this test:
function! ToBeInteger(actual, expected)
return a:actual == a:expected
endfunction
call vspec#customize_matcher('to_be_int', {'match': function('ToBeInteger')})
describe 'vspec default failure message'
it 'has undocumented representation'
let fortytwo = 42
Expect fortytwo to_be_int 43
end
end
This yields a message stating 42
does not equal [43]
, which may be misleading.
not ok 1 - vspec default failure message has undocumented representation
# Expected fortytwo to_be_int 43
# Actual value: 42
# Expected value: [43]
If an error happens outside :it
, vspec shows the error as a diagnostic message like:
# function A..B..C, line 1
# Vim:E492: Not an editor command: ThisLineIsNotAValidVimScriptStatement
1..0
This is just a diagnostic message. TAP harness does not treat it as a failure. Since a test script is not executed properly, a failure message should be generated instead.
After #33, "Bail out!" is generated for some exceptions even if they were thrown in :it
blocks. "Bail out!" should not be generated because exceptions in :it
blocks are fully under control of vim-vspec.
In line https://github.com/kana/vim-vspec/blob/master/bin/vspec#L31
vim ... -e ...
we start load test script using -e
ex-mode
Can we use
vim ... -E ....
Improve Ex-mode to do that? maybe it'll be better?
ref: What is the difference between Ex mode and improved Ex mode? http://vi.stackexchange.com/a/2695
When an expectation is failed, the default "Expected: XXX, Actual: YYY" message sometimes doesn't help much to understand the cause of the failure. So that it would be useful to output more messages about the failure by inspecting the current state, such as buffers, key mappings, and so on.
a:i.expr_expected != ''
and others with readable ones.I didn't find any clues, is this possible? Preferably rspec-style specification would be great.
I am trying to write a test that uses an autoloaded variable. The variable is part of my API and I can use it everywhere without problems, but I can't seem to use it in vspec it
... end
blocks.
path/to/bundles/autovar/autoload/autovar.vim
:
function! autovar#Number()
return 12
endfunction
let autovar#NUMBER = 12
path/to/bundles/autovar/t/test.vim
:
describe 'autovar#Number'
it 'returns 12'
Expect autovar#Number() == 12
end
end
describe 'autovar#NUMBER'
it 'fails'
Expect autovar#NUMBER == 12
end
end
Run it:
$ vspec path/to/bundles/{vspec,autovar} t/test.vim
ok 1 - autovar#Number returns 12
not ok 2 - autovar#NUMBER fails
# function <SNR>1_main..vspec#test..6, line 1
# Vim(call):E121: Undefined variable: autovar#NUMBER
1..2
Hello,
I had to reinvent assert_raise
and friends in vspec for some test cases.
Would you be open to providing functions like those if I submit a PR? 🎅
Thanks for your consideration.
P.S. Thanks for the excellent tool. 🏆 I've already used it for two plugins. 😻
@kana, I find it is quite difficult to test exceptions with vspec.
Perhaps a new built-in matcher to_throw
or similar could be useful?
Expect mylib#SomeFunction('bad_input') to_throw 'mylib error: bad input'
I believe it is currently not possible for users to implement this using a custom matcher. Instead the user has to write something like
it 'throws BadValue exception'
for s in ['',
\ 'a',
\ ' 134',
\ '0x283a',
\ '0723 ',
\ '62.382',
\ '-0.3',
\ ]
let thrown = 0
try
call bases#ParseNumber(s, 10)
catch /ERROR(BadValue)/
let thrown = 1
endtry
Expect thrown to_be_true
endfor
end
This is cumbersome, or at least it could be more elegant, don't you think? Especially for APIs where exception conditions constitute a documented part of the API better support for exceptions would be useful.
Vspec usage is
Usage: /path/to/bin/vspec [{non-standard-runtimepath} ...] {input-script}
Problem 1: $ vspec test.vim
{non-standard-runtimepath} is optional (for example, {non-standard-runtimepath} is not required when a Vim distribution includes vspec in the system runtime files). But vspec adds an empty runtime path: rtp=,/opt/share/vim/vimfiles,/opt/share/vim/vim73,/opt/share/vim/vimfiles/after,/after
.
&rtp
should not contain empty elements. /after
is not a proper runtime path.
Problem 2: $ vspec "./my dir" test.vim
Paths with spaces don't work with echo "${args[@]}" | sed
.
Derived from gh-13.
:SKIP
has the same problem, but that might be more difficult to solve, because ofcommand! -bar
.
This affects all output generated by the vspec executable.
Given a file cr.vim
:
describe 'vspec'
it 'introduces carriage return characters'
Expect 1 + 1 == 2
end
end
Do the following in the shell:
$ cd /path/to/vspec
$ ./bin/vspec . /path/to/cr.vim > temp
$ vim -c 'e ++ff=unix temp'
You will see carriage return characters at the end of every line, pointing to a mistaken "CRLF" line ending.
It could be a Vim bug, I'm not sure.
Currently, multiline strings are not shown properly in failure messages. For example,
Expect "foo\nbar\nbaz" == 'foo'
echos the following failure message:
# Expected "foo\nbar\nbaz" == 'foo'
# Actual value: 'foo
bar
baz'
# Expected value: 'foo'
And prove
without --verbose
drops lines which are invalid as TAP format. --verbose
outputs too many lines, so that I usually use prove --comments
to run tests. So that what I usually see is as follows:
# Expected "foo\nbar\nbaz" == 'foo'
# Actual value: 'foo
# Expected value: 'foo'
It's hard to guess what is wrong with messages like this. It's better to escape special characters including line feeds.
Currently, vim-vspec doesn't handle :it
blocks with the same description properly. All but the last one will never be executed. Only the last one will be executed many times.
Though it's better to write a distinct description for each :it
block, the current behavior is surprising for users, and it's annoying when writing one-time tests to check nontrivial behavior.
Try this test. It wil fail:
describe 'the read command'
it 'inserts text after the current line'
Expect getline(1) ==# ''
" read any file with text in the first line
read ~/.vimrc
Expect getline(1) ==# ''
end
end
Vim normally leaves an empty line if you read a file. For some reason vim-vspec does not seem to do this.
If I want to use e.g. nvim
instead of vim
I would currently have to add a vim
script to the path that starts nvim
.
Maybe something like VSPEC_VIM
could be used from the environment?
For jedi-vim I'm implementing some kind of testing. Because of it's Python nature, I wanted to lower dependencies and came up with this py.test
script: https://github.com/davidhalter/jedi-vim/blob/dev/conftest.py
The only important thing that it does is actually invoking vspec like this:
vspec-folder/bin/vspec vspec-folder . test/goto.vim
Where test/goto.vim
is this file: https://github.com/davidhalter/jedi-vim/blob/dev/test/goto.vim
Testing autoload files works fine, but using plugins, doesn't. Therefore I had to add
source plugin/jedi.vim
That's of course possible, but a little bit annoying. Is this intentional? Or did I call vspec
the wrong way? IMHO vspec
should automatically source plugin files, without me sourcing them.
Thanks for your work! It really seems to be a nice project for testing vim scripts.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.