Currently, if you use a glob to match a multiline message, only the first line is matched. That means that if you don't know how many lines of messages will be output, you have no way of expecting them.

We should either change the behavior of glob to be able to match across lines or add a mechanism for multiline globs.

We'd have to be careful about how greedy the glob matching is to avoid breaking vroom files that expect a multiline glob followed by more lines of output.

Using vroom's default message strictness of GUESS-ERRORS, vroom will often suppress useful error messages for developers while they're working on plugins (#55). We should probably just always include unexpected messages in the output for other types of errors (syscall expectations, buffer content checks, etc.).

Vroom has some hard coded logic that finds a display which vim can use (see Start() in vim.py). This is ugly, and we should generalize it to work in weird cases.

There is a test runner for vim named vroom.

Please consider renaming this project to something other than 'vroom'.

This is a little nicer than just tossing "[foo] is value by default" at the end of the doc paragraphs.

Any thoughts on this? Can't get any Vroom tests to run, the results are always along the lines of:

/Users/jonathan/.vim/bundle/github-issues.vim/test/gissues.vroom
ERROR on line 3: Vim quit unexpectedly, saying "E247: no registered server named "VROOM": Send expression failed."

Failed command on line 3:
> iHello, world!<ESC>
Ran 1 test in /Users/jonathan/.vim/bundle/github-issues.vim/test/gissues.vroom. 0 passing, 1 errored, 0 failed.

My gissues.vroom file is simply:

This is a vroom test.

  > iHello, world!<ESC>
  Hello, world!

My Vim version:

VIM - Vi IMproved 7.4 (2013 Aug 10, compiled May 30 2014 23:44:58)
MacOS X (unix) version
Included patches: 1-258
Compiled by Homebrew
Huge version with MacVim GUI.  Features included (+) or not (-):
+acl             +file_in_path    +mouse_sgr       +tag_binary
+arabic          +find_in_path    -mouse_sysmouse  +tag_old_static
+autocmd         +float           +mouse_urxvt     -tag_any_white
+balloon_eval    +folding         +mouse_xterm     +tcl
+browse          -footer          +multi_byte      +terminfo
++builtin_terms  +fork()          +multi_lang      +termresponse
+byte_offset     +fullscreen      -mzscheme        +textobjects
+cindent         -gettext         +netbeans_intg   +title
+clientserver    -hangul_input    +odbeditor       +toolbar
+clipboard       +iconv           +path_extra      +transparency
+cmdline_compl   +insert_expand   +perl            +user_commands
+cmdline_hist    +jumplist        +persistent_undo +vertsplit
+cmdline_info    +keymap          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con_gui  +lua             +rightleft       +windows
+diff            +menu            +ruby            +writebackup
+digraphs        +mksession       +scrollbind      -X11
+dnd             +modify_fname    +signs           -xfontset
-ebcdic          +mouse           +smartindent     +xim
+emacs_tags      +mouseshape      -sniff           -xsmp
+eval            +mouse_dec       +startuptime     -xterm_clipboard
+ex_extra        -mouse_gpm       +statusline      -xterm_save
+extra_search    -mouse_jsbterm   -sun_workshop    -xpm
+farsi           +mouse_netterm   +syntax          
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  system gvimrc file: "$VIM/gvimrc"
    user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
    system menu file: "$VIMRUNTIME/menu.vim"
  fall-back for $VIM: "/Applications/MacVim.app/Contents/Resources/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_MACVIM -Wall -Wno-unknown-pragmas -pipe  -DMACOS_X_UNIX  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1     -I/System/Library/Frameworks/Tcl.framework/Headers  -D_REENTRANT=1  -D_THREAD_SAFE=1  -D_DARWIN_C_SOURCE=1  
Linking: clang   -L. -L/usr/local/lib -L. -L/usr/local/lib -L/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -F/usr/local/Cellar/python/2.7.6_1/Frameworks -L/usr/local/lib -o Vim -framework Cocoa -framework Carbon       -lm  -lncurses -liconv -framework Cocoa  -L/usr/local/lib -llua -fstack-protector -L/usr/local/lib  -L/System/Library/Perl/5.16/darwin-thread-multi-2level/CORE -lperl -framework Python  -F/System/Library/Frameworks -framework Tcl -framework CoreFoundation -framework Ruby

No rush to look into this; just trying to make testing my github-issues plugin automatic as it's been sitting on my bucket list for a while.

Thanks for your time!

If a plugin only has an autoload/ directory and nothing else, then it's a safe bet that it's a @library plugin.

I just noticed that the public Google python guidelines recommend 4-space indent (http://google-styleguide.googlecode.com/svn/trunk/pyguide.html?showone=Indentation#Indentation). We currently have 2 space indents everywhere.

Any objections to batch-formatting the code to indent by 4 spaces instead of 2?

Using vim client-server support to drive vroom has brought several limitations and disadvantages: vroom is very slow and needs lots of arbitrary delays because we don't have a reliable way to tell when commands have finished executing (#2, #3), and vim client-server comes with its own gotchas (#8).

Instead, we could use expect/libexpect to drive a vim process and send it keys directly, as opposed to invoking a client vim process with --remote-send and --remote-expr.

We can use the expect script neovim uses for running API tests as a starting point. There, the API tests are only started when expect recognizes a special string. We could use the same mechanism to be notified when vim processes commands (e.g., call feedkeys(...) | echo 'vrooom-done').

Vim apparently overwrites blank messages with subsequent messages, which confuses vroom and seems to make it count blank messages strangely and have surprising expectations about which lines they will appear on.

Consider this vroom snippet, which runs without error:

  @messages (STRICT)
  :echomsg ''
  :echomsg ''
  ~ 
  :echomsg 'B'
  ~ 
  ~ B

The first :echomsg '' appends a blank line to :messages, but goes undetected. The next one has no effect (overwrites the blank line with another blank line), but for some reason registers as adding a blank line. The third line actually removes a blank line from :messages, overwriting it with "B", but for some reason registers as adding another blank line before the "B".

We should update vroom so it handles these cases as expected:

  @messages (STRICT)
  :echomsg ''
  ~ 
  :echomsg ''
  ~ 
  :echomsg 'B'
  ~ B

Note the slight ambiguity on the 2nd :echomsg ''. It technically has no effect on :messages, so we'd have to treat no change in that case as matching both an expectation for a blank message and no expectation.

The vroom files in the examples/ directory currently run with message strictness GUESS-ERRORS, and have some failures if you run them with message strictness STRICT.

We should clean up any strict-mode failures and enable strict message checking, both to avoid subtle bugs sneaking into vroom and to set a good example.

Testing the new initial neovim support, it seems to get confused by backslashes and terminate strings early.

Given a vroom file like

  :echomsg string(split("Foo\nbar"))
  ~ ['Foo', 'bar']

vroom --neovim will fail with E114: Missing quote: "Foo.

Looks like there are some issues with the code in the Communicate code that tries to escape a command and pass it to feedkeys.

Would be cool if we had a logo to include in the docs and the deb.

I'm assuming this would be some kind of a "V" with speed axes or tires spinning or something else speed-related, but could be something more creative if anyone has ideas.

The vroom script should be able to report its own version and exit when invoked with a --version argument.

Currently we list a version number in setup.py following semver.org guidelines, but if you don't know what version of vroom you're running, you have to do some investigating to find where the executable came from.

This is analogous to google/vimdoc#56.

Vroom is currently hard-coded to run the 'vim' executable. If you want to run vim from an alternate location, your only option is to edit the vroom code or monkey with $PATH.

I noticed that vroom doesn't detect tracebacks from embedded python as errorlike, and won't fail for them unless running with --message-strictness=STRICT.

For instance, an AssertionError like

Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError

is not considered an error message. (Although errors inside vim function calls get a message like "Error detected while processing function Foo:" and are always considered errors.)

This is probably a case that will come up here and there, so it might be worth handling it properly. Might also make sense to check behavior for other embedded languages like ruby, lua, and mzscheme, for folks who are into that sort of thing…

You should be able to --skip specific files or plugin test subdirs.

Theoretically, there should be a way to poll vim and know when it's ready. Unfortunately, it's not as easy as it sounds.

Quoth dbarnett@:

I experimented with polling "--remote-expr 1" to see when vim is initialized, but that seems to not quite take care of it: tests still fail if we don't wait the full 0.5 seconds on startup. It looks like there's still a gap between when vim responds to "--remote-expr 1" and when it's actually "ready".

Vroom seems to get confused and report old messages as the "received" message in "Expected message not received" errors.

For instance, given a vroom file like

   :echomsg 'Foo'
   ~ Foo

   :
   ~ Bar

Vroom will fail with the error

 FAILED on line 4: Expected message not received:
 "Bar" (verbatim mode)
 Message was "Foo"

Instead, it should say

 FAILED on line 4: Expected message not received:
 "Bar" (verbatim mode)
 There were no messages.

Note this doesn't seem to have any affect on vroom's checks themselves. Even though it says it received "Foo", an expectation for "Foo" will still fail.

I noticed when running vroom files with --message-strictness=STRICT that when first leaving insert mode in vim nocompatible mode, vim adds a blank line into the output of :messages 1.

Example:

  @messages (STRICT)
  :set nocompatible
  :echomsg 'A'
  ~ A
  % XXX
  :echomsg 'B'
  ~ B

FAILED on line 6: Unexpected message:


Messages:
A

B

Failed command on line 6:
:echomsg 'B'

Note that #42 affects subsequent messages and makes it even more confusing to find the right message expectations to make vroom pass.

It's possible to work around these in each affected vroom file, but that means if vim ever fixes this bug they'll start failing in STRICT message mode complaining the blank line isn't there.

Instead, vroom should be aware of this quirk and automatically ignore a blank message that appears after using insert mode. There are a few other ways to enter insert mode like > iFoo<ESC> :startinsert that we could detect if we were clever. Or we could just always treat blank lines as expected after any vroom line.

Neovim mode still doesn't handle special characters properly. I found that this code from https://github.com/google/maktaba/blob/master/examples/helloworld/vroom/mappings.vroom doesn't work properly:

  :execute 'norm' '<Leader>Hh'

It reports a "No mapping found" message even though the mapping should exist. I expect the angle braces are getting escaped but shouldn't be when they're inside a string literal.

I doubt this is the only bad corner case. We can maybe add a few more levels of cleverness to it, but before long we'll probably need some changes in neovim and/or some rethinking to get it fully correct.

It will be awesome to get integration test coverage for neovim in Travis CI. I tried enabling setting it up to run the examples dir in neovim mode, but ran into this error:

ERROR: Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/vroom/runner.py", line 70, in __call__
    self.env.vim.Start()
  File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/vroom/neovim_mod.py", line 44, in Start
    session = neovim.socket_session(self.args.servername)
  File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/neovim/msgpack_rpc/__init__.py", line 31, in socket_session
    return session('socket', path)
  File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/neovim/msgpack_rpc/__init__.py", line 17, in session
    loop = EventLoop(transport_type, *args, **kwargs)
  File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/neovim/msgpack_rpc/event_loop/base.py", line 87, in __init__
    getattr(self, '_connect_{0}'.format(transport_type))(*args)
  File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/neovim/msgpack_rpc/event_loop/asyncio.py", line 84, in _connect_socket
    self._loop.run_until_complete(coroutine)
  File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/trollius/base_events.py", line 281, in run_until_complete
    return future.result()
  File "/home/travis/virtualenv/python2.7.8/lib/python2.7/site-packages/trollius/futures.py", line 277, in result
    raise self._exception
FileNotFoundError: [Errno 2] No such file or directory

Just came across this project and really liked the idea of having a DSL designed specifically for writing tests.

I'm currently working on refactoring the way Neovim talks to user interfaces, and one of my goals was enable "sane" functional testing testing, similar to what vroom currently does. Some of the progress done in this area can be followed here: neovim/neovim#781.

The idea is that the test API will be implemented similarly to an UI(using a msgpack-rpc channel), but it will be much simpler make assertions about screen state such as syntax highlighting, window contents, etc.

I was planning to port all legacy tests(vim 'testdir' directory) to use this new testing infrastructure while also providing a way for plugin authors to test their own things. Maybe we can work together and design a standard way for testing Neovim(and possibly vim too)?

I read that one of vroom's issues is that you can't know when vim has processed commands, but this is not a problem when talking to Neovim via the msgpack-rpc API. I already have written some tests using the python client, you can see them here.

Much is still to be defined, and I would certainly consider your input before making any decisions

Vroom should define some assert functions. There's too much

:echomsg condition
~ 1

:echomsg condition
~ 0

I'm not sure if these should be functions defined by vroom, which would look like

:call Assert(condition)

(downside: can't use maktaba functions), or if we should just use maktaba#ensure#IsTrue.
(downside: verbose, requires maktaba dependency)

or if there should be builtin vroom syntax (maybe on the '?' character?), which would look like

? condition

(downside: vroom itself must be changed to add new asserts)

Suggestions welcome.

Running vroom --interactive on some vroom files with non-ASCII characters causes it to blow up with a UnicodeDecodeError and corrupt the buffer when it encounters a vroom failure (which triggers interactive mode).

For instance, running vroom on

  :echomsg 'X'
  :call len('¬')
  ~ 

works fine in non-interactive mode and prints a normal failure message, but in interactive mode it barfs a garbled version of this traceback

Ran 1 test in foo.vroom. Traceback (most recent call last):
  File "/usr/bin/vroom", line 7, in <module>
    sys.exit(vroom.__main__.main())
  File "/usr/lib/python2.7/dist-packages/vroom/__main__.py", line 52, in main
    writers.append(runner(f))
  File "/usr/lib/python2.7/dist-packages/vroom/runner.py", line 90, in __call__
    self.env.vim.Output(self.env.writer)
  File "/usr/lib/python2.7/dist-packages/vroom/vim.py", line 196, in Output
    self.Ask('VroomDie({})'.format(VimscriptString(buf.getvalue())))
  File "/usr/lib/python2.7/StringIO.py", line 271, in getvalue
    self.buf += ''.join(self.buflist)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 38: ordinal not in range(128)

The self.buflist in question is

['foo.vroom\n',
 'FAILED',
 ' on line 2',
 ': ',
 'Expected message not received:\n"" (verbatim mode)\n',
 '\n',
 u'Message was "X"\n',
 "\nFailed command on line 2:\n:call len('\xc2\xac')\n",
 '\n',
 'Last few commands (most recent last) were:\n',
 ":echomsg 'X'<CR>\n",
 ":call len('\xc2\xac')<CR>\n",
 '\n',
 '0 passing, ',
 '0 errored, ',
 '1 failed.\n']

The problem happens inside StringIO.getvalue and seems to be caused by mixing unicode and 8-bit strings. The StringIO docs say

The StringIO object can accept either Unicode or 8-bit strings, but mixing the two may take some care. If both are used, 8-bit strings that cannot be interpreted as 7-bit ASCII (that use the 8th bit) will cause a UnicodeError to be raised when getvalue() is called.

The code currently raises vroom Failures as Exceptions, with a lot of code to catch, collect, and continue and potentially really weird behavior if there are mistakes in the code.

We should restructure the code so Failures are just normal return values from functions instead of raised Exceptions.

Non-ASCII characters in vroom lines are making vroom --neovim fail with errors like UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 55-56: invalid continuation byte.

This seems to have broken when I started using vroom.DecodeHook in a3d8a38, and looks like a bug in the neovim projects: neovim/pynvim#52. It causes failures in maktaba/vroom/pluginsignals.vroom.

Some vroom tests can run potentially dangerous (or at least really annoying) shell calls with system strictness on. Vroom's shellfaker should never execute unexpected shell commands when system strictness is STRICT. It should fail early instead.

This might require making shell.vroomfaker a bit smarter and passing more information about expectations out to it.

Vroom neovim test configurations started timing out sometime between yesterday and today. A maktaba Travis run from Nov 21 failed normally (known issue), but a run from Nov 22 had errors.

Seems like a bad build might have made it into the neovim travis nightly builds or the latest python neovim client on pypi might have problems.

We should add a script to the repository that can be used to fetch the latest stable version of vroom. Plugin authors can use this in their Travis CI configuration to test against the latest stable vroom.

Once it's ready, we should update the recommended Travis CI configuration (#66) to recommend it instead of hard-coding the version number into the .travis.yml file.

I just noticed that the output of vroom --help doesn't show the default values. For instance, for --delay it lists:

  -d DELAY, --delay DELAY
                        Delay after each vim command (in seconds).

and doesn't mention that the default delay is 0.09.

We should add formatter_class=argparse.ArgumentDefaultsHelpFormatter to the ArgumentParser if we're not listing defaults explicitly in our help messages.

Vroom seems to blow up with encoding problems if it fails on a line with non-ASCII characters. For example, running a vroom file like

  :throw '¬'

causes vroom to fail with this error:

Traceback (most recent call last):
  File "/usr/bin/vroom", line 7, in <module>
    sys.exit(vroom.__main__.main())
  File "/usr/lib/python2.7/dist-packages/vroom/__main__.py", line 52, in main
    writers.append(runner(f))
  File "/usr/lib/python2.7/dist-packages/vroom/runner.py", line 75, in __call__
    self.Record(vroom.test.RESULT.FAILED, e)
  File "/usr/lib/python2.7/dist-packages/vroom/runner.py", line 111, in Record
    self.env.writer.actions.Log(result, lineno, error)
  File "/usr/lib/python2.7/dist-packages/vroom/output.py", line 324, in Log
    self._Error(result, error, lineno)
  File "/usr/lib/python2.7/dist-packages/vroom/output.py", line 393, in _Error
    self.Queue(str(error))
  File "/usr/lib/python2.7/dist-packages/vroom/test.py", line 96, in __str__
    '\n\n'.join(str(f) for f in flattened_failures))
  File "/usr/lib/python2.7/dist-packages/vroom/test.py", line 96, in <genexpr>
    '\n\n'.join(str(f) for f in flattened_failures))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xac' in position 53: ordinal not in range(128)

Vroom seems to make a bad assumption that failures come from a single, non-continued line. As a result, failures on a continued line only print the tail end of the actual vroom line responsible for the failure.

For example, this vroom file

   :call maktaba#error#Try(g:isfalse.WithArgs(
   |    2 + 2 == 4,
   |    'TWO AND TWO ARE %s',
   |    'FIVE'))
   ~ ERROR(Failure): ONE AND TWO ARE FIVE

results in an error like the following

FAILED on line 082: Multiple failures:
Suspected error message:
ERROR(Failure): TWO AND TWO ARE FIVE

Expected message not received:
"ERROR(Failure): ONE AND TWO ARE FIVE" (verbatim mode)

Messages:
ERROR(Failure): TWO AND TWO ARE FIVE

Failed command on line 082:
|    'FIVE'))

Sometimes vroom's vim process doesn't get properly initialized and vroom fails complaining about an "invalid expression".

An easy way to reproduce is to create a file echomsgs.vim with

echomsg 'Foo 1'
echomsg 'Foo 2'
echomsg 'Foo 3'

and so on, and then run vroom with

vroom -u echomsgs.vim sometest.vroom

which causes vim to get stuck at a Press ENTER or type command to continue prompt and never load the vroom init.

The problem here is that the commands to source the vroom init file are executed after the "-u" file. The vroom config should probably be the very first thing executed.

Vroom was written in python 3 style. It works okay when you run it using python 2, but has all kinds of problems handling non-ASCII input (#28, #59). After sinking hours and hours in vain trying to make the codebase seamlessly compatible between python 2 and python 3, I realized there's almost no advantage to supporting python 2.

Vroom is just an executable and doesn't need broad support for different python versions. Python 3 is widely available and should be able to run anywhere vroom needs to run (doesn't need to be the default python version). Could we just stop maintaining python 2 compatibility and change the code and setup.py to explicitly bomb out if you try to use it in python 2?

The alternatives are:

• Drop python 3 support.
• Try using 3to2 (doesn't look automatic or reliable).
• Keep trying to hack 2+3 compatibility and continue maintaining it.

And even if we did drop python 3 support, it would still be a pain fixing the encoding problems for python 2.

We already have $VROOMFILE. It might be nice to have $VROOMDIR for symmetry, as often the local directory is relevant to the test (eg, if there are sample plugin files in the same directory).

Vim plugins on github can get easy, free continuous integration running vroom in Travis CI.

Once we settle some of the details in our Travis CI configuration and get neovim mode working (#64), we should add a section to the README about how to configure Travis CI to test vim plugins using vroom, and different options you might want to use for your project.

Once neovim mode is cleaned up and ready for wider usage, we should document how to set it up and use it in the README.

See discussion in #62. Some of our Python code has doctests (which I suspect don't get run), but most of it is untested except at the integration level (by running vroom itself). We should have more Python-level unittests, and a simple way to run all the tests.

Apparently, even though "@System (RELAXED)" mode does make vroom not fail on unexpected system calls, the fact that there was an unexpected system call still hangs around and leads to failures when you return to STRICT mode.

For example, this vrooom file:

@system (RELAXED)
:call system('echo "Doesn''t fail"')
@system (STRICT)
:call system('echo "Or does it?"')
! echo "Or does it\?"

causes this error:

FAILED on line 4: Unexpected system call.
Failed command on line 4:
:call system('echo "Or does it?"')
Recent system logs are:
  RECEIVED (echo "Doesn't fail") >/tmp/vmI0QMy/0 2>&1
UNEXPECTED
  RECEIVED echo "Or does it?"
   MATCHED with "echo "Or does it\?"" (regex mode)
 RESPONDED echo "Or does it?"

We'll need to fix RELAXED mode so it flushes the state from unexpected syscalls properly.

@tarruda proposed in #29 that vroom syntax should support some kind of embedded python blocks (or python-based DSL) to enable more complex interactions and assertions. This could have a magic vim object defined to interact with vim state.

For example, here's how we could write python code that tests a plugin that highlights in red all capitalized words:

vim.push_keys('iSome word\nanother Word') # Insert some text
vim.command('InvokePlugin') # After this, all capitalized words should be highlighted in red
line1 = vim.screen.windows[0].lines[0] 
line2 = vim.screen.windows[0].lines[1]
assert line1.text == 'Some word'
assert line2.text == 'another Word'
assert line1.attributes[0]['type'] == 'Foreground'
assert line1.attributes[0]['value'] == 'Red'
assert deep_equal(line1.attributes[0].positions, [[0, 4]])
assert line2.attributes[0]['type'] == 'Foreground'
assert line2.attributes[0]['value'] == 'Red'
assert deep_equal(line2.attributes[0].positions, [[8, 12]])

The syntax could look something like:

This is a vroom test

  > iSome Word<esc>o
  > ianother Word<esc>
  > :InvokePlugin<cr>
  >>>

The '>>>' token escapes into a scripting DSL that provides direct access to the API.
We use it to perform more complex assertions. It is still possible to write vroom comments here
  line1 = screen.windows[0].lines[0]
  line2 = screen.windows[0].lines[1]
These statements are translated into python assertions
  line1.foreground should be 'red' from 0 to 4
  line2.foreground should be 'red' from 8 to 12 
The '<<<' token goes back
  <<< 

Neovim seems to run subshells as a separate invocation, causing system calls that match in vim mode to not match in neovim mode and vice versa.

For example, given

  :call system('(echo failed >&2 && exit 1) 2> /tmp/foo')

this expectation matches in vim mode:

  ! (echo failed >&2 && exit 1) 2> /tmp/* (glob)

and this expectation matches in neovim mode:

  ! echo failed >&2 && exit 1

This causes maktaba tests in vroom/system.vroom to fail.

Hi. I need to test various features of vim/neovim related to NUL (digraph: ^@) characters. I've tried this a couple different ways and all failed, one causing vroom itself to quit early.

  % ab^@cd
  ab^@cd

(The ^@ produced with Ctrl+k NU) This test might not be well-formed as no keyboard has a NUL key, but here's the output for running with --neovim:

FAILED on line 8: Expected "acd" in verbatim mode.

Found:
1   ab <<<<

Failed command on line 8:
acd

Ran 1 test in test/legacy/nul.vroom. 0 passing, 0 errored, 1 failed.

Expected "acd"? What happened to b? At least I understand the Found: line--vroom thinks NUL means termination.

Now, run with plain vim:

ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/vroom/runner.py", line 71, in __call__
    self.Run(lines)
  File "/usr/local/lib/python2.7/dist-packages/vroom/runner.py", line 162, in Run
    self.Test(self.env.buffer.Verify, line, **controls)
  File "/usr/local/lib/python2.7/dist-packages/vroom/runner.py", line 116, in Test
    self.ExecuteCommands()
  File "/usr/local/lib/python2.7/dist-packages/vroom/runner.py", line 56, in ExecuteCommands
    self._running_command.Execute()
  File "/usr/local/lib/python2.7/dist-packages/vroom/command.py", line 59, in Execute
    self.env.vim.Communicate(self.command, delay)
  File "/usr/local/lib/python2.7/dist-packages/vroom/vim.py", line 118, in Communicate
    '--remote-send', command])
  File "/usr/local/lib/python2.7/dist-packages/vroom/vim.py", line 260, in TryToSay
    env=env).communicate()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
TypeError: execve() arg 2 must contain only strings


Ran 1 test in test/legacy/nul.vroom. 0 passing, 1 errored, 0 failed.

Try 2:

  > i<C-k>NU<Esc>
  ^@

neovim:

FAILED on line 2: Expected "" in verbatim mode.

Found:
1   
 <<<<

Failed command on line 2:


Ran 1 test in test/legacy/nul.vroom. 0 passing, 0 errored, 1 failed.

vim:

FAILED on line 2: Expected "" in verbatim mode.

Found:
1    <<<<

Failed command on line 2:


Ran 1 test in test/legacy/nul.vroom. 0 passing, 0 errored, 1 failed.

Almost the same, but note that the <<<< marker for the neovim test displays on the next line.

On a lighter side, this test works with both vim and neovim.

  > i<C-k>SH<Esc>
  % �^A
  ��^A^A

Eventually, I'd like to use vroom to test system() and systemlist()'s support of NUL characters. Thank you.

It would be nice for vroom to automatically source setup.vim files adjacent to the vroom test (in the vroom/ directory) that allow you to do directory setup all in the same place.

For parsimony, this should be configurable inside of vroom files via a @setup directive (with setup.vim being the default if @setup is omitted).

This single problem leads to a host of other issues. Mostly, it forces us to use arbitrary delays between commands (instead of having some way of knowing when commands are finished). This slows down the tests considerably.

Furthermore, it means that we can't reliably know whether messages were dropped or whether we just haven't waited long enough yet. Which sucks.

If you have two vroom sessions running simultaneously, they conflict and cause failures in both.

Vroom should be able to detect the collision when it starts up and resume running with a different server name. It's actually not too hard: vim auto-assigns a number on collisions, e.g. VROOM, VROOM2, VROOM3, and you can check the servername with v:servername from inside the vim session. All we need to do is detect the actual servername on vim startup and use that for the duration of the test.

We could detect it fine-grained (test-by-test), or once per session when vroom first starts up. It shouldn't matter much either way, since as long as all running instances of vroom are polite and confirm their v:servername before trying to use it, there will never be collisions between them.

As noted in #25, Vroom currently fails when attempting to print a failure message containing non-ASCII content.

For example, running LANG=de_DE.UTF-8 vroom examples/directives.vroom currently fails because vim returns E605: Ausnahme nicht aufgefangen: I broke something! rather than the message we match against. That's simple to fix, but it suffices to demonstrate the problem:

$ LANG=de_DE.UTF-8 vroom examples/directives.vroom
Traceback (most recent call last):
  File "vroom", line 55, in <module>
    writers.append(runner(f))
  File "vroom/runner.py", line 75, in __call__
    self.Record(vroom.test.RESULT.FAILED, e)
  File "vroom/runner.py", line 111, in Record
    self.env.writer.actions.Log(result, lineno, error)
  File "vroom/output.py", line 324, in Log
    self._Error(result, error, lineno)
  File "vroom/output.py", line 416, in _Error
    QueueContext('messages', ErrorMessageContext)
  File "vroom/output.py", line 414, in QueueContext
    writer(value, self.Queue, *args)
  File "vroom/output.py", line 568, in WriteExtraData
    printer(str(datum))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 0: ordinal not in range(128)

In this case, the message that it's failing to print is actually the message-maintainer text, Übersetzt von Georg Dahn <[email protected]>, meaning that any test failure breaks vroom.

I've also seen a similar stacktrace in Failures.__str__() in the multiple-failures case.

I spent a short while looking at this yesterday. The core problem is that, from what I can see, in Python 2.x (which I guess we're assuming?) __str__() must return a byte string in an unspecified encoding, which in some cases (i.e. above) becomes ascii. (See this blog post for more detail.)

We're implementing __str__() to print a number of the objects (Failures and Log, for example), and I suspect these should implement __unicode__() instead, and probably leave __str__() effectively unimplemented. (This also means that we'd need to call unicode() instead of str(), and use u'%s' % x rather than '%s' % x.)

The other Failure subclasses already implement a useful __unicode__(), but they have the same problem if you attempt to call __str__(); we probably want to block that as well.

While attempting to reproduce #13, I noticed that the message output I'm getting on failure is different to what was quoted. Instead of e.g.

FAILED on line 4: Expected message not received:
"Bar" (verbatim mode)

Message was "Foo"

Failed command on line 4:
[...]

I'm getting

FAILED on line 4: Expected message not received:
"Bar" (verbatim mode)

Messages:

Messages maintainer: Mike Williams <[email protected]>
Foo

Failed command on line 4:

It turns out this is because I'm running with LANG=en_GB.UTF-8, which changes at least the maintainer (which is hardcoded in vroom/messages.py). Running as LANG=en_US vroom ... DTRT.

Should vroom be setting LANG=en_US (or LANG=C?) before running vim? It's not entirely clear to me whether hardcoding this would be unnecessarily restrictive, or whether it affects anything else.

Alternatively, we could conceivably work out what the "Messages maintainer" message looked like ourselves, perhaps by running vim and dumping the messages with no input (or could we just do that as the first thing unconditionally?).

Your options are currently kind of limited for debugging vim plugin code running under vroom. You can use --interactive mode and purposefully trigger failures to inspect vim state, and of course you can use print statement debugging. Using :breakadd func FuncName inside vroom files doesn't seem to work as expected, probably due to all the arbitrary delay times.

It would be great to have more convenient support for entering interactive debugging. This might mean detecting vim is in interactive debugging mode so that you could use :breakadd normally, and maybe a vroom directive to pause vim for troubleshooting (or maybe special-case :breakadd here in vroom files).

Vroom's default message strictness setting is GUESS-ERRORS, which uses heuristics to detect messages that look like errors (lines starting with E\d+, ERR, "Error detected while processing", etc). Instead, this mode should fail whenever it encounters a message echoed using the bright red ErrorMsg highlight group.

We haven't been able to do this so far because vimscript doesn't offer any way to programmatically query the highlighting for echoed messages, only the text, but using a libexpect-based runner (#41) might allow us to revisit this.