Travis neovim builds timing out about vroom HOT 12 OPEN

@tarruda @fwalch
Any idea if anything changed in neovim?

I just updated to the latest python neovim client from pypi (0.0.24) and started seeing this error, which might be related:

Traceback (most recent call last):
  File "/usr/bin/vroom", line 7, in <module>
    sys.exit(vroom.__main__.main())
  File "/usr/lib/python3/dist-packages/vroom/__main__.py", line 52, in main
    writers.append(runner(f))
  File "/usr/lib/python3/dist-packages/vroom/runner.py", line 89, in __call__
    if not self.env.vim.Quit():
  File "/usr/lib/python3/dist-packages/vroom/neovim_mod.py", line 30, in Quit
    self.nvim.quit()
  File "/usr/local/lib/python3.4/dist-packages/neovim/api/nvim.py", line 184, in quit
    self.command(quit_command)
  File "/usr/local/lib/python3.4/dist-packages/neovim/api/nvim.py", line 99, in command
    return self._session.request('vim_command', string)
  File "/usr/local/lib/python3.4/dist-packages/neovim/api/common.py", line 220, in request
    return walk(self._in, self._session.request(name, *args), self, name,
  File "/usr/local/lib/python3.4/dist-packages/neovim/msgpack_rpc/session.py", line 73, in request
    err, rv = self._blocking_request(method, args)
ValueError: need more than 0 values to unpack

from vroom.

Yes, there have been some changes that can cause this behavior. First of all, now there are two kinds of asynchronous events Neovim can handle: deferred and non-deferred.

Deferred are events that need to be handled in Nvim main loop, through the K_EVENT virtual key. This is necessary for any event that calls vimscript or events that change editor state.

Non-deferred are events that can be handled directly by the internal event loop, these don't generate the K_EVENT key. Examples are API calls that only read editor state, or events that feed Nvim with user input.

Event though it seems there's no need to defer user input API calls(it is redundant because the whole point of K_EVENT is to touch vim as if the user typed something), the vim_feedkeys function is deferred because it relies on vim's typeahead buffer code which is very messy right now and can result in crashes if handled in the internal event loop. For emulating real user input, a new function has been added: vim_input, which is not deferred by pushing data to a lower level buffer.

Now, the recent change that may be causing this timeout: now deferred events can only be handled in some editor states that explicitly state they can accept K_EVENT, see neovim/neovim#1518 for details.

Since vroom still uses vim_feedkeys, it is possible that in one of the tests the editor enters a state that does not handle deferred events(one example is the "press ENTER to continue" state, or when a error is displayed to the user), which would result in the vim_feedkeys never returning. Another possibility is that vim.quit() is freezing because vroom calls it after a test that leaves the editor in one of those "bad states"

To fix the timeout, two changes have to be performed:

• vim_input must be used to feed input to Nvim, this will ensure vroom never blocks waiting for Nvim to enter a state it can accept deferred events.
• the vim_input calls must be synchronized with non-deferred API calls that verify editor state, because it is now possible that keys still weren't processed when a non-deferred API call returns. This can be achieved by calling vim_eval("1") whenever you want to wait for all input to be processed before proceeding.

from vroom.

@tarruda I don't think that is the case, the following script

import neovim
nvim = neovim.Nvim.from_session(neovim.socket_session('/tmp/nvim'))
nvim.quit()

fails with

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    nvim.quit()
  File "/home/raf/Documents/src/python-client/neovim/api/nvim.py", line 194, in quit
    self.command(quit_command)
  File "/home/raf/Documents/src/python-client/neovim/api/nvim.py", line 99, in command
    return self._session.request('vim_command', string)
  File "/home/raf/Documents/src/python-client/neovim/api/common.py", line 220, in request
    return walk(self._in, self._session.request(name, *args), self, name,
  File "/home/raf/Documents/src/python-client/neovim/msgpack_rpc/session.py", line 73, in request
    err, rv = self._blocking_request(method, args)
ValueError: need more than 0 values to unpack

from vroom.

So is neovim python-client outright broken right now, or is it specific to the way it's being used in vroom? Didn't see any related bug filed in neovim yet.

I just updated to check and it still repros with latest nvim, latest python-client.

from vroom.

I suspect it is broken - or at least the quit() method is, nvim actually quits, but then the client raises that exception for no reason. e.g. commenting out the quit messes the output a bit, but it sort of works better.

diff --git a/vroom/neovim_mod.py b/vroom/neovim_mod.py
index 1fc2f45..b575eef 100644
--- a/vroom/neovim_mod.py
+++ b/vroom/neovim_mod.py
@@ -27,7 +27,7 @@ class Communicator(VimCommunicator):
       # Never started
       return

-    self.nvim.quit()
+    #self.nvim.quit()

from vroom.

Actually the problem is with Neovim, I found it yesterday, a failed libuv assertion makes the process crash while exiting. I will try to push a fix today

from vroom.

Any progress here? I saw some changes in neovim, but didn't look like anything that affected the symptoms here.

from vroom.

I just confirmed it's still happening 2 months later. Hopefully we can get neovim mode working again soon. =)

from vroom.

I just confirmed it's still happening 2 months later. Hopefully we can get neovim mode working again soon. =)

Sorry about that. I've been working on a major change in the UI architecture thats taking some work to get right. The hardest part will be over when I merge neovim/neovim#1820, after that I will focus on this bug.

from vroom.

Cool, I see neovim/neovim#1820 was recently merged. Do you know if fixing this would be a major change still or a quick update?

I have the builds marked as allowed_failures. The main pain point was that it always takes 10 minutes for the neovim builds to time out before the status is reported, but I just found http://blog.travis-ci.com/2013-11-27-fast-finishing-builds/ so I have a workaround for that now. Of course, it would also be nice to be testing against neovim again.

On another front, is it possible for us to get the client to either error out immediately if the remote process dies or set a timeout (~30s) and error out if the client didn't respond?

from vroom.

@dbarnett The neovim crash should be gone by now, and I've just pushed neovim/pynvim@b9187e0 which should fix the nvim.quit() exception.

It seems the tests are hanging when nvim is in "Press Enter" states. As I said, when waiting for <cr>, nvim won't accept any other requests that can potentially execute vimscript. For example, if you send a command that results in the "press enter" being displayed, you cannot follow it with an eval request because it will only be processed after <cr> is sent. One of the tests that are hanging is messages.vroom, here's a little patch that will remove the hang from that particular test:

diff --git a/examples/messages.vroom b/examples/messages.vroom
index 3fe4d73..5940ce9 100644
--- a/examples/messages.vroom
+++ b/examples/messages.vroom
@@ -7,7 +7,7 @@ the command that causes them if you want it to work.
 This mechanism allows vroom to know what lines errors occur on. You can match
 multiple messages on one command.

-  :echomsg "First" | echomsg "Second"
+  :echomsg "First" | echomsg "Second"<cr>
   ~ First
   ~ Second

diff --git a/vroom/neovim_mod.py b/vroom/neovim_mod.py
index 1fc2f45..36a8952 100644
--- a/vroom/neovim_mod.py
+++ b/vroom/neovim_mod.py
@@ -53,8 +53,7 @@ class Communicator(VimCommunicator):
       extra_delay: Delay in excess of --delay
     """
     self.writer.Log(command)
-    parsed_command = self.nvim.replace_termcodes(command, True, True, True)
-    self.nvim.feedkeys(parsed_command)
+    self.nvim.input(command)
     self._cache = {}
     time.sleep(self.args.delay + extra_delay)

Note that I have added a <cr> to the end to ensure nvim is not left in the "press enter" screen, but I assume you don't want to modify vroom tests just to make them pass with neovim.

The only way to truly fix neovim_mod.py is to refactor it to use the new remote screen facility and parse screen updates according to your expectations. Unfortunately there's no documentation explaining on how to do it, but there is a complete implementation of a remote neovim UI in the python-client package: https://github.com/neovim/python-client/tree/master/neovim/ui

FWIW we also use this facility to test neovim with the lua client, here's an example test, and here is the screen class that implements an abstract screen for testing.

I wish I could do more for neovim_mod.py but unfortunately I have little time to spare.

from vroom.

FYI, setting cmdheight helps make these hangs less common: https://github.com/google/vim-codefmt/blob/af796cf4084a3d32b85313ccc82675d626d40b59/vroom/setupvroom.vim#L22

I would still love to have a native solution to this since hanging vroom is strange behavior and a pretty poor experience.

from vroom.