chipsenkbeil / distant.nvim Goto Github PK

Currently, this plugin (and the binary itself) relies on having distant installed on the remote machine. A "nice to have" feature would be to support installing via scp by first downloading the appropriate binary from a list and then scp-ing it over.

:DistantInstall linux-x86_64-gnu example.com
:DistantInstall linux-x86_64-musl example.com
:DistantInstall mac-m1 example.com

Just pipe out to curl or something to download the binary to a temporary directory, then run scp in a terminal prompt like with launch, and then remove the copied file if successful.

The neovim LSP client builds up parameters to send that include a document URI. Unforunately, not all language servers know what to do with distant://. The ones that work default to file:// whereas others send back errors.

Instead of having a schema, we should instead use buffer-local autocmds for reading and writing, which is the only primary use of the schema.

Additionally, when opening a file using :e distant://... we should rename the buffer without the schema. This does mean that you cannot have two buffers open with the same local and remote path, but I think that's acceptable.

Lastly, when it comes to searching buffers for distant ones, we can just check for the var distant_remote_path.

Dependent on chipsenkbeil/distant#1 being integrated into distant-lua and neovim/neovim#16079 having some resolution to enable this plugin to be able to communicate as the PTY master.

For our main client, this is straightforward. For LSPs, we need distant to support reading initialization options that has a session before passing it along.

See chipsenkbeil/distant#29

While there is an option to switch to the async function for every synchronous task, writing a file is currently synchronous, but it does not have to be!

Should support a setting that specifies whether WriteBufCmd while use the synchronous api or asynchronous api. If using the async version, we need to keep track of when requests are made such that a later request doesn't get overwritten by an earlier request. While that may not be a problem for TCP (maybe), supporting UDP would yield problems.

First thought is to just keep a timestamp of last written. Regardless, can still happen if the server hasn't even received the first request before it got the second.

Alternative is to have 1) debouncing of writes and 2) queuing of writes. Debouncing avoids sending writes too quickly. Queuing means that we set up future client submissions, but still wait for the async job to finish before doing the next write.

There is a video on how to use sshfs+neovim to edit files on a remote machine: https://youtu.be/0fAygjWHpdM. How does distant.nvim differ from that?

If the directory is a symlink, we can have a problem where none of the files are triggered because they have the canonicalized file path whereas the root dir (and workspace folder dirs) keep whatever is provided in settings.

Because of that, we need to send a metadata request for each of these directories to set the appropriate path at time of client start.

This also would benefit from supporting batch requests, which starts by implementing chipsenkbeil/distant#20

I removed this originally because an LSP I use at work didn't like the scheme whereas other LSPs like rust-anaylzer just ignored it in favor of file, which worked well.

I don't want to rely on an LSP handling the scheme in a certain way, which is why having distant implement chipsenkbeil/distant#31 (lsp subcommand) to translate between file: and distant: would work wonders.

I think this would also support lir.nvim no longer clobbering the distant-dir directory. And it would make detecting distant buffers easier without relying on a specific buffer-local variable being set.

See chipsenkbeil/distant#72.

This would be a dependency bump for this plugin. Won't happen until the above issue is resolved.

Should address extra key requirements mentioned in #47.

Support on_attach like with LSP, but for when we first open a remote file or directory. Make this a setting that supports a function.

Right now, we've switched to always reload. Instead, it would be nice to have this be configurable and default to not always reloading since you can use :e to reload.

I made it further!

I am able to see the session info, my ip, port and key.

However, anytime I issue sync or async function calls, I see request timed out errors in internal/state.lua:322:
[distant] [ERROR 20:52:57] ...k/paqs/start/distant.nvim/lua/distant/internal/state.lua:322: Client failed (75): Request timed out

Presently working with this config:

local function setup_distant()
	local actions = require("distant.nav.actions")

	require("distant").setup({
		["<remote_instance>"] = {
			launch = {
				distant = "/home/<user>/.asdf/installs/rust/stable/bin/distant",
				username = "<user>",
				identity_file = "<my_priv_key>",
			},
		},
		["*"] = {
			max_timeout = 60000,
			timeout_interval = 200,
			client = {
				log_file = "~/tmp/distant-client.log",
				log_level = "trace",
			},
			launch = {
				log_file = "~/tmp/distant-launch.log",
				log_level = "trace",
			},
			file = {
				mappings = {
					["-"] = actions.up,
				},
			},
			dir = {
				mappings = {
					["<Return>"] = actions.edit,
					["-"] = actions.up,
					["K"] = actions.mkdir,
					["N"] = actions.newfile,
					["R"] = actions.rename,
					["D"] = actions.remove,
				},
			},
		},
	})
end

Looks like there's another check to see if an executable with the cmd path exists on the local machine, so we need to add a hack around that again.

We should have our tests in docker automatically delete the lsp binary so we can validate that it not existing is okay.

Something like :DistantLaunch example.com -u user, which I need to log into my remote machine. The shell command does this, so why not the wrapper? Or maybe I've missed something?

I need to verify, but pretty sure this doesn't work. If the file exists locally, neovim will open it.

To that end, the lsp client supports overriding handlers for any of the responses, which means we should be able to override to use editor.open to load the remote file and jump to it.

Following something like this: https://blog.kdheepak.com/loading-a-rust-library-as-a-lua-module-in-neovim.html

Idea would be to create a lua wrapper that provides a native distant client for lua to use to talk to a remote server instead of needing to pipe out to a binary.

Just leaving this in the backburner for now. Not planned whatsoever.

At the moment, the plugin makes a call to get metadata and if any error is received it continues forward as if the file is just missing.

1. We make this a batch request and send both exists and metadata so we get both back with exists indicating if the file is available
2. We tease out the error kid from the metadata error and see if it was a FileNotFound instance, otherwise anything else is treated as a real error

Currently, the only time that jobstart is used instead termopen for launching is when there is no UI available. This was done purely to support tests where there is no UI in headless mode.

For folks who have passwordless SSH or SSH handled by key (for me using a Yubikey + GPG), the flash of an empty termopen is not very pretty on the eyes. Should support a setting to explicitly use jobstart.

This is a bit tricky as this plugin needs a bit of setup:

1. The distant binary needs to be accessible on the machine running the tests
2. The distant binary needs to be accessible on a remote machine
3. The remote machine needs to have SSH accessible over some port
4. Credentials for SSH need to be automated, meaning passwordless SSH, to be able to run tests

With regard to the SSH credentials, this is the most demanding requirement, although the others are also annoying, especially with regard to ensuring that the plugin is operating and not using some local operation (in the case of SSHing into localhost). We want a different filesystem for the remote machine to ensure that everything that happens is remote.

My thoughts here are to stand up two docker containers, one that runs the tests and the other that hosts the server.

1. When building the containers, we need to copy SSH credentials from one container (client) to the other (server) to ensure passwordless access
2. Both containers need to have the distant binary available on path
3. Client would run tests headlessly, but I don't know if that affects the TermOpen command. With SSH being passwordless, it should run without getting stuck, but the pty bit may be a challenge.

For running on Github Actions, we can follow something like https://dev.to/mihinduranasinghe/using-docker-containers-in-jobs-github-actions-3eof

When I queried a large directory for its contents, it looks like some of the output was dropped from the beginning, resulting in broken json.

My guess is that the logic to gather stdout by line is flawed and needs to be revised.

It can be difficult to follow issues since this plugin is a bit complex. Rather than sticking in and removing print statements, it would be nice to support logging to some file if logging is enabled.

distant-lua has no problem managing multiple sessions. The majority of distant.nvim could also function against multiple sessions, but for ease of use maintains a single active session that is extracted from distant.state.session.

Easiest option would be to have state represent an active session, maintain a list of sessions within state that are mapped by id, and provide a mechanism to switch between sessions when desired. The switch would then affect all fn calls, although maybe we also want to provide the option for fn calls to take an optional session as an argument?

The session id should be embedded within each buffer so we don't have to figure out which session is applicable at that point. This also means that distant.nav.actions would extract the session from the buffer before acting upon it.

Would resolve #51 at the command level.

Today, you can't do DistantLaunch ssh.identity_files = ["file1" ,"file2"] or DistantLaunch ssh.identity_files = "file1,file2".

I think the more scalable option is to support lists directly in the parser as we won't need to have another converter by splitting a string into a list and would be able to support list items with commas.

Hello and thanks for this plugin!

Is it possible to pass a proxy command to DistantLaunch in order to connect first to a different server and then to the final destination?

similar to this: ssh -o ProxyCommand="ssh proxyserver -W %h:%p" username@finalserver

Thank you in advance!

This way, if launching via ssh isn't an option, a user could still start the server themselves and copy the information to paste into neovim.

Having this available also opens the doors for uses on top of it such as making an HTTP call to a web server that returns the credentials to connect to the distant server.

See neovim/neovim#15323

Plan is to try a workaround using before_init.

-- NOTE: root_dir is enforced as a directory on the local machine, but now that our
--       lsp instances are remote, there is no guarantee that the path is a directory
--       or exists at all. To that end, we must explicitly remove root_dir AND
--       the workspace_folders (if provided) and fill in '/' as the root_dir, swapping
--       back in the actual root dir and workspace folders during pre-init
--       in the form of `rootPath`, `rootUri`, and `workspaceFolders`
local function before_init(params, _config)
    params.rootPath = config.root_dir
    params.rootUri = vim.uri_from_fname(config.root_dir)

    if not config.workspace_folders then
        params.workspaceFolders = {{
            uri = vim.uri_from_fname(config.root_dir);
            name = string.format('%s', config.root_dir);
        }}
    else
        params.workspaceFolders = config.workspace_folders
    end

    if type(config.before_init) == 'function' then
        config.before_init(params, _config)
    end
end

-- Override the config's cmd, cmd_env, and capabilities
-- as we take those existing config fields and alter them to work on
-- a remote machine
return vim.lsp.start_client(u.merge(
    config,
    {
        before_init = before_init;
        cmd = cmd;
        cmd_env = cmd_env;
        capabilities = capabilities;

        -- Must zero these out to ensure that we pass validation
        -- TODO: Support Windows local machine
        root_dir = '/';
        workspace_folders = nil;
    }
))

Spent multiple days trying to figure this out and I can't find a solution that avoids pushing Docker images to a remote repository.

If anyone wants to help out on this one, I'd appreciate it.

Dependent on chipsenkbeil/distant#63. When we switched from the external client to the lua module, we lost the batching feature. Once it gets re-introduced in the lua module, we can tackle #16 and make other performance improvements.

Thanks for this useful plugin and your talk in VimConf.

I see that the distant binary has support for setting identity_files.

However, I can't figure out what to feed the :DistantLaunch command so that it parses the ssh.identity_files input into something that mlua can smoothly deserialize into a Vec<PathBuf>, e.g.

:DistantLaunch my.host.com ssh.identity_files=["~/.ssh/id_rsa_distant"]

Unfortunately, I don't think we can use nvim-lspconfig out of the box as each language server needs to define a root dir, which is making use of util.root_pattern to figure out the root of a project. Since that involves globbing on the local filesystem, we can't use that setting.

Instead, we'll have to support providing a naive LSP settings option during setup. The plus side is that this enables us to separate networked LSP servers from local LSP servers.

require('distant').setup {
    -- ...
    lsp = {
        -- Is this the filetype? Or just a label? If it's a label, we need a filetype key/value
        rust = {
            -- Gets translated into `distant action proc-run -- rust-analyzer`
            cmd = 'rust-analyzer',
            root_dir = '...',
        }
    }
}

It can get really, really expensive to do a bunch of network calls to look around for the root. Instead, this may be something we want to support as another action on the server. Probably something like find in terms of action that can be configured with support for a direction and regex to select a match.

• Introduction
• Installation
• Settings
• Commands
• Lua functions under distant.fn
• Lua functions under distant.nav.actions
• Changelog (#37)
• Credits
• License

• Verify that LSP works
  • Make sure that the client doesn't die randomly (more on distant than this plugin)
  • Verify that navigation such as gd properly load the remote file that we are navigating to
  • Verify that refactorings like rename properly apply to remote files and not local
• Observe how responds when client dies
• Observe how responds when LSP proxy dies
• Write fn unit tests
• Write fn e2e tests

When I was trying to test out the LSP client with a typescript server or gopls prior to the vimconf.live talk, I had the client die with an invalid message error. I'm not sure if this is due to a change in neovim's LSP client (one machine used 0.5.0, another used 0.5.1) or if something broke with the switch from the distant CLI to a shared library.

Our test succeeds with a shared library, but this is with rust analyzer as the language server.

Could use extra help to investigate if this is a fix that needs to happen on the shared library side or the lua plugin side.

Currently, if you're in a directory, and use the distant.nav.action.newfile command, you jump to the new file. If you navigate up from there back into the directory, the file is not shown as we are not reloading the directory by default. When creating a new file, we should also update the buffer of the directory it is within.

Hey really cool plugin.
Got this error when trying to connect to remote with :DistantLaunch my.host mode=ssh ssh.user=user ssh.port=11022
Although if I set ssh = { port = 11022 } it in my config it works fine

callback error
stack traceback:
^I[C]: in ?
^I[C]: in function 'pcall'
^I.../.vim/plugged/distant.nvim/lua/distant/editor/launch.lua:75: in function 'load'
^I.../.vim/plugged/distant.nvim/lua/distant/editor/launch.lua:66: in function 'launch'
^I.../mater/.vim/plugged/distant.nvim/lua/distant/command.lua:147: in function 'launch'
^I[string ":lua"]:1: in main chunk
caused by: deserialize error: invalid type: string "11022", expected u16
callback error
stack traceback:
^I[C]: in ?
^I[C]: in function 'pcall'
^I.../.vim/plugged/distant.nvim/lua/distant/editor/launch.lua:75: in function 'load'
^I.../.vim/plugged/distant.nvim/lua/distant/editor/launch.lua:66: in function 'launch'
^I.../mater/.vim/plugged/distant.nvim/lua/distant/command.lua:147: in function 'launch'
^I[string ":lua"]:1: in main chunk
caused by: deserialize error: invalid type: string "11022", expected u16

Ps: Distant ask me passphase to decrypt id_ed25519 and id_rsa for user@host everytime I try to connect to my remote server. Since I don't have any ssh key passphase I just hit enter to skip it but it kinda annoying. I'm not sure what it is and how to stop Distant from asking. Usually I just enter my user's password when ssh over the command line.

When I try to start distant with the command ":DistantLaunch 127.0.0.1", I got this error:

The plugin ask successfully for the password, but crashes just after.
I'm using Ubuntu 20.04.3 LTS and have installed distant.nvim with the option "1. Download a prebuilt lib" and choosed the version "1. v0.15.0".

With the Lua language server I use, it has a custom Lua doc called EmmyLua similar to LDoc.

I've been writing docs that mostly apply, but it would be annoying to repeat these in a vim doc. Turns out, there's already a treesitter implementation that reads Lua documentation and creates a vim doc for it.

See telescope.nvim's gendocs.lua, telescope.nvim's github action, and the treesitter dependency.

Additionally, here is a guide specifically geared towards EmmyLua for doc generation.

Rather than taking a string, should make it take a table and convert into an argument string that we wrap in doublequotes.

We know what the minimum supported lua module is, so we should be able to filter the available versions to only those that are supported.

Need to parse v{MAJOR}.{MINOR}.{PATCH}[-{PRE-RELEASE}[.{PRE-VERSION}]] and use a constant like we did before to evaluate.

Due to problems with ssh2-rs and not being able to upgrade wezterm-ssh (see #48) right now, we need another way to execute DistantLaunch. Should do the old process of termopen(...) with an ssh command, reading the result to look for the DISTANT CONNECT -- <PORT> <KEY> and then running DistantConnect programmatically.

Will require DistantConnect to support supplying the key programmatically. Probably shouldn't expose this in the command but instead allow it as part of editor.connect(...).

While the end goal is to support plugins like lir.nvim, at its core this plugin needs to support a barebones remote file navigator out of the box.

Hi, I have installed distant.nvim with Plug, and even though the binary works, distant.nvim seems not able to locate the dll:

:DistantSystemInfo
Rust library not found! What would you like to do?
1. Download a prebuilt lib
2. Build from source
3. Copy local lib
Type number and <Enter> or click with the mouse (q or empty cancels):

I have tried 1 and 3 with no success. I have downloaded distant_lua-win64.dll, renamed it to distant_lua.dll, and put it in the binary location, as well as in the {plugins_folder}/distant.nvim/lua/.

It would be helpful to reflect changes in a changelog going forward.

Unable to use it because :DistantRun echo hello does not translate into a list of args for the second argument of distant.fn.run.

Instead, we need a wrapper function like

function distant_run(cmd, ...)
    return require('distant').fn.run(cmd, {...})
end

Currently, you have to provide a specific label with its own settings. There are common features such as on_attach that we want to support to avoid duplicating the lsp settings for each one individually.

require('distant').setup {
    ['example.com'] = {
        lsp = {
            ['Some Project'] = { cmd = '...', root_dir = '...' },
            ['*'] => { on_attach = ... },
        }
    }
}

Versus it being an exact match, it would be nice if we could do something like

require('distant').setup {
    -- Any subdomain of example.com including example.com
    ['*example.com'] = { ... },
}

Challenge here is about order of setting merging. Currently, we have exactly two situations: * and an explicit hostname.

We could say that settings are arbitrarily merged, except we want to guarantee that * is always overridden by anything else, so that would be a special case.

To support automatic interfacing with other plugins that make use of the local filesystem and processes, we could potentially create a function that temporarily replaces all APIs that would do local filesystem and process execution with mirrors that perform distant operations.

I'm not sure how feasibility this is to do and it wouldn't work for vimscript where we cannot replace calls to system() or job control. We can only replace vim.fn.system and vim.fn.jobstart.

List of functions to overwrite

Used by nvim-lspconfig

• uv.os_homedir -> session.system_info
• uv.os_uname -> session.system_info
• uv.fs_stat -> session.metadata
• uv.fs_realpath -> session.metadata({ canonicalize = true })
• vim.fn.jobstart -> session.spawn
• vim.fn.jobwait -> proc.wait

Tracker for support in other ways: neovim/nvim-lspconfig#1317

Used by toggleterm.nvim

• vim.fn.termopen -> session.spawn with pty = true

To support passing back errors more clearly

This might be due to expansion, but I think this also applies to the fn.spawn_wait and others as well. The environment variable gets translated on the local machine before the command is sent remotely.

Currently, when quitting neovim with a client running, it will shut down and then still report an exit code. If we know that we are shutting down and have issued the request, we should suppress output like that.