The VSCode extension fails to start the language server. Visual Stud

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Hi, with esbonio entries in settings.json : <div c

`ModuleNotFoundError` upon startup of VSCode extension,about swyddfa/esbonio

Comments (28)

lextm commented on September 15, 2024 1

I think the first run experience can be very difficult to improve, but there needs to be something to help make Python: Select Interpreter and Esbonio: Restart Language Server easier to access than searching in command palette. Currently I think a status bar icon might be the best entry.

from esbonio.

ExaneServerTeam commented on September 15, 2024 1

@alcarney regarding the separator, this PR should do the job: #719

from esbonio.

ExaneServerTeam commented on September 15, 2024 1

I tried to investigate a bit.

In code/src/node/client.ts

Line 168

    // Isolate the Python interpreter from the user's environment - we brought our own.                                                                                                                      │
    command.push(...pythonCommand, "-S")

No site-package is loaded by default, I am fine with that

line 203

    let serverEnv: ExecutableOptions = {                                                                                                                                                                         │
      env: {                                                                                                                                                                                                     
        PATH: process.env.PATH,                                                                                                                                                                                  
        PYTHONPATH: join(this.context.extensionPath, "bundled", "libs")                                                                                                                                          
      }                                                                                                                                                                                                          
    }

Execution variable are very restricted here.
It would be nice if APPDATA could be transfered too here when running on Windows
https://nodejs.org/api/os.html#ostype

It looks like python is using/require this var APPDATA to expand the user site-packages:
https://github.com/python/cpython/blob/3.12/Lib/site.py#L277
Fallback is run by expanding "~"
https://github.com/python/cpython/blob/3.12/Lib/ntpath.py#L350

I confirm that enforcing APPDATA in the python code make the
sphinx agent spawning work.

In client_subprocess.py

    async def start(self, config: SphinxConfig):
        """Start the client."""
        command = []
        if config.enable_dev_tools:
            command.extend([sys.executable, "-m", "lsp_devtools", "agent", "--"])

        command.extend([*config.python_command, "-m", "sphinx_agent"])
        env = get_sphinx_env(config)
        env['APPDATA'] = "C:\\Users\\xxxxx\\AppData\\Roaming"

        self.logger.debug("Sphinx agent env: %s", json.dumps(env, indent=2))
        self.logger.debug("Starting sphinx agent: %s", " ".join(command))

        await self.start_io(*command, env=env, cwd=config.cwd)

from esbonio.

alcarney commented on September 15, 2024 1

@ExaneServerTeam pre-release v0.92.1 of the extension is now available. This isn't the first issue I've had with missing environment variable so I've changed the approach to pass everything through - except for the ones the extension/server is explicitly modifying.
In theory at least, you shouldn't have to mess with the pythonPath configuration option anymore either 🤞, let me know how it goes!

from esbonio.

ExaneServerTeam commented on September 15, 2024 1

@alcarney: I confirm the behavior:

Edit a .rst file & save
I see the sphinx build launched in the Esbonio Debug output
Spawn the renderer -> It works

-> I'll be away for 10days, I will probably be in mute within that time

from esbonio.

alcarney commented on September 15, 2024

Have you tried setting esbonio.server.enableDevTools to false? If you're only looking to try esbonio itself, you shouldn't need to have this option enabled.

The esbonio.server.enableDevTools option is mostly for me at the moment, where I am experimenting with integrating lsp-devtools with the VSCode extension to help with the development of the server.

While the server and it's dependencies are bundled with the extension itself, it looks like I didn't ship v0.91.0 with all dependencies required for lsp-devtools (and I'm not likely to change that, since I've noticed it makes the extension download much bigger).

If you are trying to play with the lsp-devtools integration you're probably better off installing the package with pipx and trying the version of the VSCode extension on the current develop branch, as that now looks for lsp-devtools installed on your PATH instead.

Hope that helps! :)

from esbonio.

rpfeifer-tcr commented on September 15, 2024

Hi, with esbonio entries in settings.json:

    "esbonio.server.logLevel": "debug",
    "esbonio.server.enableDevTools": false

I get the following import errors.

[client] Using environment /bin/python: /bin/python
[client] Server start command: /bin/python -S -m esbonio.server
[client] LanguageClientOptions: {
  "documentSelector": [
    {
      "scheme": "file",
      "language": "restructuredtext"
    },
    {
      "scheme": "file",
      "language": "python"
    }
  ],
  "initializationOptions": {
    "server": {
      "logLevel": "debug",
      "logFilter": [],
      "showDeprecationWarnings": false,
      "completion": {
        "preferredInsertBehavior": "replace"
      }
    }
  },
  "outputChannel": {
    "name": "Esbonio"
  },
  "connectionOptions": {
    "maxRestartCount": 0
  },
  "middleware": {
    "workspace": {}
  }
}
[client] Starting Language Server
[esbonio] Loaded extension 'esbonio.server.features.sphinx_manager'
[esbonio] Loaded extension 'esbonio.server.features.preview_manager'
[esbonio] Loaded extension 'esbonio.server.features.symbols'
[esbonio] Initialising esbonio v0.16.1
[esbonio] Language client: Visual Studio Code 1.85.1
[esbonio] workspace/configuration: {
  "items": [
    {
      "scopeUri": "file:///home/rpfeifer/repos/tpl-gw/doc/source/systemvarianten/systemvarianten.rst",
      "section": "esbonio.sphinx"
    }
  ]
}
[client] Using environment /bin/python: /bin/python
[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "/bin/python"
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": []
}
[esbonio.SphinxManager] Trying path: /home/rpfeifer/repos/tpl-gw/doc/source/systemvarianten
[esbonio.SphinxManager] Trying path: /home/rpfeifer/repos/tpl-gw/doc/source
[esbonio.SphinxManager] Cwd: /home/rpfeifer/repos/tpl-gw
Exception in callback JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)
handle: <Handle JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)>
Traceback (most recent call last):
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/features/sphinx_manager/manager.py", line 180, in _create_client
    sphinx_info = await client.create_application(resolved)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/features/sphinx_manager/client_subprocess.py", line 170, in create_application
    sphinx_info = await self.protocol.send_request_async("sphinx/createApp", params)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/pygls/feature_manager.py", line 81, in wrapped
    return await f(server, *args, **kwargs)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/setup.py", line 61, in on_document_open
    await call_features(ls, "document_open", params)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/setup.py", line 91, in call_features
    await result
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/features/sphinx_manager/manager.py", line 68, in document_open
    await self.get_client(uri)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/features/sphinx_manager/manager.py", line 163, in get_client
    return await self._client_creating
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/pygls/protocol.py", line 268, in _execute_notification_callback
    if future.exception():
asyncio.exceptions.CancelledError
Exception in callback JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)
handle: <Handle JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)>
Traceback (most recent call last):
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/features/sphinx_manager/manager.py", line 180, in _create_client
    sphinx_info = await client.create_application(resolved)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/features/sphinx_manager/client_subprocess.py", line 170, in create_application
    sphinx_info = await self.protocol.send_request_async("sphinx/createApp", params)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/pygls/feature_manager.py", line 81, in wrapped
    return await f(server, *args, **kwargs)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/setup.py", line 61, in on_document_open
    await call_features(ls, "document_open", params)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/setup.py", line 91, in call_features
    await result
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/features/sphinx_manager/manager.py", line 68, in document_open
    await self.get_client(uri)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/server/features/sphinx_manager/manager.py", line 139, in get_client
    await self._client_creating
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/pygls/protocol.py", line 268, in _execute_notification_callback
    if future.exception():
asyncio.exceptions.CancelledError
[esbonio.SphinxManager] Build command: ['sphinx-build', '-M', 'dirhtml', '/home/rpfeifer/repos/tpl-gw/doc/source', '/home/rpfeifer/.cache/esbonio/3dc5d2b04d86660c5f65422c4103614e']
[esbonio.SphinxManager] Sphinx agent env: {
  "PYTHONPATH": "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio"
}
[esbonio.SphinxManager] Starting sphinx agent: /bin/python -m sphinx_agent
[esbonio.SphinxManager] sphinx-agent process exited with code: 1
[esbonio.SphinxManager] Stderr:
Traceback (most recent call last):
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/sphinx_agent/__main__.py", line 4, in <module>
    from esbonio.sphinx_agent.server import main
ModuleNotFoundError: No module named 'esbonio'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/sphinx_agent/__main__.py", line 6, in <module>
    from .server import main
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/sphinx_agent/server.py", line 14, in <module>
    from .handlers import SphinxHandler
  File "/home/rpfeifer/.vscode/extensions/swyddfa.esbonio-0.91.0/bundled/libs/esbonio/sphinx_agent/handlers.py", line 17, in <module>
    from sphinx import __version__ as __sphinx_version__
ModuleNotFoundError: No module named 'sphinx'
[esbonio.SphinxManager] Cancelled future '<Future at 0x7f45c3624790 state=cancelled>' for pending request '803cc08c-60e7-4bb3-b36c-1252406c71d2'

from esbonio.

alcarney commented on September 15, 2024

I see... the good news is that server appears to be working as expected - it just needs a bit of configuration.

[esbonio] workspace/configuration: {
  "items": [
    {
      "scopeUri": "file:///home/rpfeifer/repos/tpl-gw/doc/source/systemvarianten/systemvarianten.rst",
      "section": "esbonio.sphinx"
    }
  ]
}
[client] Using environment /bin/python: /bin/python
[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "/bin/python"
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": []
}

Here the server is asking VSCode which Python environment it should use to build your documentation and it's responding with /bin/python rather than whatever Python environment you have setup. If you

run the Python: Select Interpreter command via the command palette
choose your project's environment and
run Esbonio: Restart Language Server via the command palette

It should hopefully start working for you.

(I haven't yet figured out how to make this first-run experience work nicely yet, so if you have any suggestions I'd be happy to hear them)

from esbonio.

ExaneServerTeam commented on September 15, 2024

Hello,

I have almost the same issue with no python path issue with the 0.91.0 release on Windows + VScode.
Did not find any workaround yet.

[client] Using environment C:\PROGRAM FILES\PYTHON311\PYTHON.EXE: C:\Program Files\Python311\python.exe
[client] Server start command: c:\Program Files\Python311\python.exe -S -m esbonio.server
[client] LanguageClientOptions: {
  "documentSelector": [
    {
      "scheme": "file",
      "language": "restructuredtext"
    },
    {
      "scheme": "file",
      "language": "python"
    }
  ],
  "initializationOptions": {
    "server": {
      "logLevel": "debug",
      "logFilter": [],
      "showDeprecationWarnings": false,
      "completion": {
        "preferredInsertBehavior": "replace"
      }
    }
  },
  "outputChannel": {
    "name": "Esbonio"
  },
  "connectionOptions": {
    "maxRestartCount": 0
  },
  "middleware": {
    "workspace": {}
  }
}
[client] Starting Language Server
[esbonio] Loaded extension 'esbonio.server.features.sphinx_manager'
[esbonio] Loaded extension 'esbonio.server.features.preview_manager'
[esbonio] Loaded extension 'esbonio.server.features.symbols'
[esbonio] Initialising esbonio v0.16.1
[esbonio] Language client: Visual Studio Code 1.85.1
[esbonio] workspace/configuration: {
  "items": [
    {
      "scopeUri": "file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst",
      "section": "esbonio.sphinx"
    }
  ]
}
[client] Using environment C:\PROGRAM FILES\PYTHON311\PYTHON.EXE: C:\Program Files\Python311\python.exe
[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "c:\\Program Files\\Python311\\python.exe"
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": [],
  "confDir": "${workspaceFolder}"
}
[esbonio.SphinxManager] Trying path: c:\cygwin64\home\xxxxx\Devel\doc\Kubernetes
[esbonio.SphinxManager] Cwd: c:\cygwin64\home\xxxxx\Devel\doc\Kubernetes
[esbonio.SphinxManager] Build command: ['sphinx-build', '-M', 'dirhtml', 'c:\\cygwin64\\home\\xxxxx\\Devel\\doc\\Kubernetes', 'C:\\Users\\xxxxx\\AppData\\Local\\swyddfa\\esbonio\\Cache\\27853ef7fb004a29cf44833871fe4857']
[esbonio.SphinxManager] Sphinx agent env: {
  "PYTHONPATH": "c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio",
  "SYSTEMROOT": "C:\\WINDOWS"
}
[esbonio.SphinxManager] Starting sphinx agent: c:\Program Files\Python311\python.exe -m sphinx_agent
[esbonio.DocumentSymbols] doc: restructuredtext file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst
[esbonio.SphinxManager] sphinx-agent process exited with code: 1
[esbonio.SphinxManager] Stderr:
Traceback (most recent call last):
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\sphinx_agent\__main__.py", line 4, in <module>
    from esbonio.sphinx_agent.server import main
ModuleNotFoundError: No module named 'esbonio'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\sphinx_agent\__main__.py", line 6, in <module>
    from .server import main
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\sphinx_agent\server.py", line 14, in <module>
    from .handlers import SphinxHandler
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\sphinx_agent\handlers.py", line 17, in <module>
    from sphinx import __version__ as __sphinx_version__
ModuleNotFoundError: No module named 'sphinx'
[esbonio.SphinxManager] Cancelled future '<Future at 0x2f095cd6910 state=cancelled>' for pending request '5d0550f7-12bb-492a-a40a-52c67fa62424'
Exception in callback JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)
handle: <Handle JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)>
Traceback (most recent call last):
  File "c:\Program Files\Python311\Lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\pygls\protocol.py", line 268, in _execute_notification_callback
    if future.exception():
       ^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\pygls\feature_manager.py", line 81, in wrapped
    return await f(server, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\setup.py", line 61, in on_document_open
    await call_features(ls, "document_open", params)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\setup.py", line 91, in call_features
    await result
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 68, in document_open
    await self.get_client(uri)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 163, in get_client
    return await self._client_creating
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 180, in _create_client
    sphinx_info = await client.create_application(resolved)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\client_subprocess.py", line 170, in create_application
    sphinx_info = await self.protocol.send_request_async("sphinx/createApp", params)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

from esbonio.

alcarney commented on September 15, 2024

Do you have a virtual environment setup to build your docs with? And have you tried the steps outlined in the comment above ?

Looking at your log messages, you look to be in a similar situation, where the server is asking for an environment to use and VSCode is responding with your default Python installation

[esbonio] workspace/configuration: {
  "items": [
    {
      "scopeUri": "file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst",
      "section": "esbonio.sphinx"
    }
  ]
}
[client] Using environment C:\PROGRAM FILES\PYTHON311\PYTHON.EXE: C:\Program Files\Python311\python.exe
[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "c:\\Program Files\\Python311\\python.exe"   <-- This should be set to the path to python.exe in your virtual environment
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": [],
  "confDir": "${workspaceFolder}"
}

from esbonio.

rpfeifer-tcr commented on September 15, 2024

I do not have a virtual or system environment setup to build the docs. I do have a (separate) Docker container that builds the docs.

From the documentation of the VSCode Extension I did not understand that a local doc-building environment is a requirement.

If this is indeed the missing requirement then this issue could best be resolved by adding a paragraph to the extension description.

from esbonio.

ExaneServerTeam commented on September 15, 2024

@alcarney I double checked, but:

No Virtualenv on my end
Any extra python lib installed relative to my Windows Home path C:\Users\xxxx\AppData\Roaming\Python\Python311

So the python interpreter is the right pick on my end (Eg: C:\Program Files\Python311\python.exe).

I have a bit of an improvement when having the following setup (pythonPath is setup)

esbonio import does not fait anymore

[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "c:\\Program Files\\Python311\\python.exe"
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": [
    "c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs"
  ],
  "confDir": "${workspaceFolder}"
}

Now I still get this error

I am wondering why the call ain't something c:\Program Files\Python311\python.exe -m esbonio.sphinx_agent as it looks more logical to me vs PYTHONPATH values

[esbonio.SphinxManager] Starting sphinx agent: c:\Program Files\Python311\python.exe -m sphinx_agent
[esbonio.SphinxManager] sphinx-agent process exited with code: 1
[esbonio.SphinxManager] Stderr:
c:\Program Files\Python311\python.exe: No module named sphinx_agent

from esbonio.

alcarney commented on September 15, 2024

From the documentation of the VSCode Extension I did not understand that a local doc-building environment is a requirement.

@rpfeifer-tcr Apologies, a local environment for building your docs is indeed required. I will make sure to update the documentation to make this more clear.

I double checked, but:

No Virtualenv on my end

Any extra python lib installed relative to my Windows Home path C:\Users\xxxx\AppData\Roaming\Python\Python311

@ExaneServerTeam I see... that's not a setup I had considered! I wonder why the sphinx_agent process cannot see the libraries installed in your AppData folders.... 🤔

It should however, be possible to make it work using something like the following configuration

[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "c:\\Program Files\\Python311\\python.exe"
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": [
    "c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio", 
    "C:\\Users\\xxxx\\AppData\\Roaming\\Python\\Python311\\Lib\\site-packages",
  ],
  "confDir": "${workspaceFolder}"
}

You will have to double check if I guessed the C:\Users\xxxx\AppData\Roaming\Python\Python311\Lib\site-packages path correctly. It should be set to the folder that contains sphinx and all your other libraries.

I am wondering why the call ain't something c:\Program Files\Python311\python.exe -m esbonio.sphinx_agent

It's related to how the next major version of esbonio is designed.

The sphinx_agent is designed to be "injected" into your documentation's Python environment (using the PYTHONPATH env variable) where it provides what esbonio needs to be able to inspect your documentation. To ensure compatibility with any potential environment, the sphinx_agent only depends on Python's standard library and sphinx itself.

If PYTHONPATH was set to ...\bundled\libs - enabling the use of python -m esbonio.sphinx_agent, all of esbonio's dependencies would also be visible to your build environment which could lead to conflicts. So, instead PYTHONPATH is set to ...\bundled\libs\esbonio, where only the modules in esbonio itself are visible and should greatly reduce the risk of introducing a conflict.

from esbonio.

ExaneServerTeam commented on September 15, 2024

@alcarney

You will have to double check if I guessed the C:\Users\xxxx\AppData\Roaming\Python\Python311\Lib\site-packages path correctly. It should be set to the folder that contains sphinx and all your other libraries.

I confirm that enforcing the AppData / User relative python lib PATH within pythonPath config fix the sphinx import issue, but I still have the sphinx_agent issue

[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "c:\\Program Files\\Python311\\python.exe"
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": [
    "C:\\Users\\xxxxx\\AppData\\Roaming\\Python\\Python311\\Lib\\site-packages"
  ],
  "confDir": "${workspaceFolder}"
}
[esbonio.SphinxManager] Trying path: c:\cygwin64\home\xxxxx\Devel\doc\Kubernetes
[esbonio.SphinxManager] Cwd: c:\cygwin64\home\xxxxx\Devel\doc\Kubernetes
[esbonio.SphinxManager] Build command: ['sphinx-build', '-M', 'dirhtml', 'c:\\cygwin64\\home\\xxxxx\\Devel\\doc\\Kubernetes', 'C:\\Users\\xxxxx\\AppData\\Local\\swyddfa\\esbonio\\Cache\\27853ef7fb004a29cf44833871fe4857']
[esbonio.SphinxManager] Sphinx agent env: {
  "PYTHONPATH": "C:\\Users\\xxxxx\\AppData\\Roaming\\Python\\Python311\\Lib\\site-packages",
  "SYSTEMROOT": "C:\\WINDOWS"
}
[esbonio.SphinxManager] Starting sphinx agent: c:\Program Files\Python311\python.exe -m sphinx_agent
Exception in callback JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)
handle: <Handle JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)>
Traceback (most recent call last):
  File "c:\Program Files\Python311\Lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\pygls\protocol.py", line 268, in _execute_notification_callback
    if future.exception():
       ^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\pygls\feature_manager.py", line 81, in wrapped
    return await f(server, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\setup.py", line 61, in on_document_open
    await call_features(ls, "document_open", params)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\setup.py", line 91, in call_features
    await result
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 68, in document_open
    await self.get_client(uri)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 163, in get_client
    return await self._client_creating
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 180, in _create_client
    sphinx_info = await client.create_application(resolved)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\client_subprocess.py", line 170, in create_application
    sphinx_info = await self.protocol.send_request_async("sphinx/createApp", params)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError
[esbonio.SphinxManager] sphinx-agent process exited with code: 1
[esbonio.SphinxManager] Stderr:
c:\Program Files\Python311\python.exe: No module named sphinx_agent
[esbonio.SphinxManager] Cancelled future '<Future at 0x1c4748269d0 state=cancelled>' for pending request '07546799-b3ad-4549-a68f-693da4f583e8'
[esbonio.DocumentSymbols] doc: restructuredtext file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst

from esbonio.

alcarney commented on September 15, 2024

@ExaneServerTeam Sorry, I should have been more explicit, for your setup you will need to include 2 entries for pythonPath

  "pythonPath": [
    "c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio", 
    "C:\\Users\\xxxx\\AppData\\Roaming\\Python\\Python311\\Lib\\site-packages",
  ],

"c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio" fixes the sphinx_agent issue
"C:\\Users\\xxxx\\AppData\\Roaming\\Python\\Python311\\Lib\\site-packages" makes your libraries including sphinx available

from esbonio.

ExaneServerTeam commented on September 15, 2024

@alcarney My fault too. I think we are on the same page though. I also made your setup proposal earlier but still did not work (I also double checked that the Pathes in pythonPath were alright.

[client] Using environment C:\PROGRAM FILES\PYTHON311\PYTHON.EXE: C:\Program Files\Python311\python.exe
[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "c:\\Program Files\\Python311\\python.exe"
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": [
    "c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio",
    "C:\\Users\\xxxxx\\AppData\\Roaming\\Python\\Python311\\site-packages"
  ],
  "confDir": "${workspaceFolder}"
}
[esbonio.SphinxManager] Trying path: c:\cygwin64\home\xxxxx\Devel\doc\Kubernetes
[esbonio.SphinxManager] Cwd: c:\cygwin64\home\xxxxx\Devel\doc\Kubernetes
[esbonio.SphinxManager] Build command: ['sphinx-build', '-M', 'dirhtml', 'c:\\cygwin64\\home\\xxxxx\\Devel\\doc\\Kubernetes', 'C:\\Users\\xxxxx\\AppData\\Local\\swyddfa\\esbonio\\Cache\\27853ef7fb004a29cf44833871fe4857']
[esbonio.SphinxManager] Sphinx agent env: {
  "PYTHONPATH": "c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio:C:\\Users\\xxxxx\\AppData\\Roaming\\Python\\Python311\\site-packages",
  "SYSTEMROOT": "C:\\WINDOWS"
}
[esbonio.SphinxManager] Starting sphinx agent: c:\Program Files\Python311\python.exe -m sphinx_agent
[esbonio.SphinxManager] sphinx-agent process exited with code: 1
[esbonio.SphinxManager] Stderr:
c:\Program Files\Python311\python.exe: No module named sphinx_agent
[esbonio.SphinxManager] Cancelled future '<Future at 0x156c69fa490 state=cancelled>' for pending request 'a1ae4458-a29a-4434-9652-f0ff76931c8d'
Exception in callback JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)
handle: <Handle JsonRPCProtocol._execute_notification_callback(<Task cancell...anager.py:80>>)>
Traceback (most recent call last):
  File "c:\Program Files\Python311\Lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\pygls\protocol.py", line 268, in _execute_notification_callback
    if future.exception():
       ^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\pygls\feature_manager.py", line 81, in wrapped
    return await f(server, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\setup.py", line 61, in on_document_open
    await call_features(ls, "document_open", params)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\setup.py", line 91, in call_features
    await result
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 68, in document_open
    await self.get_client(uri)
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 163, in get_client
    return await self._client_creating
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\manager.py", line 180, in _create_client
    sphinx_info = await client.create_application(resolved)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\server\features\sphinx_manager\client_subprocess.py", line 170, in create_application
    sphinx_info = await self.protocol.send_request_async("sphinx/createApp", params)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError
[esbonio.DocumentSymbols] doc: restructuredtext file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst

from esbonio.

ExaneServerTeam commented on September 15, 2024

@alcarney And of course this works in regular windows CMD (With no specific setup)

"Still be aware that I have a space in my python path and I can't fix that (Corporate Setup)

c:\Users>cd "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio"

c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio>"c:\Program Files\Python311\python.exe" -m sphinx_agent
```

from esbonio.

alcarney commented on September 15, 2024

@ExaneServerTeam So close yet so far! 😢

That config (in theory at least!) should work, but it looks like you've found a bug in how the server constructs the PYTHONPATH environment variable on Windows - it looks like it's using the wrong separator character to combine the paths.

[esbonio.SphinxManager] Sphinx agent env: {
  "PYTHONPATH": "c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio:C:\\Users\\xxxxx\\AppData\\Roaming\\Python\\Python311\\site-packages",
                                                                                                      ^ ':' should be ';' on Windows I think....
}

I'll see if I can get a fix out sooner rather than later :)

In the meantime, if you are able, you could try creating a virtual environment with the venv module for your dependencies to see if will work that way.

I also need to figure why you need to do all this configuration in the first place... with virtualenvs at least, calling just the interpreter is enough for the libraries to be visible to the process making this extra config unnecessary 🤔
If you run set PYTHONPATH in your regular CMD window does it have a value already set? It's possible the server is accidentally overriding it.

"Still be aware that I have a space in my python path and I can't fix that (Corporate Setup)

Ok, thankfully I don't think that is an issue in this situation :)

from esbonio.

ExaneServerTeam commented on September 15, 2024

@alcarney regarding the var in CMD, no PYTHONPATH is defined. For info the python modules lookup path, when a python is spawned

-> Interesting to see that C:\\Users\\xxxxx\\AppData\\Roaming\\Python\\Python311\\site-packages is setup before 'C:\Program Files\Python311\Lib\site-packages' as expected ;)

Python 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', 'C:\\Program Files\\Python311\\python311.zip', 'C:\\Program Files\\Python311\\DLLs', 'C:\\Program Files\\Python311\\Lib', 'C:\\Program Files\\Python311', 'C:\\Users\\xxxxx\\AppData\\Roaming\\Python\\Python311\\site-packages', 'C:\\Program Files\\Python311\\Lib\\site-packages']

from esbonio.

alcarney commented on September 15, 2024

Thanks, now I'm even more confused, since that folder is already on sys.path I can't think of a reason why it didn't work for you originally 😅

from esbonio.

ExaneServerTeam commented on September 15, 2024

@alcarney looking at the start function in client_subprocess.py I do not see where the c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio path is supposed to be injected before running the python -m sphinx_agent

My understanding of the code says it should be somewhere in the PYTHONPATH env var as python is spawned.

from esbonio.

alcarney commented on September 15, 2024

The start function assumes PYTHONPATH already contains the required paths.

As part of the startup process, the SphinxManager calls config.resolve(...)
which will inject the c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio path (assuming the user has not already provided pythonPath in the config)

from esbonio.

ExaneServerTeam commented on September 15, 2024

@alcarney I don't see anywhere in the code where PYTHONPATH Env var is updated.

So my take so far would be to have:

❯ git diff lib/esbonio/esbonio/server/features/sphinx_manager/client_subprocess.py                                                                                                       │
│diff --git a/lib/esbonio/esbonio/server/features/sphinx_manager/client_subprocess.py b/lib/esbonio/esbonio/server/features/sphinx_manager/client_subprocess.py                           │
│index 29cd485..943c4f4 100644                                                                                                                                                            │
│--- a/lib/esbonio/esbonio/server/features/sphinx_manager/client_subprocess.py                                                                                                            │
│+++ b/lib/esbonio/esbonio/server/features/sphinx_manager/client_subprocess.py                                                                                                            │
│@@ -150,6 +150,7 @@ class SubprocessSphinxClient(JsonRPCClient):                                                                                                                         │
│                                                                                                                                                                                         │
│         command.extend([*config.python_command, "-m", "sphinx_agent"])                                                                                                                  │
│         env = get_sphinx_env(config)                                                                                                                                                    │
│+        env['PYTHONPATH'] = f"{os.path.dirname(__file__)}{os.pathsep}{env['PYTHONPATH']}"                                                                                               │
│                                                                                                                                                                                         │
│         self.logger.debug("Sphinx agent env: %s", json.dumps(env, indent=2))                                                                                                            │
│         self.logger.debug("Starting sphinx agent: %s", " ".join(command))

Or having cwd setup to os.path.dirname(__file__)

from esbonio.

alcarney commented on September 15, 2024

Perhaps I am missing what you are looking for...

From my perspective, if we consider your original comment we see that

With no pythonPath set manually

[esbonio] configuration: {
  "buildCommand": [],
  "pythonCommand": [
    "c:\\Program Files\\Python311\\python.exe"
  ],
  "cwd": "",
  "envPassthrough": [],
  "enableSyncScrolling": true,
  "enableDevTools": false,
  "pythonPath": [],
  "confDir": "${workspaceFolder}"
}

The server is able to correctly locate the parent folder of sphinx_agent and place it on the process' PYTHONPATH
(Which is done by the config.resolve() function I mentioned earlier and the get_sphinx_env() function you helped fix in #719)

[esbonio.SphinxManager] Trying path: c:\cygwin64\home\xxxxx\Devel\doc\Kubernetes
[esbonio.SphinxManager] Cwd: c:\cygwin64\home\xxxxx\Devel\doc\Kubernetes
[esbonio.SphinxManager] Build command: ['sphinx-build', '-M', 'dirhtml', 'c:\\cygwin64\\home\\xxxxx\\Devel\\doc\\Kubernetes', 'C:\\Users\\xxxxx\\AppData\\Local\\swyddfa\\esbonio\\Cache\\27853ef7fb004a29cf44833871fe4857']
[esbonio.SphinxManager] Sphinx agent env: {
  "PYTHONPATH": "c:\\Users\\xxxxx\\.vscode\\extensions\\swyddfa.esbonio-0.91.0\\bundled\\libs\\esbonio",
  "SYSTEMROOT": "C:\\WINDOWS"
}

The sphinx_agent module is found and is launched, but quickly fails

[esbonio.SphinxManager] Starting sphinx agent: c:\Program Files\Python311\python.exe -m sphinx_agent
[esbonio.SphinxManager] sphinx-agent process exited with code: 1
Traceback (most recent call last):
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\sphinx_agent\__main__.py", line 4, in <module>
    from esbonio.sphinx_agent.server import main
ModuleNotFoundError: No module named 'esbonio'

We ignore the red herring about esbonio not being found (since that is expected and handled here)
And see that it crashes because it's unable to find sphinx, and supposedly any other module installed in your AppData site-packages folder

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\sphinx_agent\__main__.py", line 6, in <module>
    from .server import main
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\sphinx_agent\server.py", line 14, in <module>
    from .handlers import SphinxHandler
  File "c:\Users\xxxxx\.vscode\extensions\swyddfa.esbonio-0.91.0\bundled\libs\esbonio\sphinx_agent\handlers.py", line 17, in <module>
    from sphinx import __version__ as __sphinx_version__
ModuleNotFoundError: No module named 'sphinx'

Which is what led us down the road of trying to configure pythonPath option ourselves, to try and include both the sphinx_agent and your AppData site-packages folder....

Personally I'm not sure what adding {os.path.dirname(__file__)} to PYTHONPATH will achieve, since, (if I'm not mistaken), will only make this folder visible to the sphinx_agent, when the issue (again, to me at least) is that the sphinx_agent process cannot find your libraries installed in AppData

from esbonio.

alcarney commented on September 15, 2024

Thanks for digging into it!
I'm due to push out an update soon^TM anyway, I'll make sure to include the relevant changes

from esbonio.

ExaneServerTeam commented on September 15, 2024

@alcarney: clear improvement on my end, no more issue on loading sphinx libs and spawning processes.
Right now I am few other issues (Having 2 different setup under testing).

Main blocker right (Hit on both setup) now is related to the render path.

[esbonio.PreviewManager] Previewing file: 'file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst'
[esbonio.PreviewManager] Unable to preview file 'file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst', not included in build output.

from esbonio.

alcarney commented on September 15, 2024

clear improvement on my end, no more issue on loading sphinx libs and spawning processes.

Glad to hear it!

Main blocker right (Hit on both setup) now is related to the render path.

[esbonio.PreviewManager] Previewing file: 'file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst'
[esbonio.PreviewManager] Unable to preview file  'file:///c%3A/cygwin64/home/xxxxx/Devel/doc/Kubernetes/index.rst', not included in build output.

Has the server done a full sphinx-build yet? If not, making a change to an *.rst file and saving it should be enough to kick it off. Once the build has finished, opening the preview again should hopefully work 🤞.

The previous version did automatically kick off a build when opening the preview for the first time, however that had the tendency to spawn infinite processes! So I've taken it out in this one while I think of a better approach 😅

from esbonio.

alcarney commented on September 15, 2024

Closing as I think this is resolved now

from esbonio.

`ModuleNotFoundError` upon startup of VSCode extension about esbonio HOT 28 CLOSED

Comments (28)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent