X11 Forwarding? about maniskill HOT 3 CLOSED

By default, the demo_manual_control.py only uses OpenCV which requires no Vulkan support. Thus, you might need to manually set the display name with export DISPLAY=:X command.

Nonetheless, if you start the script with the ``--enable-sapien-viewer` option, then the viewer does require Vulkan to be supported for the given display Unfortunatelely，it is a bit tricky to use X11 forwarding with Vulkan as it requires GLX extension to be present. However, there are two ways to support headless interactive visualization.

1. Docker (Recommended): If you have access to a working nvidia-docker installation on your server, this would be the easiest solution. By using this docker image and setting the corresponding environment variables, you would be able to create a VNC session that support our interactive viewer.

2. VirtualGL: If you prefer creating a systemwide VNC session, you could use VirtualGL and TurboVNC. However, this requires you change the default display manager from gdm to lightdm on the host machine to avoid conflict with the VNC sessions. A brief setup instruction has been tested on Ubuntu 20.04:

   1. Download and install deb files of VirtualGL and TurboVNC.
   2. Stop the display manager by sudo systemctl set-defaults multi-user.target and a reboot.
   3. Run sudo vglserver_config and select 1. Configure server for use with VirtualGL (GLX + EGL back ends). We recommend using its recommended settings.
   4. Add gdm, lightdm, and any users wishing use GPU to the vglusers group: sudo usermod -aG vglusers USER.
   5. Switch the default display manager by sudo dpkg-reconfigure gdm3 and select lightdm.
   6. Re-enable graphical login interface by sudo systemctl set-defaults graphical.target and reboot
   7. Add TurboVNC to your path: export PATH=$PATH:/opt/TurboVNC/bin

   After setting up TurboVNC, you should be able to create a GLX-enabled VNC session with the command vncserver and access the session with SSH port forwarding at port 590X, where X is the display name.

Both the Docker and VirtualGL methods create a VNC session with a fully functional display manager. Thus, the DISPLAY=:<DISPLAY> is not necessary if you decide to run our manual control script inside the VNC session. Nonetheless, if you run the script outside of the session, you need to pass the display name into the script to property start up the interactive viewer.

from maniskill.

Additional comment: can you first try to open any OpenCV window through X11 forwarding to check the basic configuration?

from maniskill.

Hi everyone,

Thank you for your help! I'll let you know if I run into any further issues. Again, great work!

from maniskill.