SteamVR Integration about dxvk HOT 13 CLOSED

(Quickly: OpenVR is the API, and SteamVR is the runtime. Not really sure why there's a distinction but it is what it is.)

That is because SteamVR is not the only implementation of that Interface, just like Wayland has multiple compositors. The other implementation is sorry for the confusion OSVR which is a free software project and offers compatibility with various different hardware. it would be cool if that worked as well.

from dxvk.

I see.

In that case, all the more reason to be strictly a wrapper. Let whatever runtime happens to be present pick up the proverbial phone and do all the actual work. OpenVR itself is crossplatform in the truest sense -- there's a direct 1:1 mapping for everything as long as Direct3D doesn't get involved.

If OSVR doesn't work under that arrangement, it would be an issue with the application or OSVR itself, not DXVK or any separate OpenVR wrapper in between.

from dxvk.

That is because SteamVR is not the only implementation of that Interface, just like Wayland has multiple compositors. The other implementation is OSVR which is a free software project and offers compatibility with various different hardware.

Sorry for off topic, but

OSVR is not an implementation of OpenVR, it has only its own API. For running OpenVR applications you absolutely need SteamVR, a proprietary runtime you can only download on the proprietary Steam.

OpenVR is called OpenVR because the API is somewhat documented so it is possible to implement your own openvr library with different backends which I started at https://github.com/ChristophHaag/openvr_api-libre but it's really far from being complete enough to be usable and I expect it to be rendered unnecessary by the release of OpenXR long before it's ready.

Anyway, why would VK_EXT_external_memory have so much of an overhead? On Linux, SteamVR is using it for OpenGL support...

I don't really know anything about d3d11, but from what I read in 5 minutes, the application need to textures explicitly with a flag that says they are shareable before you can import them elsewhere.

On Linux it seems you can always make a new OpenGL context that shares resources like textures with an existing context, so I think SteamVR is probably doing that to import textures from applications that are not already using the external memory extensions.
From my 5 minute googling it seems you can not really do that in the original d3d11, but since dxvk is the d3d11 implementation, it could just allow you to do that anyway...

from dxvk.

Anyway, why would VK_EXT_external_memory have so much of an overhead? On Linux, SteamVR is using it for OpenGL support...

external_memory's overhead is virtually nonexistent, barring driver bugs. The overhead of option 3 comes from maintaining a third Vulkan instance (DXVK + SteamVR + the OpenVR wrapper in between), calling through a second wrapper (possibly third, depending on how Wine does NT handles; I never could wrap my head around it), and sharing the texture twice (once in the wrapper, once in SteamVR), plus all the synchronization paperwork to make sure all these extra pieces don't clobber each other. In addition, the DXGI/D3D API calls involved will inevitably at some point have side effects or extra code that simply slow things down in this usecase but are needed for other things (the entire NT handle business makes this pretty likely out of the gate). In contrast, options 1 and 2 have trivial overhead as they cut out the middleman and perform the actual texture submission (quasi-)internally.

(On a side note: SteamVR's compositor fails to start if external_memory is missing. It appears to rely on it even for internal things.)

I don't really know anything about d3d11, but from what I read in 5 minutes, the application need to textures explicitly with a flag that says they are shareable before you can import them elsewhere.

A fun discovery of mine is that the textures submitted by all the apps I checked don't have said flag. I suspect SteamVR is creating its own buffer for eye textures, exporting that into the application's rendering context (whatever D3D calls it) and copying the submitted texture to its own buffer from within said context.

On Linux it seems you can always make a new OpenGL context that shares resources like textures with an existing context, so I think SteamVR is probably doing that to import textures from applications that are not already using the external memory extensions.

SteamVR's Linux compositor is actually built on Vulkan internally, so functionality for sharing between OGL contexts doesn't do it any favors -- it needs to pull submitted OGL textures across the API "barrier" into its Vulkan instance somehow. It has two ways to do this:

The "fast path" uses OpenGL counterparts to the Vulkan sharing extensions; namely, GL_EXT_memory_object and GL_EXT_semaphore. Due to limitations in memory_object, this fast path would need to also do the "make own eye texture, export and copy" process. Since OpenGL state is process-local, it doesn't need any special information from the application to do this.

The "slow path" appears to be literally copying the submitted texture to system memory, then loading it into the compositor's Vulkan instance from there. Obviously, the compositor and the application are keeping separate buffers in this case as well.

(On a side note: they keep the slow path around because the fast path is presently Nvidia exclusive -- implementation of GL_EXT_semaphore in radeonsi has yet to be started, last I checked.)

DXVK, utilizing Vulkan internally, theoretically makes that whole discussion moot (and would actually be fast on AMD hardware since RADV already has everything that matters).

Making its own texture buffer and copying what the app submits to that buffer may simply be the MO of the compositor for all applications regardless of OS or rendering API. But, we don't really know. Always fun when you're forced to guess at implementation details, isn't it?

From my 5 minute googling it seems you can not really do that in the original d3d11, but since dxvk is the d3d11 implementation, it could just allow you to do that anyway...

If we're going to be implementing a special out-of-band API, we might as well go with option 1, no?

from dxvk.

(On a side note: they keep the slow path around because the fast path is presently Nvidia exclusive -- implementation of GL_EXT_semaphore in radeonsi has yet to be started, last I checked.)

lostgoat has patches for it: https://github.com/lostgoat/mesa/commits/wip-semaphore-gl-v2
but when I tested v1 it had some issues with the steamvr compositor that they wanted to fix first (see ValveSoftware/SteamVR-for-Linux#78 (comment))
Anyway, SteamVR doesn't actually require it. The "fast path" is used for radeonsi too and it doesn't seem to have major downsides. Maybe they use some other (possibly slower) method for synchronization.

But I see that a pipeline where you have to synchronize multiple texture accesses sequentially would be a problem.

A fun discovery of mine is that the textures submitted by all the apps I checked don't have said flag. I suspect SteamVR is creating its own buffer for eye textures, exporting that into the application's rendering context (whatever D3D calls it) and copying the submitted texture to its own buffer from within said context.

Well libopenvr_api is opened as a library so it runs in the same process as the application and can theoretically run commands in the rendering context of the application.

I mean on linux in OpenGL applications it could just enable GL_EXT_memory_object and then access the textures directly from its Vulkan renderer. Making a second, shared, OpenGL context with it enabled seems "safer" to me as to not accidentally messing up the rendering of the application, but I don't really know what they do.

Anyway, enough sidenotes.

from dxvk.

I was hoping to hear an opinion from @doitsujin in particular but he hasn't checked in far as I can tell. In a week or so I may take a crack at implementing 1 myself unless he has a thought otherwise.

from dxvk.

I've been following the discussion, but be honest, I don't know anything at all about VR from a programmer's perspective.

By the looks of it, option 1 seems to be a reasonably simple approach that should also help keep things out of DXVK that don't necessarily belong there. However, I'd like to ask a few things:

• Do we need DXGI support for Stereo 3D swap chains and video modes etc.?
• Is there anything else required on the D3D11/DXGI side of things to support this, and does it affect any part of the API other than presentation?
• How would this actually be tested?

I'm certainly not against it, but some clarification about what precisely the requirements are would be great.

from dxvk.

Individual applications may require stereo swap chain functionality or other features special to VR, but there are no such requirements in the OpenVR API itself. Those would be handled separately in a way independent of any SteamVR integration; for that matter, they're equally if not more relevant to OpenXR, OSVR, and any other similar future APIs. In any case, I suspect most applications don't have any special requirements beyond inter-process resource sharing (Edit: Actually, it's the VR compositor that needs inter-process resource sharing, and that happens in native Vulkan land, so even that is a non-issue), given how the VR compositor interacts with the application.

Looking over openvr.h again, it's a little more involved than I thought. We'd need:

1. Access to DXVK's instance, device, and a queue handle as well as said queue's family index (struct passed to IVRCompositor::Submit) Exactly what queue this should be is not explained in the header, but I could probably figure it out from the hellovr_vulkan example.
2. Ability to map a VkPhysicalDevice to a DXGI adapter index (IVRSystem::GetDXGIOutputInfo, IVRSystem::GetOutputDevice, IVRExtendedDisplay::GetDXGIOutputInfo).
3. Ability to fetch the VkImage underneath an ID3D11Texture (a lot of things, but most importantly IVRCompositor::Submit)
4. The inverse, ability to share a VkImage as an ID3D11Texture (a lot of things, but most importantly IVRRenderModels::LoadTextureD3D11_Async)
5. This one may be tricky: specify additional instance and device extensions DXVK should ask for (IVRCompositor::GetVulkanInstanceExtensionsRequired & IVRCompositor::GetVulkanDeviceExtensionsRequired). Upon entering IVRSystem::GetDXGIOutputInfo I know that a) I have been called by an OpenVR scene application intending to use Direct3D 11 for rendering and b) it hasn't set up its D3D rendering stuff just yet, but you'd know better than I do whether that's likely to predate creation of DXVK's instance.

I may have missed some things in IVRApplications but that part of the API is rarely used; this should cover everything scene applications (i.e. games) want.

from dxvk.

As for testing... simple unit tests would be straightforward (e.g. render something in D3D, pull it into Vulkan, verify the VkImage contains the expected result... then do the reverse). The big picture smoke test, I think, would be simply using it -- (me) writing WineOpenVR around it, finding a VR-optional game that's at least technically playable in DXVK, then running it in VR mode and, let's be honest, figuring out why it explodes.

At that point I'll be spending a lot of time writing testcases to reproduce things I find :)

from dxvk.

It JUST dawned on me what you meant by the first bullet point; the words "swap chain" didn't quite register until just now. This addresses the second a bit more as well:

An OpenVR scene application isn't involved in presentation at all. It simply hands off the rendered frames to the VR compositor, which manages its own swapchain and is in exclusive control of the headset's display -- the application cannot interact with the display itself. Said compositor is an integral part of the SteamVR runtime, meaning in this case it's a native Linux program using Vulkan.

The app technically doesn't have to so much as create a window unless their graphics API of choice separately requires it (e.g. OpenGL via GLX). However, most do; usually they simply mirror one eye to a desktop window, but some will render an additional angle as a third-person "spectator view" and a few of the more creative apps invite a second user to the desktop monitor (e.g. Mass Exodus).

I can't imagine any of this requiring anything of DXVK above and beyond your typical desktop game in terms of swapchains or video modes.

One potential sticking point is whether you have implemented DXGI adapter selection/enumeration stuff yet, but I don't really have a problem with assuming that DXVK would pick the same physical device as SteamVR (which is extremely likely on any remotely VR-capable system) for the time being.

from dxvk.

Seeing the discussion in #38 where @oiaohm said (and I definitely agree):

Really with dxgi we need vk9, dxvk and wine project vkd3d to attempt all be on the same page

It strikes me that this idea has similarities to a hypothetical relationship between DXVK and WineOpenVR. Therefore there probably should be a more general discussion on how to make various Vulkan-backed alternative API implementations (is that an appropriate term?) share the same Vulkan instance and related objects. WOVR could easily piggyback on whatever form that would take.

@doitsujin do you want to discuss that here, there, or make a separate issue?

from dxvk.

Since this affects API interop in general I'd suggest opening a separate issue for that.

from dxvk.

master implements basic OpenVR interop to pull in the required device and instance extensions. This should be sufficient.

from dxvk.