stereokit / stereokit Goto Github PK

It's currently possible to blit from texture to texture with a Material. We need to be able to render a full scene to the texture as well.

This could be implemented as a render queue stack, so you can push and pop render queues. This would also need to work well with the Hierarchy as well, perhaps pushing and popping the whole hierarchy stack too.

• Toggle hand visualization on/off.
• Let user swap to user defined materials.
• User defined meshes too?

Issue: With C# it's not possible to know if the OpenXR initialization has succeeded. The application will just run silently.

Steps to reproduce:

• Don't attach a Windows MR headset
• Run the StereoKitTest application

Result: The application starts normally, however nothing happens. As a developer I have no way of knowing if the application has a correct OpenXR xr_session running.

Expected: There is either an exception if no xr_session could be created or some other way of knowing.

Details: With CPP it is possible to know if the initialization has failed as sk_init() returns a bool. With C# there is no such thing as the constructor is just calling sk_init() and not listening to any return.

Suggestion: Change StereoKit class constructor to:

        public StereoKitApp(string name, Runtime runtime)
        {
            if (!sk_init(name, runtime))
            {
                throw new Exception("StereoKit: Could not initialize " + runtime.ToString() + " runtime");
            }
        }

Proposal: Add exceptions to StereoKitC library to let them properly bubble up, regardless of CPP or C#. At least for the fatal ones (like creating xr_session, events, actions, rendering etc.)

Right now, Model is the most basic of 3d object containers! It would be good to flesh this out with additional information about the model, the ability to parse through a model loaded from file in greater detail, and assemble more complicated models.

Once 0.1 is stabilized a bit, it needs to be easy to use and create a project with! NuGet is a great way to do this. Paired with a project template, it should be simple to get started.

StereoKit currently uploads raw RGBA data to the GPU, which is generally slow and inefficient. Textures should be compressed if they're not being regularly altered.

Basis Universal could be an excellent intermediate or preferred format for compressed textures.
https://github.com/binomialLLC/basis_universal

Here's a basisu sample implementation over here in the Sokol samples:
https://github.com/floooh/sokol-samples/tree/master/libs/basisu

And another implementation in the Wicked engine:
https://github.com/turanszkij/WickedEngine/blob/master/WickedEngine/Utility/utility_common.cpp

Right now, inputs are hands only. This is great for MR, but limiting for debug/flatscreen. Also, MR devices can still have mice and keyboards! This will also open up stronger compatibility with ImGui.

We'll limit this to just Keyboard + Mouse for now.

This is not part of an Action system, though it may function as a lower layer for one later on.

x86 builds won't display properly with this glitch present. Looks like the problem may be fixed somewhere upstream, so I'll put this here for revisiting later.

https://developercommunity.visualstudio.com/content/problem/562932/directxmath-matrix-multiplication-incorrect-when-r.html

UI should have the ability to display 3D models as part of the visuals. This can be as replacement for buttons, or as their own UI elements.

UI models are largely for use with UI interactions, like grab/move/scale UI interactions.

Following getting started, I downloaded the vs template, but was unable to open it, with the error:

"This extension is not installable on any currently installed products."

I only had Visual Studio 2017 installed, but it turns out StereoKit needs Visual Studio 2019. It would be nice to mention this in the documentation.

Currently, shaders don't have enough information to use normal maps. Meshes don't have tangents! So that'll need calculated and added to the mesh information.

Alternatively, there may be methods for doing normals without precalculated tangents?
http://www.thetenthplanet.de/archives/1180

Need to finish adding comment docs to C++ and C#.

Would be great to have this in a way where it's visible both to intellisense, -and- the documentationn generator, while avoiding docs duplication, but this might be an uphill battle.

• Material
• Mesh
• Model
• Shader
• Solid
• Sprite
• Tex
• Plane
• Pose
• Quat
• Ray
• Sphere
• Vec2
• Vec3
• Vec4
• Input
• Log
• Renderer
• UI
• Color
• StereoKitApp
• Native Types

A little further down the line at the moment, but still worth putting on the roadmap.

A component system for implementing quick and re-usable behaviors! C# should do a much better job of this than C, so I think it's an excellent place to start moving over into the C# world. This would probably look a lot like what Unity has in many ways.

I'd love to have components stored in large arrays of structs for cache locality.

A small API for detecting intersections between basic shapes! This will largely be used by the UI system for detecting interaction with inputs.

Should include functions for the following:
Rays/Points/Lines/AABBs/Spheres intersect with
Planes/AABBs/Spheres

Where Spheres are low priority.

Splitting this out from HL2 Best Practices, since I'm pushing it to v0.2. Depth buffer is important, but most important at low frame rate. StereoKit has been extremely fast so far, any performance issues right now are typically blocking IO operations.

Depth sharing recommendation details are here:
https://github.com/microsoft/OpenXR-SDK-VisualStudio#understand-openxr-samples-and-best-practices-for-hololens-2

I noticed this in a project I'm working on; if I take a copy of the default material for a mesh, then attempt to set a texture to that material, it ends in an AccessViolationException. If I don't copy the material everything is fine, except all the models in my scene inherit that texture.

Here's some code that you can drop into your test project to repro:

public class DemoTextureCrash : IDemo
{
    private Model _Model;

    public void Initialize()
    {
        _Model = Model.FromMesh(Mesh.GenerateCylinder(10, 1, Vec3.Up), Default.Material.Copy());            
    }

    public void Shutdown()
    {

    }

    private bool _DoOnce;

    public void Update()
    {
        if (!_DoOnce)
        {
            var mat = _Model.GetMaterial(0);
            mat.SetTexture("diffuse", Tex.FromFile("Floor.png"));

            _DoOnce = true;
        }

        _Model.Draw(Matrix.T(Vec3.Zero));
    }
}

The exception is on this line in render.cpp during a call to Step:

d3d_context->VSSetShader(shader->vshader, nullptr, 0)

Did a design review with Dan recently and got a great chunk of feedback to try out! The UI needs at least a cursory pass at these things before we can call it ready. Has to look at least a little bit good in order to get people to bite :)

• Color Scheme! Light text on darker background. Needs some palette work too.
• Finger Glow. Finger proximity feedback can be helpful for understanding distance to interface.
• Interaction Feedback: More feedback for interaction, colors, visual effects, noises, etc.
• Round Buttons: They look more professional :)
• Button Frames: Something a little extra around the base of the buttons, make them feel like they're not just boxes.

ISpatialAudioClient is the recommended method for rendering audio on a HoloLens! It does spatial audio and mixing on the HPU, so is largely free to use as far as the app is concerned. Win-win all around.

Reference implementation is here:
https://github.com/microsoft/spatialaudio-unity/tree/develop/Source/Spatializer/HoloLens2

Need functionality for translating SpatialCoordinateSystem into a usable position for use with OpenXR.

Hi and at first - thanks for the nice work!

I have a problem, when I try to run the example in the HoloLens 2 Emulator. As I unfortunately don't have a HL2 available yet I wanted to have look at you framework in the emulator.

If I am right the emulator for HL2 does not use ARM64 like the real device, but x64 or x86 to run the application. For that reason I choose the config Debug, x64, StereoKitTest_UWP, HoloLens 2 Emulator. But when starting this, I am getting an exception complaining about a missing DLL file StereoKitC.dll.
Building and running it as x64 for Local Machine works as expected. (Beside the fact I would expect to run the app in the Windows MR Portal, instead of a UWP Desktop application - but that's not related to my question here ;) )

Did I miss something or do I need to run some scripts first?

It would be great to offload graphics to a separate thread! Multi-threading is definitely optimal and wonderful, but the primary objective here is to try and prevent users from slowing down the rendering.

If the user ever does anything on the main thread that blocks execution, the render thread should be able to continue along, rendering the most recently provided list using newer head/hand position information.

A set of MR interactions where you'll be able to grab UI elements and move/rotate/scale objects. This is similar to the current sk_ui_affordance function, but with an expanded set of capabilities.

OpenXR is supported on a number of different platforms, so it would be advantageous to be able to render to those platforms too! Right now StereoKit is based on DX11, but this only gives us the Windows platform.

For full compatibility, Metal support, as well as Vulkan or OpenGL would also be needed. That's at least two more renderers that'll need to be written.

Alternatively, using a rendering library such as bgfx (https://github.com/bkaradzic/bgfx) could be really good! Would be great to survey around a bit and see what other options would work as well.

This topic also has a big impact on how materials and shaders look in the future.

Pretty common to see .fbx files around!

Here's a nice library for it:
https://github.com/nem0/OpenFBX

A C# binary works pretty much everywhere, but a C++ binary is platform and architecture specific! The StereoKit library binds to a native DLL to gain access to lower level functionality, so it needs to be referencing the correct native DLL for the system that it's currently on!

It's possible to manually load a DLL in advance (at least on Windows?) and references will just use the manually loaded DLL instead of trying to load it again. This allows us to decide and load a platform specific DLL before anything starts making calls to it!

Some initial work for this is is here: 8d8d489

Currently, we've got some pointer code! But this was written pretty early, and I haven't used it for much. It needs review, updating, and integration with the UI system for far interactions.

This is dependent on the intersections API.

StereoKit needs a C# template for getting started. This might need a NuGet package first? Needs some research. This is definitely dependent on 0.1 being relatively stable first.

Main page needs to reflect all the great work that's been happening lately!

• Getting started guide
• Link to NuGet and VS templates
• Link to new docs
• Current feature list update
• Roadmap
• More???

Hello,

This looks like a great project. I see that it currently is only compatible to build for ARM64, which is 64-bit and thus only works on the HL2. Is there planned availability for the 32-bit HL1?

Thanks in advance, and I look forward to hopefully being able to use StereoKit soon.

techtide

Currently, shaders are compiled to the application's temporary folder at first launch. But this can cause a long pause on the very first run: not great for first impressions!

Since shaders are StereoKit specific creations anyhow, not generic assets, it would be fine to just pre-compile them, and use the compiled shader as the final asset. This could be a command-line utility executed through something like the documentation tool, or a standalone tool/IDE for building shaders that outputs a compiled file. Command-line may be a good stop-gap until a shader editor can be built.

This should also consider multi-platform rendering, so a pre-compiled shader asset should contain binary data for DirectX as well as OpenGL or Vulkan, whatever we settle on for rendering on non-windows platforms. Doesn't need to be in until we have multi-platform rendering, but it's good to consider when writing it.

Asset loading is the most common form of blocking operation, so lets make it not block! Since we're doing all our asset pipeline work during runtime, this could end up being a lot, especially for big, non-optimal files.

This will allow us to do things like a staggered loading system:

• Begin loading, set a 'fallback' texture while we're grabbing the initial file.
• Load the image, set that as the active object so it can be seen right away.
• Compress the texture, and again replace the existing object.

It should be possible to specify a containing rectangle when rendering text! A bounding box that lets text wrap, align to different sides, and clip at the bottom.

Worth considering multi-column text too, as long as we're at it. But not a requisite for this ticket!

There's still some remaining recommendations for OpenXR performance I haven't gotten to for StereoKit. Don't need to get them all in for 0.1, just need to review and snag the low hanging fruit for now.

Recommendations are all here:
https://github.com/microsoft/OpenXR-SDK-VisualStudio#understand-openxr-samples-and-best-practices-for-hololens-2

• Rendertarget color format
• Texture array rendertarget
• Unbound reference space
• MRC alpha blend

Currently using left handed values. Should be easy enough, XM library says to just negate each value:https://docs.microsoft.com/en-us/windows/win32/api/directxmath/nf-directxmath-xmquaternionrotationrollpitchyaw

There's a DLL issue when running x64/ARM64 UWP builds in Debug mode. This only happens in the github example project, not projects built through the template!

Anchors are important for hologram stability, so this is an important detail for people that want to create a quality application. Could consider adding this as part of the Hierarchy system? But could still be better as an individual item.

See the OpenXR spec for spatial anchors here:
https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#XR_MSFT_spatial_anchor

Reflections are a super important component of PBR! Without it, metals and shiny objects just don't look right.

StereoKit currently has some basic PBR support with directional light sources, but still needs a fair bit of cubemap and spherical harmonics work before reflections are in.

Sooo, cubemaps! Spherical harmonics! And mip-mapping, since it's quite related. Existing work can be found here: b4b0120

As I am going through the code, I would love to propose some (very opinionated) updates of the file structure of the project:

Collect examples in their own topical subfolders with CPP and CS underneath
Reason: This would give us the ability to add more focused examples as we go along.

• Create a /Examples folder in the root directory
• Add a folder for each topic, currently a /GettingStarted one and add simple README for folder
• Move /StereoKitCTest to /GettingStarted/CPP
• Move StereoKitTest to /GettingStarted/CS

Move topics in the core to respective subfolders
Reason: The current prefixes help to identify topics, however the number of files will grow and splitting them into concerns will keep the project manageable.

• Mirror the current VS project structure as folders by adding a /Platform folder and adding openxr.cpp and .h, win32.cpp and .h and win32_input.cpp and .h there
• Start creating topical folders for things that have more than 2 files, for example /Input or /AssetManagement

What do you think?

Current text implementation is raster, SDF (Signed Distance Field) could increase visual quality! Also may allow for inline bold text :)

Right now, a shader is just a single shader. It's often quite useful to multi-compile a shader to remove costly operations that aren't being used! Like normal mapping.

This will require some kind of sub-shader resource that contains the actual compiled shader.

Compiling all variants at startup could be costly for large shaders (see Unity compile time) so it could be advantageous to delay shader variant compile until it actually gets used?

StereoKit has Solid physics objects, but I only ever got a basic joint system implemented. Might be sitting in a stash or branch somewhere? This needs to be revisited, polished up, and exposed to C#!

I like .ply, it's an easy format that supports vertex colors! .ply files are often used for point clouds, so it could be good to consider point cloud rendering when importing .ply files as well.

Might be able to find a nice library for this on here:
https://github.com/nothings/single_file_libs

Docs need guides! And in StereoKit, we do that by writing demos! So here's some demos we still need:

• Getting Started
• User Interface
• Input

I tried StereoKit with the Oculus runtime and ran into the issue at openxr.cpp#L165 because the laptop I was using has Optimus and creates an integrated rather than discrete graphics device.

Currently chaining the Initialization() calls fails (for example calling it once with Runtime.MixedReality and if that fails with Runtime.Flatscreen) because of some internal configuration that already happened. Ideally this should work.

As per @DirkSonguer's observation.

There's a lot of warnings right now. Ooops, bad form! It's on the ToDo list now.

A simple text input that will receive characters from the keyboard.

For now we'll limit functionality to just typing characters. No in-MR virtual keyboard or text selection.

Functions need a consistent prefix throughout the project!

Something for managing a scene, swapping out scenes, and loading scenes from files! This could probably be mixed with some kind of prefab system, since a prefab is really just a small scene. Bonus points for something that already has an editor?

This could be a core item for a UI framework later on as well, especially if it pulls double duty as a prefab system?

If this is also a prefab, then having a system for promoting child parameters to the root interface would be essential. For example, if there was a complex prefab hierarchy with models, text, icons, etc. like a button, the text field should be promoted to the root so it's easily visible and changeable without having knowledge of the button's prefab hierarchy.

Physics is critical to realistic interactions, and the heart of MR UI!

Looking to integrate Bullet (https://pybullet.org/wordpress/) as a core component of StereoKit. Bullet looks like one of the better options, but I'm still pretty open to alternatives!