doughtmw / hololensforcv-unity Goto Github PK

HoloLens research mode streams for use in Unity.

C# 100.00%

hololensforcv-unity's Introduction

HoloLensForCV-Unity

HoloLens research mode streams made available for use in Unity through IL2CPP Windows Runtime support. See my blog post for more information on the sample.

Incorporates:

HoloLensForCV sample from Microsoft
depthPvMapper from cyberj0g

Requirements

Tested with Unity 2018.4 LTS
Visual Studio 2017/2019
Minimum RS4, tested with OS Build 17763.678
HoloLens with research mode enabled

Sensor Stream Sample

Git clone repo. From the main project directory, clone submodules with:

git submodule update --init

Copy precompiled dlls and HoloLensForCV.winmd file from the Prebuilt->x86 folder to the Assets->Plugins->x86 folder of the HoloLensForCVUnity project.

Optional: build project from source

Open HoloLensForCV sample in VS2017/2019 and install included OpenCV nuget package to HoloLensForCV project. In the nuget package manager type:

Install-Package ..\OpenCV.HoloLens.3411.0.0.nupkg -ProjectName HoloLensForCV

Build the HoloLensForCV project (x86, Debug or Release)
Copy all output files from HoloLensForCV output path (dlls and HoloLensForCV.winmd) to the Assets->Plugins->x86 folder of the HoloLensForCVUnity project

Open HoloLensForCVUnity Unity project and build using IL2CPP, allow unsafe code under Unity Player Settings->Other Settings
Navigate to Unity project build folder and modify the Package.appxmanifest file to include:

Restricted capabilities package:

<Package 
  xmlns:mp="http://schemas.microsoft.com/appx/2014/phone/manifest" 
  xmlns:uap="http://schemas.microsoft.com/appx/manifest/uap/windows10" 
  xmlns:uap2="http://schemas.microsoft.com/appx/manifest/uap/windows10/2" 
  xmlns:uap3="http://schemas.microsoft.com/appx/manifest/uap/windows10/3" 
  xmlns:uap4="http://schemas.microsoft.com/appx/manifest/uap/windows10/4" 
  xmlns:iot="http://schemas.microsoft.com/appx/manifest/iot/windows10" 
  xmlns:mobile="http://schemas.microsoft.com/appx/manifest/mobile/windows10" 
  xmlns:rescap="http://schemas.microsoft.com/appx/manifest/foundation/windows10/restrictedcapabilities" 
  IgnorableNamespaces="uap uap2 uap3 uap4 mp mobile iot rescap" 
  xmlns="http://schemas.microsoft.com/appx/manifest/foundation/windows10">

Modified capabilities with with new package:

  <Capabilities>
    <rescap:Capability Name="perceptionSensorsExperimental" />
    <Capability Name="internetClient" />
    <Capability Name="internetClientServer" />
    <Capability Name="privateNetworkClientServer" />
    <uap2:Capability Name="spatialPerception" />
    <DeviceCapability Name="webcam" />
  </Capabilities>

Open VS solution, build then deploy to device

hololensforcv-unity's People

Contributors

Stargazers

Watchers

Forkers

avatarchik aaman-git mnrmja007 zhangjianhao666 bingogome dpiccolomd gobanana520 lhhj elithenicasio yuechenw213

hololensforcv-unity's Issues

Get 3D World point

Hi,
How to get the 3D world point of a pixel position in the color frame?

Error on clone submodules

Hi,

The following error is populating when run the command
git submodule update --init

D:\Work\HiTechTEDs\AR\experimental\HoloLensForCV-Unity>git submodule update --init
Cloning into 'D:/Work/HiTechTEDs/AR/experimental/HoloLensForCV-Unity/HoloLensForCV'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:doughtmw/HoloLensForCV.git' into submodule path 'D:/Work/HiTechTEDs/AR/experimental/HoloLensForCV-Unity/HoloLensForCV' failed
Failed to clone 'HoloLensForCV'. Retry scheduled
Cloning into 'D:/Work/HiTechTEDs/AR/experimental/HoloLensForCV-Unity/HoloLensForCV'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Initializing MediaFrameSourceGroups...

Hi, I followed the instructions in your repo. I have built and deployed the app to the HoloLens 2. I can see the blank Canvas with the Text which never goes beyond the message: "Initializing MediaFrameSourceGroups". I have questions in order to clarify your instructions to make sure I did it correctly:

In step 2: I presume I need to create the subfolder folder named x86 in Assets->Plugins in the Unity project and copy the 2 dlls and winmd files into it.

In step 3: "build using IL2CPP" Does this involve importing the MRTK2 Foundation package to the Unity project, or does this just involve switching platform to Universal Windows Platform (UWP)?

In step 4: Do I manually edit the <Capabilities> xxx </Capabilities> xml myself to match what you've shown or does this happen as a result of editing the code above it during build time.

In step 5: To build and deploy from Visual Studio, do I target ARM or ARM64? The HoloLens 2 device does not support x86.

From your personal blog:

" be sure to select the Assembly-CSharp.Player ": Where can I find this setting in Visual Studio?

Other potential issues:

I downloaded the zip file from the GitHub repo and worked from there. The "HoloLensForCV" folder is empty when I do this.
I am using Unity 2019.4.15f, but there doesn't seem to be any issues with this.
Should I manually enable any capabilities in the Player->Publishing Settings ?

I would be obliged and indebted to you if you can clarify these questions. Thank you in advance.

x86 configuration not available

x86 configuration is not available for HololensForCV in configuration manager dropdown. I wanted to add some extra bit of code to "DepthPvMapper" but not sure how to build it after that..Please help

Abnormal depth value

I successfully configured your project, this is a very good example. But I have some questions to ask you. I changed the code to directly read the depth map information and save it in byte[]. When I display it on the texture, it seems to be almost completely black. My code as follow:
` var DepthFrame = latestShortDepthCameraFrame.SoftwareBitmap;
DepthFrame = SoftwareBitmap.Convert(DepthFrame, BitmapPixelFormat.Rgba8, BitmapAlphaMode.Premultiplied);
var w = DepthFrame.PixelWidth;
var h = DepthFrame.PixelHeight;

         Destroy(_pvDepthTexture);
        _pvDepthTexture = new Texture2D(
            w,
            h,
            TextureFormat.RGBA32, false);
        if (bytes==null)
            {
                bytes = new byte[w * h * 4];
            }
       
        DepthFrame.CopyToBuffer(bytes.AsBuffer());
        DepthFrame.Dispose();
      
     _pvDepthTexture.LoadRawTextureData(bytes);
                   `

I also noticed that in your example, it seems that all the depths are displayed in white. I am a novice and it is really difficult to understand. Please help me to solve this confusion.

How to enable Assembly-CSharp.Player?

From your blog:

"Open C# Unity scripts in Visual Studio, and be sure to select the Assembly-CSharp.Player viewing option for code, otherwise ENABLE_WINMD_SUPPORT code in the #define directive with appear greyed out."

How exactly do I enable Assembly-CSharp.Player?

Nice work,
Thank you.

hololensforcv project built error: Unable to resolve external symbols...

I want to get output files of hololensforcv project, but it built error because:Unable to resolve external symbols... I konw that maybe configurations' wrong, but do not know how to slove it. Hope you can help me.
I did not have opencv4.1.1 in my PC and not creat NuGet package(that is necessary?), and I don't know if it can cause the error.

Visualization in world space

Wonderful project, with the Readme descriptions the build worked without a problem, thanks!

Two issues I am having:

The app crashes on my Hololens quite often. Is that just for me or did you also experience that?
Do you plan to visualize the depth information in world space instead of a screen?

how can i use this code in Hololens2?

First of all, thank you for your hard work.

my project is creating a 3D model on a specific color area as some kind of marker.

Basically it works like a point cloud that transformed a specific pixel area of color range in a color image into spatial coordinates.

In my opinion, It's a necessary process that pixel-by-pixel mapping between color and depth images, for this project.

So your code is a very good reference to me, but I don't know how to apply it in hololens2 .

https://github.com/petergu684/HoloLens2-ResearchMode-Unity

Apart from this, I am trying to translate a python script in the official repo of the link to c++, for frame-by-frame processing.

can you help me?

Unable to load image

It can be deployed to the HoloLens successfully, but it is prompted that it cannot load "kernelbase.pdb/dll" during operation. After ignoring that, I can not the image, only blank. However, the text and the gesture recognition are normal. Did you meet this problem???