Dear @doughtmw

I am quite new in C++ , can you tell me what is the symbol '^' in the end of some datatype meaning, like below

Windows::Foundation::Collections::IVector<DetectedArUcoMarker^>^ detectedMarkers
= ref new Platform::Collections::Vector<DetectedArUcoMarker^>();

Thanks.

Hi!! I'm trying to run the ArUcoDetectionHololensUnity project on my Hololens 1. When running the ArUcoDetection scene, I receive the following exception at runtime.

(Translated: "Access violation when reading path 0x000000".)

The exception is thrown from the HoloLensForCV dll. After several tests I noticed that the exception is thrown by the following line in ArUcoMarkerDetection.cs

The demo works fine until an ArucoMarker is recognised, than this exception is thrown. These are the last lines from the output console.

I have used the dlls in the Prebuilt Folder (x86). I have also tried to build the HoloLensForCV project from Visual Studio, but the OpenCv.Hololens.3411 does not exist in the NuGet package store.

Do you have any advice on how to solve it?

Thanks in advance!!

How did you know you had a memory leak while displaying the video feed to a Unity texture. Is there a setting in Visual Studio debugger to show this?

Also the video feed is flipped upside down when rendered on a texture in Unity. Is there a handy method to flip it right side up.

Thanks.

I run the untiy project and installed to Hololens2
the cube showed when put the marker in front of H2,
but the cube position is not matching the marker,
the marker size is about 2 cm , so I changed the Marker Size to 0.02.
my unity3d version is 2019.4.8f1 , no MRTK sdk and any package installed,
is it any missing or wrong, Thanks any helps .

After deployment, the Hololens-1 is showing "Initializing MediaFrameScourceGroups..." message only. In the Debugger Output I get:
'ArUcoDetectionHoloLensUnity.exe' (Win32): Loaded 'U:\USERS\DefaultAccount\AppData\Local\DevelopmentFiles\ArUcoDetectionHoloLensUnityVS.Release_Win32.abdul\HoloLensForCV.dll'. Cannot find or open the PDB file.
'ArUcoDetectionHoloLensUnity.exe' (Win32): Unloaded 'U:\USERS\DefaultAccount\AppData\Local\DevelopmentFiles\ArUcoDetectionHoloLensUnityVS.Release_Win32.abdul\HoloLensForCV.dll'
Exception thrown at 0x77003AE2 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x02A0EE9C.
Exception thrown at 0x77003AE2 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x02A0EFB8.
Exception thrown at 0x77003AE2 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x02A0F228.
The thread 0x175c has exited with code 0 (0x0).

I run the example in my H2 well , but I need to use the higher resolution of camera
like the 19201080 or higher that supported by H2,
Is it possible to change the resolution from 896504 to higher?? Thanks.

Dear Doughty

I want to get the 2D position of detected marker and I add the variable like std::vecotrcv::Point2f marker2DPosition in the class DetectedArUcoMarker , but got the error code c2871 , 'cv' : a namespace with this name does not exist
and c2653, 'cv' : is not a class or namespace name

below is the modified class , and any help plz.

public ref class DetectedArUcoMarker sealed
{
public:
DetectedArUcoMarker(
In int id,
In Windows::Foundation::Numerics::float3 position,
In Windows::Foundation::Numerics::float3 rotation,
In std::vectorcv::Point2f marker2DPosition,
In Windows::Foundation::Numerics::float4x4 cameraToWorldUnity);

	property int Id;
	property Windows::Foundation::Numerics::float3 Position;
	property Windows::Foundation::Numerics::float3 Rotation;
	property std::vector<cv::Point2f> Marker2DPosition;
	property Windows::Foundation::Numerics::float4x4 CameraToWorldUnity;
};

Hi Mitchel,

First of all thank you for this amazing repo and your work in this area!
I'm currently pursuing a project where we try to implement ARuco marker detection for augmented Infrared ARuco tags. Thus we would like to run something very similar to your pipeline, but on the short throw infrared camera stream. Your repo was a huge help in bringing those research streams into a unity project, but you mentioned that it would require more input processing to use the other streams for the Marker detection.

Do you have an idea/a pointer to the parts in your code where one would have to make these changes? Does the current ARuco marker detection pipeline expect black and white markers for example, and thus thresholding would be required etc.?

Also I was wondering if the current code that implements the different research streams already handles all the transformations from the headset to the specific camera sensor, or if that is currently assumed to be the front facing photo video camera (thus needing a separate calibration for the other camera streams).

Thanks for any help on this!

Hi Mitchell,

I ran the project on my Holo2 (after calibration) and the cube model is hovering above the ArUco marker
I validated that the results are fine in a PC simulation (by rendering the cube on the camera image).

I would appreciate your advice on the matter.

Thanks,

Yonatan

Hi everyone!
Is it possible to access to detected aruco tag id?
Thanks in advance,

Filippo

Severity Code Description Project File Line Suppression State
Error (active) E1740 lambda captured variable of type "HoloLensForCV::MediaFrameSourceGroup ^" cannot be copied to closure class field of type "HoloLensForCV::MediaFrameSourceGroup ^*" HoloLensForCV C:\Users\avina\Desktop\ArUcoDetectionHoloLens-Unity-master\HoloLensForCV\HoloLensForCV-master\Shared\HoloLensForCV\MediaFrameSourceGroup.cpp 237

I am seeing this error when I was trying to understand why it is taking so much time for initializing the MediaFrame Source Group.

Please do help me with it.

When I first built the code on HoloLens, it was working fine with outputs of marker count to the debugger. However when I tried to run it again, with no changes, I started getting this error. (Deleted and reinstalled, same crash with the same stack trace. Though it magically? worked once again after reinstallation, not in the first try.)

Here is the stack trace:

'ArUcoDetectionHoloLensUnity.exe' (Win32): Unloaded 'C:\Program Files\Intel\Media SDK\libmfxhw32.dll'
Exception thrown at 0x76D02552 (KernelBase.dll) in ArUcoDetectionHoloLensUnity.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
MediaFrameSourceGroup::TryInitializeMediaCaptureAsync: MediaCapture is successfully initialized in shared mode.
MediaFrameSourceGroup::InitializeMediaSourceWorkerAsync: selected group has 2 media frame sources
MediaFrameSourceGroup::GetSensorType:: assuming SensorType::PhotoVideo per _mediaFrameSourceGroupType check (source id is 'Source#1@\?\DISPLAY#INT22B8#4&27b432bd&0&UID139960#{e5323777-f976-4f5b-9b55-b94699c46e44}{CDD6871A-56CA-4386-BAE7-D24B564378A9}')
MediaFrameSourceGroup::GetSubtypeForFrameReader: evaluating MediaFrameSourceKind::Color with format Video-NV12 @30000/1001Hz and resolution 896 x 504
MediaFrameSourceGroup::InitializeMediaSourceWorkerAsync: created the 'PhotoVideo' frame reader
'ArUcoDetectionHoloLensUnity.exe' (Win32): Loaded 'C:\Windows\System32\msvproc.dll'.
The thread 0x308 has exited with code 0 (0x0).
MediaFrameSourceGroup::InitializeMediaSourceWorkerAsync: started the 'PhotoVideo' frame reader
MediaFrameSourceGroup::GetSensorType:: assuming SensorType::PhotoVideo per _mediaFrameSourceGroupType check (source id is 'Source#0@\?\DISPLAY#INT22B8#4&27b432bd&0&UID139960#{e5323777-f976-4f5b-9b55-b94699c46e44}{CDD6871A-56CA-4386-BAE7-D24B564378A9}')
MediaFrameSourceGroup::InitializeMediaSourceWorkerAsync: sensor type has already been initialized!
Exception thrown at 0x76D02552 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0260F484.
The thread 0x6c8 has exited with code 0 (0x0).
Exception thrown at 0x64C8E1FE (HoloLensForCV.dll) in ArUcoDetectionHoloLensUnity.exe: 0xC0000005: Access violation reading location 0x00000020.

Hi, Mitchell,

Thanks for your great work! I'm reading your code and have a problem about the transformation between the frame 'unity' and 'marker' in the script ArUcoMarkerDetection.cs.
// Get pose from OpenCV and format for Unity
Vector3 position = CvUtils.Vec3FromFloat3(detectedMarker.Position);
position.y *= -1f;
Quaternion rotation = CvUtils.RotationQuatFromRodrigues(CvUtils.Vec3FromFloat3(detectedMarker.Rotation));
Matrix4x4 cameraToWorldUnity = CvUtils.Mat4x4FromFloat4x4(detectedMarker.CameraToWorldUnity);
Matrix4x4 transformUnityCamera = CvUtils.TransformInUnitySpace(position, rotation);
// Use camera to world transform to get world pose of marker
Matrix4x4 transformUnityWorld = cameraToWorldUnity * transformUnityCamera;
In this function CvUtils.RotationQuatFromRodrigues(CvUtils.Vec3FromFloat3(detectedMarker.Rotation)), we input the rotation detectedMarker.Rotation which is between 'opencv' and 'aruco'. The output is the rotation between unity and aruco. But I don't know why we need to multiply Quaternion.Euler(0, 0, 180) in this function CvUtils.RotationQuatFromRodrigues(). Because the euler angle between 'unity' and 'opencv' is q = Quaternion.Euler( -1.0f * q.eulerAngles.x, q.eulerAngles.y, -1.0f * q.eulerAngles.z).
q = Quaternion.Euler( -1.0f * q.eulerAngles.x, q.eulerAngles.y, -1.0f * q.eulerAngles.z) * Quaternion.Euler(0, 0, 180);

Would you give me some suggestions? Thanks!

Best regards,
Ang

Hi,
Thank you, again for this contribution. It would be really helpful if you could help me understand an issue I am having with your app. I wanted to understand why there is a drift/movement in the MarkerCube's pose once detection is stopped by double-tapping. The following is an image that presents this issue. The image is taken after the app is stopped by double tapping and then hololens is moved to a different location. You can see that the cube appears to have drifted to the left significantly. It seems that this is the case whenever hololens is moving around and the cube is set using the last detected world-pose of the marker.

A bit of context:
In my work, I am trying to register a Kinect for Azure camera with Hololens-1 using your app. I first get the world pose of the marker using your app, while keeping the hololens still on a 3D printed structure. After that, I copy that info in my 2nd app to initiate the MarkerCube's world pose. Next, I start holoremoting form the 2nd app and pick up my hololens. While walking around, I see the same problem as mentioned above. The hologram of the cube is not aligned anymore with the marker although I had previously determined the hologram's world pose.
I did make sure the transforms of MainCamera in your and my scenes are all set to 0. One difference was that you were using a vertical fov of 30 in your scene. In my scene, MRTK sets that to 16 by default. I made sure both are the same in your and my scenes. That seemed to have improved the misalignment along z-axis significantly.

Any suggestions would be very helpful. Thank you.

Dear doughtmw
Sorry , I need to post again after changed the MediaFrameSourceGroup file to the other resolution like 1920x1080 , 2272 x 1278 that supported by Hololens 2 mentioned in https://docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/locatable-camera

I only modified the x and y of the code below for 1920x1080 and 2272x1278 of two kind setting

if (format->VideoFormat->Width == x && format->VideoFormat->Height == y)

and then got 2272x1278 but very slow about 0.1FPS , and got no frame of 1920x1080 , can you help to make it can using with the 2272x1278 resolution and 30FPS??

Hi, I'm trying to get the samples to run on the HoloLens 1. However, I'm stuck on the Initializing MediaFrameSourceGroups. The following exception had happened.

'ArUcoDetectionHoloLensUnity.exe' (Win32): Loaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\ArUcoDetectionHoloLensUnityVS.Debug_Win32.user\HoloLensForCV.dll'. 
'ArUcoDetectionHoloLensUnity.exe' (Win32): Unloaded 'U:\Users\DefaultAccount\AppData\Local\DevelopmentFiles\ArUcoDetectionHoloLensUnityVS.Debug_Win32.user\HoloLensForCV.dll'
Exception thrown at 0x76FF6182 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x026DE25C.
Exception thrown at 0x76FF6182 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x026DE7B4.
The thread 0x1098 has exited with code 0 (0x0).
Exception thrown at 0x76FF6182 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x026DF0E8.
Exception: Exception of type 'System.Exception' was thrown.
  at HoloLensForCV.SensorFrameStreamer..ctor () [0x00000] in <00000000000000000000000000000000>:0 
  at ArUcoDetectionHoloLensUnity.ArUcoMarkerDetection+<StartHoloLensMediaFrameSourceGroups>d__23.MoveNext () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) [0x00000] in <00000000000000000000000000000000>:0 
  at ArUcoDetectionHoloLensUnity.ArUcoMarkerDetection.StartHoloLensMediaFrameSourceGroups () [0x00000] in <00000000000000000000000000000000>:0 
  at ArUcoDetectionHoloLensUnity.ArUcoMarkerDetection+<Start>d__19.MoveNext () [0x00000] in <00000000000000000000000000000000>:0 
  at System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) [0x00000] in <00000000000000000000000000000000>:0 
  at ArUcoDetectionHoloLensUnity.ArUcoMarkerDetection.Start () [0x00000] in <00000000000

Viewing this, it appears that the constructor for SensorFrameStreamer is throwing an error. Not too sure why, though...
After that, I get the below, which makes sense because it is not initialized (from the exception above)

Exception thrown at 0x76FF6182 (KernelBase.dll) in ArUcoDetectionHoloLensUnity.exe: 0x40080202: WinRT transform error (parameters: 0x8000000B, 0x80070490, 0x00000014, 0x017CDFB8).
[7.615456 / 16.812085] - OnWindowActivated event - Deactivated.
Exception thrown at 0x76FF6182 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x07B5F1B0.
Exception thrown at 0x76FF6182 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x026DEB18.
Exception thrown at 0x76FF6182 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x026DF0E8.
NullReferenceException: Object reference not set to an instance of an object.
at ArUcoDetectionHoloLensUnity.ArUcoMarkerDetection.<Update>b__21_0 () [0x00000] in <00000000000000000000000000000000>:0 

Not too sure why this happened, and this isn't the first time I'm deploying to the HoloLens. I built x86 on Debug/Release, using Unity 2019.4 and VS2019. I'm using the master branch as well. I've also built the HoloLensForCV project from source, x86 Debug/Release.

Any ideas why this could be happening, or some idea of where I can look for a fix?

Hi @doughtmw

Thank you for sharing your project.

When I first open the HoloLensForCV sample, I noticed that there are two projects in the solution that say "load failed": Debugging and Io. If I just ignore then and proceed trying to install the OpenCV.UWP.411 nugget package, I don't find any with this name, instead I find OpenCV.HoloLens v3411.0.0 (highlighted as missing from the solution) and OpenCV.Windows v411.0.0. I then just click "Restore" under the "Manage Nuget Packages" menu. When I try to build the project (x86, Debug or Release) I got an error that says:

• The imported project file "C:\Program Fules (x86)\Windows Kits\10\bin\10.0.17134.0\XamlCompiler\Microsoft.Windows.UI.Xaml.Common.targets" could not be loaded. Root element is missing.

Have you come across this issue?

Best,

Rodrigo

Hi doughtmw,

I'm not able to clone the current master of HoloLensForCV submodule, when opening the link I get a 404 error.

Cheers

Hi,
First of all, thank you for your awesome work with this repo.

I am currently using this repo to detect aruco marker attached to movable object. The result is extremely good and i would like to continue using it in the future. But my projects has some commercial usage, so i would like to ask if i can still use this repo in future released products.
Sorry to trouble you because i do not understand what i can do with your included MIT license in project.

Greetings from Japan.

Using OpenCvSharp for HoloLens 2. When I call new Mat() I catch the exception:

System.UnauthorizedAccessException: Failed getting the path of a special folder. Access denied -> System.Environment.GetWindowsFolderPath(System.Int32 folder)

Any suggestions appreciated

Hi guys,
I need your help. I have this issue in my code and i don't understand why .... Do you have any ideas ?
Exception thrown at 0x7077C6B6 (opencv_world3411.dll) in GnathiX.exe: 0xC0000005: Access violation writing location 0x00000000.

Hi, I want to see the ID of the tag. Therefore I use this to print and display on screen:

foreach (var detectedMarker in detections)
{
Debug.Log("Marker ID : " + detectedMarker.markerId);
ID.text =(string) detectedMarker.markerId;
}

But this gives an error :

Assets\Scripts\ArUcoMarkerDetection.cs(270,63): error CS1061: 'DetectedArUcoMarker' does not contain a definition for 'markerId' and no accessible extension method 'markerId' accepting a first argument of type 'DetectedArUcoMarker' could be found (are you missing a using directive or an assembly reference?)

Can anyone say how I can display the ID?

i am unable to retrieve the unity coordinate system (Line 205) as it returns null. I have enable windows mr in XR setting

"com.unity.xr.windowsmr": "3.1.6-preview.1"

Please let me know what is missing

Respected Sir/Ma'am,

I would like to incorporate this method into my project. I have some questions regarding it.

1. I would like to know what is the Component 'Pvgo' in the Aruco Script Holder and why is 'Frames' component being added into the field? I also wanted to know what is the Frames Component itself and how should I create one for my use?

2. What other components should I incorporate from here, so that my project could function correctly?

Please do help! I am entirely new to this field.

Thanking you,

Yours sincerely,

Avinash Shanmugam

Vlc_rf_parameters.txt
Vlc_lf_parameters.txt

Hello @doughtmw , i've run a camera calibration for the Visible Light LF and RF cameras obtaining the parameters in the attached files. Maybe you can include them in the winmd to let aruco tracking works also with these research sensors.

Moreover, can you please explain me how do you obtain the output files .winmd and .dll

HololensForCV.dll HololensForCV.winmd opencv_aruco411.dll etc...
Following the guide in the readme i'm not able to obtain them, I get a lots of errors trying to compile the HoloLensForCV VS solution.
Which nuget packages whould i include? I see OpenCV.HoloLens.411 is pre-included, but using only it the building fails. If i include also OpenCV.UWP.411, the build succeed but when I build on unity and deploy the application something goes wrong.
Thank you in advance.

Hey Mitchell, I am trying out your ARUCO tracking project here. Great job!
As I navigate through the project, I spent some time figuring out the NuGet package installation and restoration issue. You could consider put the prebuilt binaries in Unity Plugins folder so new users could easier bring up the app.

Hi,

I was wondering if you have (already) tested support for Unity 2019.3 or are aware of any issues migrating to 2019.x from 2018.x? We've upgraded recently from 2018.x to 2019.3 and ran into a few issues such as null pointer exceptions (on arguments) such as:

"Exception thrown at 0x76CB3AE2 in ArUcoDetectionHoloLensUnity.exe: Microsoft C++ exception: Platform::InvalidArgumentException ^ at memory location 0x0240ECF4. HRESULT:0x80070057 The parameter is incorrect.WinRT information: Argument cannot be null."

at

IL2CPP_EXTERN_C IL2CPP_METHOD_ATTR RuntimeObject* MediaFrameSourceGroup_DetectArUcoMarkers_mD7AE9A0A18CAA3E7B1F7F5BDA13E6EFC50DBFCC4 (MediaFrameSourceGroup_t81A62BB7229CDBDE57DACE8362C9A7657B19DEB8 * __this, int32_t ___type0, const RuntimeMethod* method)

This might be related to our project setup, but before digging to far into I thought I give it a shot and reach out to you.

Thanks in advance!
Best regards,
Michael

Hi,
thank u for sharing the amazing repo for us.
I am using it for some work recently. There is a question: when i try to use only the secondary scene, HLArUcoExample, instead of entering it from the Main scene, the camera can be invoked on PC but cannot be awaken on my HoloLens2 after deployment.
To reduce the memory usage, it is a must that i use only the secondary scene named above.
Here is the normal condition on PC.

Can you help me with this? Thanks again!

Hi,

I just build your repo and tested the ArUCo detection. Works fine, but I always get some offset between the cube and the marker.
Is there any calibration that need to be done ?

PS : the screenshot is not really relevant because the MRC doesn't align Holograms correctly with the world but still wanted to share it..
(https://docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/mixed-reality-capture-for-developers)

I run the example in my H2 well , but I need to use the higher resolution of camera
like the 19201080 or higher that supported by H2,
Is it possible to change the resolution from 896504 to higher?? Thanks.

after PM> Install-Package ..\OpenCV.HoloLens.3411.0.0.nupkg -ProjectName HoloLensForCV

got errors

PM> Install-Package ..\OpenCV.HoloLens.3411.0.0.nupkg -ProjectName HoloLensForCV
Install-Package : Some NuGet packages are missing from the solution. The packages need to be restored in order to build the dependency graph. Restore the packages before perfor
ming any operations.
At line:1 char:1

• Install-Package ..\OpenCV.HoloLens.3411.0.0.nupkg -ProjectName HoloLe ...

•   + CategoryInfo          : InvalidOperation: (:) [Install-Package], InvalidOperationException
    + FullyQualifiedErrorId : NuGetMissingPackages,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
  
  

PM>

Something missing or wrong??

Hi,
I saw no information in the Readme about licensing or usage. Can (and if to what degree) can we use your code?
TY in advance!
Greetings from Germany.

Hi,
I tried to build the Unity project for HL2 but without success. I also tried to build the DLLs HololensForCV but also without success. Can somebody provide me DLLs?

Exception:

BuildFailedException: Failed to build Visual Studio project using arguments 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\MSBuild.exe "C:\0_PROJECTS\Unity\aruco\ArUcoDetectionHoloLensUnity.sln" /nologo /maxcpucount /p:Configuration=Release /p:Platform=ARM64 /p:SolutionDir="C:\\0_PROJECTS\\Unity\\aruco\\" /t:Build /clp:Verbosity=minimal'.
Output:  Building GameAssembly.dll with MsvcWinRtToolChain
  	Msvc Install Version: 15.0
  	Msvc Install SDK Directory: C:\Program Files (x86)\Windows Kits\10
  	Msvc Linker Path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\HostX64\ARM64\link.exe
  	Msvc Compiler Path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\HostX64\ARM64\cl.exe
  
  	Output directory: C:\0_PROJECTS\Unity\aruco\build\bin\ARM64\Release
  	Cache directory: C:\0_PROJECTS\Unity\aruco\build\obj\il2cppOutputProject\ARM64\Release
  ObjectFiles: 366 of which compiled: 0
  Total compilation time: 263 milliseconds.
  Cleaned up 1 object files.
C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2203,5): error MSB3271: There was a mismatch between the processor architecture of the project being built "" and the processor architecture, "ARM", of the implementation file "C:\0_PROJECTS\Unity\aruco\ArUcoDetectionHoloLensUnity\HoloLensForCV.dll" for "C:\0_PROJECTS\Unity\aruco\ArUcoDetectionHoloLensUnity\HoloLensForCV.winmd". This mismatch may cause runtime failures. Please consider changing the targeted processor architecture of your project through the Configuration Manager so as to align the processor architectures between your project and implementation file, or choose a winmd file with an implementation file that has a processor architecture which matches the targeted processor architecture of your project. [C:\0_PROJECTS\Unity\aruco\ArUcoDetectionHoloLensUnity\ArUcoDetectionHoloLensUnity.vcxproj]


ApplicationLauncherImpl.RunMSBuild (System.String args, Utility+VisualStudio vs) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/ApplicationLauncher.cs:478)
ApplicationLauncherImpl.Build (Utility+VisualStudio vs) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/ApplicationLauncher.cs:333)
ApplicationLauncher.BuildAndRunProject (UnityEditor.Modules.BuildLaunchPlayerArgs args, UnityEditor.WSABuildAndRunDeployTarget deployTarget, System.String platform, Utility+VisualStudio vs) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/ApplicationLauncher.cs:67)
ApplicationLauncher.BuildAndRun (UnityEditor.Modules.BuildLaunchPlayerArgs args) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/ApplicationLauncher.cs:39)
UnityEditor.UWP.BuildPostprocessor.LaunchPlayer (UnityEditor.Modules.BuildLaunchPlayerArgs args) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/ExtensionModule.cs:77)
UnityEditor.PostprocessBuildPlayer.Launch (UnityEditor.BuildTargetGroup targetGroup, UnityEditor.BuildTarget buildTarget, System.String path, System.String productName, UnityEditor.BuildOptions options, UnityEditor.Build.Reporting.BuildReport buildReport) (at <4a1b55d6b009421a9ad1d16b54b782d5>:0)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

It would be great if you could add support for Unity 2020, as opening the project with that version currently leads to compilation errors with the scripts.

Pretty straight forward title.

Hi everyone!
I have followed your guide, but I am still not able to deploy the project on Hololens 2. Am I losing something?
I can't set Device as target debugger and I do not know why. It only happens with this project to me.

Hi @doughtmw ,
I'm trying to use your code with the Kalman Filter but i have an error of violation writing location in media groupe framework. Do you any idea how to resolve this problem ?

Thanks

hello and thank you for reading. in this issue i'll walk through my process and why i cant seem to run this project. I'm unable to reproduce my initial issue. so at the end i try to recount that as best i can. if anyone has Hints and or steps i can use please notify me!

I'm using:

• VS2019
• Unity 2019.4.20f1/ 2020.3.0f1
• the RS4 link results in an error 404 and my windows OS should be up to date
• i would like to detect markers from the 4X4_50 dictionary.

I am not doing the optional building the project from source steps.

my exact steps:

• git clone the repo
• CD into repo
• git submodule update --init
  these are all the mentioned steps necessary.

i have done camera calibration before to do some native ArUco tracking. these are my HoloLens2 values:

Running an example

from here there are no exact steps for running the scenes and i use the general setup guide from the MRTK tutorials
https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/tutorials/mr-learning-base-02

step 1: adding the project in unity Hub
i simply navigate to the project folder and select
C:\UNITYPROJECTS\ubiquitous_computing\ArUcoDetectionHoloLens-Unity\ArUcoDetectionHoloLensUnity

step 2: update and open

the used unity version is a bit outdated. here i switch it to a slightly newer 2019 LTS version normally. in this trail run i'm trying the 2020 LTS version

this looks like a mistake as it results in various errors

i'll retry using the 2019.4.20f1 LTS version

this time i'll attach the updrade.log file
No errors this time!
Packages-Update.log
i realise now that this might explain later errors.

step 3: building the unity project

• first i enter the calibration data into the inspector of the script holder object. However, i won't arrive at a point where this could be an issue.
  

• then i got to file -> buildsettings and switch to the UWP platform

• then i use the most fitting settings and build
  

Issues and how i handle them

issue 1 the built UWP project .sln when opened in VS2019 is not loaded

C:\UNITYPROJECTS\ubiquitous_computing\ArUcoDetectionHoloLens-Unity\ArUcoDetectionHoloLensUnity\build\ArUcoDetectionHoloLensUnity\ArUcoDetectionHoloLensUnity.vcxproj : error  : The application for the project is not installed.

here i choose to reload with dependencies

which causes an error

C:\UNITYPROJECTS\ubiquitous_computing\ArUcoDetectionHoloLens-Unity\ArUcoDetectionHoloLensUnity\build\ArUcoDetectionHoloLensUnity\ArUcoDetectionHoloLensUnity.vcxproj : error  : Cannot load project with duplicated project items: C:\UNITYPROJECTS\ubiquitous_computing\ArUcoDetectionHoloLens-Unity\ArUcoDetectionHoloLensUnity\build\ArUcoDetectionHoloLensUnity\opencv_world3411.dll is included as 'None' and as 'None' item types. 

this isn't my original issue but it seems I've failed to reproduce it.
Still an issue non the less.

Thank you to any who could help me with this issue!

[edit] description of earlier issues. from memory and older pictures! only usefull as context!!

I used to be able to reload the project. When i would try to build and deploy VS19 would warn me my build tools were outdated and MUST be updated.
Doing so resulted in a warning when building (luckily isaved an image of this)

i tried to see what happens when i ignore this warning.

this resulted in an architecture mismatch

i think i navigated to the opencv submodule and copied the ARM dll and winmd but cannot be sure.

eventually i could deploy the example but it threw a NullReference exception.
I am looking forward to any kind of input you could give me.

Hi,
My cube on my marker always moves a little.
I would like it to be more stable to do real time measurement.
Do you have any ideas to solve that ?

Thank you for this repository. It's really helpful. Following is the error that I am getting. Would appreciate any comments.

Submodule 'HoloLensCamCalib' (https://github.com/doughtmw/HoloLensCamCalib.git) registered for path 'HoloLensCamCalib' Submodule 'HoloLensForCV' ([email protected]:doughtmw/HoloLensForCV.git) registered for path 'HoloLensForCV' Cloning into 'D:/Unity Projects/ArUcoDetectionHoloLens-Unity/HoloLensCamCalib'... Cloning into 'D:/Unity Projects/ArUcoDetectionHoloLens-Unity/HoloLensForCV'... [email protected]: Permission denied (publickey). fatal: Could not read from remote repository.

int patchRadius
At DepthPvMapper.cpp
I can find depthrangeFrom and depthrangeTo,which are 200mm and 7500mm
But,i cannot figure out what is patchRadius.
Can u help me ，thank you so much！

I run the example in my H2 well , but I need to use the higher resolution of camera
like the 19201080 or higher that supported by H2,
Is it possible to change the resolution from 896504 to higher?? Thanks.

Hi

Is it possible to use this package in conjugation with the Mixed Reality Toolkit (MRTK) https://github.com/microsoft/MixedRealityToolkit-Unity?

I would like to use some build-in gesture features.

Thanks in advance!

Best regards

When I restart the detection algorithm, I get the following issue:

Exception thrown at 0x76E4E2AF in Visual Tag Detector.exe: Microsoft C++ exception: Platform::COMException ^ at memory location 0x0234F428.
HRESULT:0x80072740 Only one usage of each socket address (protocol/network address/port) is normally permitted.
WinRT information: Only one usage of each socket address (protocol/network address/port) is normally permitted.

How do I solve this?

Hi,

first of all, thank you for your awesome work with this repo.

I am currently using this repo to display a streamed live pointcloud with an aruco marker as anchor. The performance is quite bad while detecting and simultaneously displaying the pointcloud, understandably.
It gets better when I double tap, however, I found that just displaying the pointcloud without any aruco detection is quite a bit faster even after double tapping. After double tap its about 30fps. Without any aruco stuff I get to almost 90fps.
In general it is still smooth when moving around in the pointcloud. It just seems that rendering a new pointcloud is slower.

Do you have any idea how I could diagnose that problem, or even already an idea why that might be?

The pointcloud code is exactly the same in both scenarios.

Respected People,

I am currently working on a project (with Hololens 2)
which demands the Arucomarker detection and therefore came across this
Publication and tried incorporating it in my Project. I followed each and every
Instruction as per the publication. Upon deploying the app on Hololens 2
I am able to see only this message saying “ Initializing MediaFrame Source Groups”.
When I face my Camera at the Arucomarker available on this Publication
(https://doughtmw.github.io/posts/ArUco-Detection-HoloLens-4) I am not able to view the Cube.
I believe I am missing some Detail which I am not sure.

I just have some doubts and confusions regarding this:

1. I am basically deploying this app from unity, so does this project need the OpenCV plugin
   (https://assetstore.unity.com/packages/tools/integration/opencv-for-unity-21088) from the
   Unity Asset store? or is the Nuget package enough?

2. Does this Project need a Camera Calibration as such for the Aruco Marker detection to Work or should I have to manually
   choose the Camera Resolution for the Research Mode?

3. I understand we need a specific Marker ID for the Aruco Marker to function but I am not
   able to see one in your Project?

It would be really a great Help for me if someone could help me with these Questions. Since I am very new to this
Technology, I am having a hard time.

Thank You!

Hi, I am working with the Short Throw ToF Reflectivity stream. When I try to access it from unity via a mediaframegroup backed by a multiframe buffer I get an out of bound error from the windows runtime originating from the following line in MediaFrameReaderContext.cpp

Platform::Object^ mfMtUserData =
                frame->Properties->Lookup(c_MFSampleExtension_Spatial_CameraProjectionTransform);

I tried just replacing the transform with the identify matrix, but when I go further and pass the frame to opencv for aruco marker detection I fail an opencv assertion that the frame not be empty. I think this is likely some error with regards to accessing the Short Throw ToF Reflectivity sensor via the Unity il2cpp scripting backend, as I've had no trouble accessing the stream on UWP.

Hello, first of all thanks for this cool project! I tried your sample without any modification, the cube correctly appears on top of the marker.
I'm currently trying to detect the marker using the light right front camera, without any success.
I followed the steps you included to use research mode sensors, but when i run i obtain the following exceptions:

ArUcoDetectionHoloLensUnity.ArUcoMarkerDetection:Start()
...
HoloLensForCVUnity.ArUcoDetection.StartHoloLensMediaFrameSourceGroup: Starting the media frame source group
UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
...
Exception thrown at 0x7743C7D2 (KernelBase.dll) in Aruco_Hololens.exe: 0x40080202: WinRT transform error (parameters: 0x8000000B, 0x80070490, 0x00000014, 0x00DCD4E8).
'Aruco_Hololens.exe' (Win32): caricamento di 'C:\Windows\System32\ksuser.dll' completato.
...
MediaFrameSourceGroup::TryInitializeMediaCaptureAsync: MediaCapture is successfully initialized in shared mode.
...
Exception thrown at 0x7743C7D2 (KernelBase.dll) in Aruco_Hololens.exe: WinRT originate error - 0xC00DABE0 : 'No capture devices are available.'.
...
MediaFrameSourceGroup::GetSensorType:: found sensor name 'Visible Light Right-Front' in MF_MT_USER_DATA (blob has 52 bytes)
MediaFrameSourceGroup::GetSubtypeForFrameReader: evaluating MediaFrameSourceKind::Depth with format Video-ARGB32 @30/1Hz
...
Exception thrown at 0x5CCAE1FE (HoloLensForCV.dll) in Aruco_Hololens.exe: 0xC0000005: Access violation reading location 0x00000020.

Could you please help me?