Git Product home page Git Product logo

Comments (8)

dakesse avatar dakesse commented on May 14, 2024 2

The current research mode APIs can not be used from c#. They are c++ only. I think it is possible to write a C++ unity plugin that would access the cameras using native c++.

Yes, you can do that. You need to wrap the needed API calls in accessible functions in your DLL, via __declspec(dllexport). In Unity you can import these with [DllImport("myDll.dll", EntryPoint = "-dll funtion name-")] and use them.
I currently rebuild my DLL so that it fits our requirements. But generally you need a function to initialize the sensors and some to query the respective data. you can orientate on the SensorVisualization sample for that or have a look in the API doc

from hololens2forcv.

dorinung avatar dorinung commented on May 14, 2024

The current research mode APIs can not be used from c#. They are c++ only. I think it is possible to write a C++ unity plugin that would access the cameras using native c++.

from hololens2forcv.

dorinung avatar dorinung commented on May 14, 2024

Another possibility is using COM interop in .NET: https://docs.microsoft.com/en-us/dotnet/standard/native-interop/cominterop .
Visual studio has a C# Holographic DirectX11 App(Universal WIndows) template. This shows using COM interop in .NET with DX11 interfaces. Template instructions at:

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/native/creating-a-holographic-directx-project

image

from hololens2forcv.

autimator avatar autimator commented on May 14, 2024

Thank you for the information. Over the last days I managed to deploy a simple C++ dll on the HoloLens. But I sadly don’t have the skills I need to create a dll for the research mode.

Do you know if there are plans to create an unity plugin?

from hololens2forcv.

cookieofcode avatar cookieofcode commented on May 14, 2024

I found two C++ Unity Plugins HoloLens2-Unity-ResearchModeStreamer and HoloLens2-ResearchMode-Unity that provide some features of the Research Mode using a DLL, they should work fine.

I tried enabling the research mode in C# and .NET using COM interop (https://docs.microsoft.com/en-us/dotnet/standard/native-interop/cominterop) as mentioned by @dorinung. Using the following snippets, I'm aquiring an IntPtr to the unmanaged ResearchModeSensorDevice:

IntPtr hrResearchMode = NativeAPI.LoadLibraryA("ResearchModeAPI");
if (hrResearchMode == IntPtr.Zero) throw new InvalidOperationException("The ResearchModeAPI library could not be loaded.");
IntPtr procAddress = NativeAPI.GetProcAddress(hrResearchMode, "CreateResearchModeSensorDevice");
if (procAddress == IntPtr.Zero) throw new InvalidOperationException("Procedure CreateResearchModeSensorDevice not found.");
CreateResearchModeSensorDevice pfnCreate = (CreateResearchModeSensorDevice) Marshal.GetDelegateForFunctionPointer(procAddress, typeof(CreateResearchModeSensorDevice));
Marshal.ThrowExceptionForHR(pfnCreate(out IntPtr device));

Guid guid = typeof(IResearchModeSensorDevicePerception).GUID;
int queryInterface = Marshal.QueryInterface(device, ref guid, out IntPtr devicePerception);
Marshal.ThrowExceptionForHR(queryInterface);
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
delegate int CreateResearchModeSensorDevice([Out] out IntPtr sensorDevice);
static class NativeAPI
{
     [DllImport("kernel32.dll")]
     public static extern IntPtr LoadLibrary(string dllToLoad);

     [DllImport("kernel32.dll", CharSet = CharSet.Ansi, ExactSpelling = true)]
     public static extern IntPtr LoadLibraryA(string lpLibFileName);

     [DllImport("kernel32.dll")]
     public static extern IntPtr GetProcAddress(IntPtr hModule, string procedureName);

     [DllImport("kernel32.dll")]
     public static extern bool FreeLibrary(IntPtr hModule);
}

But I can't use Marshal.GetObjectForIUnknown(devicePerception) (API), which leads to an InvalidCastException. The HRESULT error code corresponds to 0x80004002 (E_NOINTERFACE). Using Marshal.GetTypedObjectForIUnknown(devicePerception, typeof(IResearchModeSensorDevicePerception) (API) leads to a MissingMethodException)

[ComImport]
[Guid("C1678F4B-ECB4-47A8-B6FA-97DBF4417DB2")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IResearchModeSensorDevicePerception
{
     [PreserveSig]
     int GetRigNodeId([Out] out Guid rigNodeId);
}

I took IResearchModeSensorDevicePerception instead of IResearchModeSensorDevice for simplicity, but tried the above with both interfaces. I'm stuck here, does anyone have an idea or hint?

from hololens2forcv.

dakesse avatar dakesse commented on May 14, 2024

Had some issues with marshaling too and ended up using an unsafe block to circumvent them and just worked with pointers in C#:

...
 unsafe {
   // get data from dll
   SensorDataStruct* dataPointer = getAccDataBatched();

   // save it in struct
   for (int i = 0; i < ACCBATCHSIZE; i++) {
     accData.timestamp = dataPointer[i].VinylHupTicks;
     accData.sensorData = new Vector3(dataPointer[i].SensorValueX, dataPointer[i].SensorValueY, dataPointer[i].SensorValueZ);

     ...
     // fire event with new sensor data
     ...
   }
 }
...

The sruct on the plugin side looks like this:

extern "C" {
  //##### Structs #####
  struct SensorDataStruct {
    uint64_t VinylHupTicks;
    float SensorValueX;
    float SensorValueY;
    float SensorValueZ;
  };

  //##### Class #####
  class DllExports {
public:
    DllExport static SensorDataStruct* GetAccDataBatched();

    ...
  };
}

from hololens2forcv.

JiangTao100 avatar JiangTao100 commented on May 14, 2024

@dakesse
I built my own dll below, but when I imported the dll file in unity and deployed it to hololens2, there was something wrong . The program stopped at OpenAccSensorStream().
I 'm kind of confused where the error is.Could you help me?
`
#pragma once

#include "pch.h"
#include "ResearchModeAPI.h"

#ifdef myResearchMode
#define DllExport _declspec(dllexport)
#else
#define DllExport _declspec(dllimport)
#endif

#ifdef _cplusplus
extern "C"
{
#endif
struct SensorDataStruct
{
uint64_t VinylHupTicks;
uint64_t SocTicks;
float SensorValueX;
float SensorValueY;
float SensorValueZ;
float temperature;
};
class DllExport DllExports
{
public:
static void InitSensors();
static void OpenAccSensorStream();

		static SensorDataStruct *GetAccDataBatched();

		static void CloseAccSensorStream();

		static void ReleaseSensors();

		static void ImuAccessOnComplete(ResearchModeSensorConsent consent);

};
	//##### Global Vars #####
	// static vars for research mode init
       static IResearchModeSensorDeviceConsent* m_pSensorDeviceConsent;
	static IResearchModeSensorFrame *pSensorFrame;

	static IResearchModeSensorDevice *m_pSensorDevice;
	static std::vector<ResearchModeSensorDescriptor> m_sensorDescriptors;		
	static IResearchModeAccelFrame *m_pSensorAccelFrame;

	// static vars for sensor access
	static IResearchModeSensor *m_pAcclSensor;
	
	HRESULT hr = S_OK;

#ifdef _cplusplus
}
#endif
`
and the .cpp file like this

#define myResearchMode

#include "pch.h"
#include "myResearchMode.h"

extern "C"
HMODULE LoadLibraryA(
LPCSTR lpLibFileName
);
static ResearchModeSensorConsent imuAccessCheck;
static HANDLE imuConsentGiven;

_declspec(dllexport) void ImuAccessOnComplete(ResearchModeSensorConsent consent)
{
imuAccessCheck = consent;
SetEvent(imuConsentGiven);
}

_declspec(dllexport) void InitSensors()

{
size_t sensorCount = 0;
imuConsentGiven = CreateEvent(nullptr, true, false, nullptr);

HMODULE hrResearchMode = LoadLibraryA("ResearchModeAPI");
if (hrResearchMode)
{
	
	typedef HRESULT(__cdecl* PFN_CREATEPROVIDER) (IResearchModeSensorDevice** ppSensorDevice);
	PFN_CREATEPROVIDER pfnCreate = reinterpret_cast<PFN_CREATEPROVIDER>(GetProcAddress(hrResearchMode, "CreateResearchModeSensorDevice"));
	if (pfnCreate)
	{
		hr = pfnCreate(&m_pSensorDevice);
	}
	else
	{
		hr = E_INVALIDARG;
	}

}
m_pSensorDevice->QueryInterface(IID_PPV_ARGS(&m_pSensorDeviceConsent));
m_pSensorDeviceConsent->RequestIMUAccessAsync(ImuAccessOnComplete);

m_pSensorDevice->DisableEyeSelection();

m_pSensorDevice->GetSensorCount(&sensorCount);
m_sensorDescriptors.resize(sensorCount);

m_pSensorDevice->GetSensorDescriptors(m_sensorDescriptors.data(), m_sensorDescriptors.size(), &sensorCount);

for (auto& sensorDescriptor : m_sensorDescriptors)
{
	if (sensorDescriptor.sensorType == IMU_ACCEL)
	{
		
		m_pSensorDevice->GetSensor(sensorDescriptor.sensorType, &m_pAcclSensor);
	}
}

}

_declspec(dllexport) void OpenAccSensorStream()
{

m_pAcclSensor->GetNextBuffer(&pSensorFrame);
hr = pSensorFrame->QueryInterface(IID_PPV_ARGS(&m_pSensorAccelFrame));

}

_declspec(dllexport) SensorDataStruct* GetAccDataBatched()
{

// get the sensor Frame and next buffer
	const AccelDataStruct* accelBuffer;
    size_t BufferOutLength;
	// read sensor values ...
	hr = m_pSensorAccelFrame->GetCalibratedAccelarationSamples(&accelBuffer, &BufferOutLength);

	SensorDataStruct* data = new SensorDataStruct[BufferOutLength];  // ?
	// … and copy them to return array
	for (UINT i = 0; i < BufferOutLength; i++)
	{
		data[i].VinylHupTicks = accelBuffer[i].VinylHupTicks;
		data[i].SocTicks = accelBuffer[i].SocTicks;
		data[i].SensorValueX = accelBuffer[i].AccelValues[0];
		data[i].SensorValueY = accelBuffer[i].AccelValues[1];
		data[i].SensorValueZ = accelBuffer[i].AccelValues[2];
		data[i].temperature = accelBuffer[i].temperature;
	}
	return data;

}

_declspec(dllexport) void CloseAccSensorStream()
{
m_pAcclSensor->CloseStream();
}

_declspec(dllexport) void ReleaseSensors()
{

m_pSensorAccelFrame->Release();

}

from hololens2forcv.

dakesse avatar dakesse commented on May 14, 2024

Sorry for my very late response. As far as I could see the difference between our approaches is that you export the whole class into the DLL. I can remember that I had some issues with that, so I exported every element/function of the class instead the class itself.
So:

class DllExports {
  public:
    DllExport static void InitSensors();
    DllExport static void OpenAccSensorStream();
    DllExport static void GetAccDataStamped(float& accX, float& accY, float& accZ, UINT64& timestamp);
    DllExport static void CloseAccSensorStream();
    ....
    DllExport static void ReleaseSensors();
};

instead of:

class DllExport DllExports {
  public:
    static void InitSensors();
    static void OpenAccSensorStream();
    static SensorDataStruct *GetAccDataBatched();
    static void CloseAccSensorStream();
    static void ReleaseSensors();
    static void ImuAccessOnComplete(ResearchModeSensorConsent consent);
};

from hololens2forcv.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.