Comments (8)
The current research mode APIs can not be used from c#. They are c++ only. I think it is possible to write a C++ unity plugin that would access the cameras using native c++.
Yes, you can do that. You need to wrap the needed API calls in accessible functions in your DLL, via __declspec(dllexport)
. In Unity you can import these with [DllImport("myDll.dll", EntryPoint = "-dll funtion name-")]
and use them.
I currently rebuild my DLL so that it fits our requirements. But generally you need a function to initialize the sensors and some to query the respective data. you can orientate on the SensorVisualization sample for that or have a look in the API doc
from hololens2forcv.
The current research mode APIs can not be used from c#. They are c++ only. I think it is possible to write a C++ unity plugin that would access the cameras using native c++.
from hololens2forcv.
Another possibility is using COM interop in .NET: https://docs.microsoft.com/en-us/dotnet/standard/native-interop/cominterop .
Visual studio has a C# Holographic DirectX11 App(Universal WIndows) template. This shows using COM interop in .NET with DX11 interfaces. Template instructions at:
from hololens2forcv.
Thank you for the information. Over the last days I managed to deploy a simple C++ dll on the HoloLens. But I sadly donβt have the skills I need to create a dll for the research mode.
Do you know if there are plans to create an unity plugin?
from hololens2forcv.
I found two C++ Unity Plugins HoloLens2-Unity-ResearchModeStreamer and HoloLens2-ResearchMode-Unity that provide some features of the Research Mode using a DLL, they should work fine.
I tried enabling the research mode in C# and .NET using COM interop (https://docs.microsoft.com/en-us/dotnet/standard/native-interop/cominterop) as mentioned by @dorinung. Using the following snippets, I'm aquiring an IntPtr to the unmanaged ResearchModeSensorDevice:
IntPtr hrResearchMode = NativeAPI.LoadLibraryA("ResearchModeAPI");
if (hrResearchMode == IntPtr.Zero) throw new InvalidOperationException("The ResearchModeAPI library could not be loaded.");
IntPtr procAddress = NativeAPI.GetProcAddress(hrResearchMode, "CreateResearchModeSensorDevice");
if (procAddress == IntPtr.Zero) throw new InvalidOperationException("Procedure CreateResearchModeSensorDevice not found.");
CreateResearchModeSensorDevice pfnCreate = (CreateResearchModeSensorDevice) Marshal.GetDelegateForFunctionPointer(procAddress, typeof(CreateResearchModeSensorDevice));
Marshal.ThrowExceptionForHR(pfnCreate(out IntPtr device));
Guid guid = typeof(IResearchModeSensorDevicePerception).GUID;
int queryInterface = Marshal.QueryInterface(device, ref guid, out IntPtr devicePerception);
Marshal.ThrowExceptionForHR(queryInterface);
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
delegate int CreateResearchModeSensorDevice([Out] out IntPtr sensorDevice);
static class NativeAPI
{
[DllImport("kernel32.dll")]
public static extern IntPtr LoadLibrary(string dllToLoad);
[DllImport("kernel32.dll", CharSet = CharSet.Ansi, ExactSpelling = true)]
public static extern IntPtr LoadLibraryA(string lpLibFileName);
[DllImport("kernel32.dll")]
public static extern IntPtr GetProcAddress(IntPtr hModule, string procedureName);
[DllImport("kernel32.dll")]
public static extern bool FreeLibrary(IntPtr hModule);
}
But I can't use Marshal.GetObjectForIUnknown(devicePerception)
(API), which leads to an InvalidCastException. The HRESULT error code corresponds to 0x80004002
(E_NOINTERFACE). Using Marshal.GetTypedObjectForIUnknown(devicePerception, typeof(IResearchModeSensorDevicePerception)
(API) leads to a MissingMethodException)
[ComImport]
[Guid("C1678F4B-ECB4-47A8-B6FA-97DBF4417DB2")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IResearchModeSensorDevicePerception
{
[PreserveSig]
int GetRigNodeId([Out] out Guid rigNodeId);
}
I took IResearchModeSensorDevicePerception
instead of IResearchModeSensorDevice
for simplicity, but tried the above with both interfaces. I'm stuck here, does anyone have an idea or hint?
from hololens2forcv.
Had some issues with marshaling too and ended up using an unsafe block to circumvent them and just worked with pointers in C#:
...
unsafe {
// get data from dll
SensorDataStruct* dataPointer = getAccDataBatched();
// save it in struct
for (int i = 0; i < ACCBATCHSIZE; i++) {
accData.timestamp = dataPointer[i].VinylHupTicks;
accData.sensorData = new Vector3(dataPointer[i].SensorValueX, dataPointer[i].SensorValueY, dataPointer[i].SensorValueZ);
...
// fire event with new sensor data
...
}
}
...
The sruct on the plugin side looks like this:
extern "C" {
//##### Structs #####
struct SensorDataStruct {
uint64_t VinylHupTicks;
float SensorValueX;
float SensorValueY;
float SensorValueZ;
};
//##### Class #####
class DllExports {
public:
DllExport static SensorDataStruct* GetAccDataBatched();
...
};
}
from hololens2forcv.
@dakesse
I built my own dll below, but when I imported the dll file in unity and deployed it to hololens2, there was something wrong . The program stopped at OpenAccSensorStream().
I 'm kind of confused where the error is.Could you help me?
`
#pragma once
#include "pch.h"
#include "ResearchModeAPI.h"
#ifdef myResearchMode
#define DllExport _declspec(dllexport)
#else
#define DllExport _declspec(dllimport)
#endif
#ifdef _cplusplus
extern "C"
{
#endif
struct SensorDataStruct
{
uint64_t VinylHupTicks;
uint64_t SocTicks;
float SensorValueX;
float SensorValueY;
float SensorValueZ;
float temperature;
};
class DllExport DllExports
{
public:
static void InitSensors();
static void OpenAccSensorStream();
static SensorDataStruct *GetAccDataBatched();
static void CloseAccSensorStream();
static void ReleaseSensors();
static void ImuAccessOnComplete(ResearchModeSensorConsent consent);
};
//##### Global Vars #####
// static vars for research mode init
static IResearchModeSensorDeviceConsent* m_pSensorDeviceConsent;
static IResearchModeSensorFrame *pSensorFrame;
static IResearchModeSensorDevice *m_pSensorDevice;
static std::vector<ResearchModeSensorDescriptor> m_sensorDescriptors;
static IResearchModeAccelFrame *m_pSensorAccelFrame;
// static vars for sensor access
static IResearchModeSensor *m_pAcclSensor;
HRESULT hr = S_OK;
#ifdef _cplusplus
}
#endif
`
and the .cpp file like this
#define myResearchMode
#include "pch.h"
#include "myResearchMode.h"
extern "C"
HMODULE LoadLibraryA(
LPCSTR lpLibFileName
);
static ResearchModeSensorConsent imuAccessCheck;
static HANDLE imuConsentGiven;
_declspec(dllexport) void ImuAccessOnComplete(ResearchModeSensorConsent consent)
{
imuAccessCheck = consent;
SetEvent(imuConsentGiven);
}
_declspec(dllexport) void InitSensors()
{
size_t sensorCount = 0;
imuConsentGiven = CreateEvent(nullptr, true, false, nullptr);
HMODULE hrResearchMode = LoadLibraryA("ResearchModeAPI");
if (hrResearchMode)
{
typedef HRESULT(__cdecl* PFN_CREATEPROVIDER) (IResearchModeSensorDevice** ppSensorDevice);
PFN_CREATEPROVIDER pfnCreate = reinterpret_cast<PFN_CREATEPROVIDER>(GetProcAddress(hrResearchMode, "CreateResearchModeSensorDevice"));
if (pfnCreate)
{
hr = pfnCreate(&m_pSensorDevice);
}
else
{
hr = E_INVALIDARG;
}
}
m_pSensorDevice->QueryInterface(IID_PPV_ARGS(&m_pSensorDeviceConsent));
m_pSensorDeviceConsent->RequestIMUAccessAsync(ImuAccessOnComplete);
m_pSensorDevice->DisableEyeSelection();
m_pSensorDevice->GetSensorCount(&sensorCount);
m_sensorDescriptors.resize(sensorCount);
m_pSensorDevice->GetSensorDescriptors(m_sensorDescriptors.data(), m_sensorDescriptors.size(), &sensorCount);
for (auto& sensorDescriptor : m_sensorDescriptors)
{
if (sensorDescriptor.sensorType == IMU_ACCEL)
{
m_pSensorDevice->GetSensor(sensorDescriptor.sensorType, &m_pAcclSensor);
}
}
}
_declspec(dllexport) void OpenAccSensorStream()
{
m_pAcclSensor->GetNextBuffer(&pSensorFrame);
hr = pSensorFrame->QueryInterface(IID_PPV_ARGS(&m_pSensorAccelFrame));
}
_declspec(dllexport) SensorDataStruct* GetAccDataBatched()
{
// get the sensor Frame and next buffer
const AccelDataStruct* accelBuffer;
size_t BufferOutLength;
// read sensor values ...
hr = m_pSensorAccelFrame->GetCalibratedAccelarationSamples(&accelBuffer, &BufferOutLength);
SensorDataStruct* data = new SensorDataStruct[BufferOutLength]; // ?
// β¦ and copy them to return array
for (UINT i = 0; i < BufferOutLength; i++)
{
data[i].VinylHupTicks = accelBuffer[i].VinylHupTicks;
data[i].SocTicks = accelBuffer[i].SocTicks;
data[i].SensorValueX = accelBuffer[i].AccelValues[0];
data[i].SensorValueY = accelBuffer[i].AccelValues[1];
data[i].SensorValueZ = accelBuffer[i].AccelValues[2];
data[i].temperature = accelBuffer[i].temperature;
}
return data;
}
_declspec(dllexport) void CloseAccSensorStream()
{
m_pAcclSensor->CloseStream();
}
_declspec(dllexport) void ReleaseSensors()
{
m_pSensorAccelFrame->Release();
}
from hololens2forcv.
Sorry for my very late response. As far as I could see the difference between our approaches is that you export the whole class into the DLL. I can remember that I had some issues with that, so I exported every element/function of the class instead the class itself.
So:
class DllExports {
public:
DllExport static void InitSensors();
DllExport static void OpenAccSensorStream();
DllExport static void GetAccDataStamped(float& accX, float& accY, float& accZ, UINT64& timestamp);
DllExport static void CloseAccSensorStream();
....
DllExport static void ReleaseSensors();
};
instead of:
class DllExport DllExports {
public:
static void InitSensors();
static void OpenAccSensorStream();
static SensorDataStruct *GetAccDataBatched();
static void CloseAccSensorStream();
static void ReleaseSensors();
static void ImuAccessOnComplete(ResearchModeSensorConsent consent);
};
from hololens2forcv.
Related Issues (20)
- variable sampling frequency of rgb and depth cameras in StreamRecorder application
- CameraWithCVAndCalibration runtime error: Unable to activate Microsoft Store app HOT 3
- Can I get spatial mapping directly from the Research Mode?
- PV camera frame's origin HOT 1
- Interpreting depth camera data
- Samples cannt install after the system is updated HOT 1
- Explanation about vectors in StreamRecorder
- sample code StreamRecorder can compile on unity?
- Low resolution on PV stream HOT 1
- Long Throw mode resolution
- Undistort frames from LF camera
- Any ways to extract the TSDF volume?
- A question about getting the 3D coordinates in world space coordinates system according to a pv image point HOT 1
- HOLOLENS 2 RESEARCH MODE COORDINATE SYSTEM AND ROTOTRANSLATIONS BETWEEN SENSORS
- Low-signal depth mode in the Windows 11 update
- problem with git lfs
- Internal Error when Deploying Sensor Visualization(twinapi.core.dll)
- MediaFrameReader, no FrameArrived callback
- What does head_hand_eye.csv file contains?
- Calling hololens2's depth camera in ue4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hololens2forcv.