Comments (11)
Also, I've compiled the library with Kinect v2 support (undefined COMPILE_WITHOUT_Kinect2
).
I noticed that in Kinect2Engine.cpp, all the important stuff are commented out:
void Kinect2Engine::getImages(ITMUChar4Image *rgbImage, ITMShortImage *rawDepthImage)
{
//Vector4u *rgb = rgbImage->GetData(MEMORYDEVICE_CPU);
//if (colorAvailable)
//{
//}
//else memset(rgb, 0, rgbImage->dataSize * sizeof(Vector4u));
//float *depth = out->depth->GetData(MEMORYDEVICE_CPU);
//if (depthAvailable)
//{
// IDepthFrame* pDepthFrame = NULL;
// UINT16 *pBuffer = NULL;
// UINT nBufferSize = 0;
// HRESULT hr = data->depthFrameReader->AcquireLatestFrame(&pDepthFrame);
// if (SUCCEEDED(hr))
// {
// if (SUCCEEDED(hr))
// hr = pDepthFrame->AccessUnderlyingBuffer(&nBufferSize, &pBuffer);
// if (SUCCEEDED(hr))
// {
// for (int i = 0; i < imageSize_d.x * imageSize_d.y; i++)
// {
// ushort depthPix = pBuffer[i];
// depth[i] = depthPix == 0 ? -1.0f : (float)depthPix / 1000.0f;
// }
// }
// }
// SafeRelease(pDepthFrame);
//}
//else memset(depth, 0, out->depth->dataSize * sizeof(short));
//out->inputImageType = ITMView::InfiniTAM_FLOAT_DEPTH_IMAGE;
return /*true*/;
}
What's going on here?
from infinitam.
OK, I managed to get something out of it by modifying the function getImages
a bit. Now I do get reconstruction with Kinect v2. However, it seems that tracking is not very robust. I imagine that's due to the fact that in the function (in its current form) there is no color information assigned to rgbImage
.
Here's what I have right now:
void Kinect2Engine::getImages(ITMUChar4Image *rgbImage, ITMShortImage *rawDepthImage)
{
// Set all color stuff to 0;
Vector4u *rgb = rgbImage->GetData(MEMORYDEVICE_CPU);
memset(rgb, 0, rgbImage->dataSize * sizeof(Vector4u));
short *depth = rawDepthImage->GetData(MEMORYDEVICE_CPU);
if (depthAvailable)
{
IDepthFrame* pDepthFrame = NULL;
UINT16 *pBuffer = NULL;
UINT nBufferSize = 0;
HRESULT hr = data->depthFrameReader->AcquireLatestFrame(&pDepthFrame);
if (SUCCEEDED(hr))
{
hr = pDepthFrame->AccessUnderlyingBuffer(&nBufferSize, &pBuffer);
for (int i = 0; i < imageSize_d.x * imageSize_d.y; i++)
{
ushort depthPix = pBuffer[i];
depth[i] = depthPix == 0 ? -1 : depthPix;
}
}
SafeRelease(pDepthFrame);
}
else memset(depth, 0, rawDepthImage->dataSize * sizeof(short));
return;
}
Any help from the silent population is greatly welcome :D
from infinitam.
Nope...
I fixed the above code, it now writes pixel colors to rgbImage
as well but still tracking is terrible. Even small movements confuses the system. I'm thinking maybe this could be a calibration problem... Maybe wrong calibration data is loaded (if any data loaded for Kinect v2 at all).
This might also provide more background information:
I tried the newly compiled InfiniTAM library with SemanticPaint. With Kinect v1, everything works the way it is supposed to.
With the fixed Kinect v2 code, SemanticPaint works, but I'm seeing an "offset" between the mouse cursor location and the one rendered on the OpenGL window (look at the image below). I initially thought this could be due to wrong calibration. But I could be wrong (sorry, super new to InfiniTAM and SemanticPaint).
By the way, will submit a pull request with the modifications I made so that others could also use Kinect v2 with InfiniTAM. Just have to clean the code up a bit :)
from infinitam.
The offset in SemanticPaint is likely to be a calibration issue, as you suggest. If you take a look in
https://github.com/torrvision/spaint/blob/master/apps/spaintgui/main.cpp
you'll observe that SemanticPaint is currently hard-coded to use the internal calibration of the camera. You could try turning off internal calibration and loading a proper calibration file for the Kinect v2. I haven't tried SemanticPaint with a Kinect v2 (I don't have one), so I don't have a calibration file for it, but @victorprad or @olafkaehler might. I'll ask them tomorrow.
from infinitam.
Thank you for your response.
Does that possibly explain the tracking problem? I'm not sure whether TAM tracks using depth or RGB data.
The thing is, moving the sensor in any direction by anything more than 10 inches will mess tracking and the reconstruction. No matter how small the movement is.
from infinitam.
- The default tracking in InfiniTAM on its own is set up to use the depth image only. Look at ITMLibSettings to change this, but by default, RGB images are completely ignored, as is the calibration of RGB to depth images. This is not the case for Semantic Paint, where colour is accumulated by default, but it's still not used for tracking.
- The Kinect v2 has a different resolution for its depth images, so it would almost certainly also require different intrinsic parameters for the depth camera. I once calibrated some values like fx 358.435, fy 357.912, cx 256.526, cy 205.474, but they may be arbitrarily wrong for your Kinect v2.
- Apart from that, the Kinect v2 has a time of flight camera rather than a structured light sensor. Due to imperfections in the modulation of the active illumination, it is known that most of these time of flight sensors tend to have biased depth values, e.g. at a distance of 2m, everything is about 5cm closer than measured, at a distance of 3m everything is about 5cm further away than measured. I've never investigated how good the Kinect v2 actually is and the given numbers are just random examples. In practice, we have noticed a few 3D geometry artefacts due to this effect, though, and since this is an inherent property of time of flight sensors, we have largely ignored the Kinect v2 so far. Ideally you'd calibrate the biases for your specific camera and such a calibration should then be used by InfiniTAM, but most of the time we just stick with ASUS, PrimeSense, Occipital StructureSensor or other structured light devices that do not have this problem.
from infinitam.
@olafkaehler Thank you for your response.
I tried your calibration values as well, but the reconstruction still works terribly (I think tracking is lost too quickly). Yes the depth camera's resolution is 512 x 424. Also, I can't be certain the issue you mentioned with ToF camera is the evil here. Say there is a cup on the table. If the sensor is stationary, nothing happens and the model looks fine. As soon as the sensor is rotated a bit so that the cup is translated in the image a bit, the system thinks it's seeing a new cup and will start "desolving" the previous one and will create a new one.
from infinitam.
It really was not so bad with the kinect 2 when I tested (version 1 on
InfiniTAM). Reconstruction worked overall fine. The only problem was that
the reflective areas were not considered noise, but rather developed
very weird artefacts. Maybe have a look at the depth calibration. Make sure
the appropriate conversion is done -- I think the kinect 2 just needs a
division by 1000, whereas for the primesense we used calibration. I've got
a very bad flu right now, but I will recheck the kinect 2 version by the
end of the week, when I get back to work.
On Tuesday, 8 December 2015, Mehran Maghoumi [email protected]
wrote:
@olafkaehler https://github.com/olafkaehler Thank you for your response.
I tried your calibration values as well, but the reconstruction still
works terribly (I think tracking is lost too quickly). Yes the depth
camera's resolution is 512 x 424. Also, I can't be certain the issue you
mentioned with ToF camera is the evil here. Say there is a cup on the
table. If the sensor is stationary, nothing happens and the model looks
fine. As soon as the sensor is rotated a bit so that the cup is translated
in the image a bit, the system thinks it's seeing a new cup and will start
"desolving" the previous one and will create a new one.β
Reply to this email directly or view it on GitHub
#33 (comment)
.
[This email was sent from a wireless mobile device]
from infinitam.
@victorprad That division by 1000 was there (in the commented out code) but they whole thing only worked after I got rid of it. I inspected the values, and they look very similar to the ones obtained by OpenNI from Kinect 1. Would be cool if we could get it to work.
Hope the flu goes away soon and you feel better.
from infinitam.
Hello,
I am trying to use Infinitam with KinectV2 on window 8.1. What is the
right way to compile it? There is no flag concerning Kinect in Cmake
while there is one for REal sense. Do I have to undefine
COMPILE_WITHOUT_Kinect2 and add path manually?
Thanks
from infinitam.
@vestri Yes, undefine COMPILE_WITHOUT_Kinect2 and make sure the kinect 2 api is in the path. Also add the adjustments the OP mentioned.
from infinitam.
Related Issues (20)
- Supporting iPhone Depth Data
- from rgb to depth HOT 10
- Show nothing when running InfiniTAM.exe with files in Teddy HOT 5
- Multi-camera Application HOT 1
- FileBasedTracker doesnβt fuse the scene HOT 2
- How big is the voxel grid? HOT 1
- Saving 3D Reconstruction
- Python bindings HOT 6
- Azure Kinect Support HOT 1
- C1083 Cannot open include file: 'GL/glut.h': No such file or directory HOT 5
- Extracting the current VoxelGrid for an additional processing step HOT 1
- iterations algorithm HOT 1
- Python Buffer access to ITMLocalVBA HOT 1
- Cannot access the voxel blocks on device using Nsight VSE Debugger HOT 1
- ITMLocalVBA alone extractable as 3D Reconstruction? HOT 1
- cannot read images HOT 11
- Which version is the newest HOT 1
- Project doesn't build
- Does InfiniTAM support for MS Azure Kinect (soon)?
- Blensor Scans
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from infinitam.