Filtering noise from depth image/point cloud about realsense-ros HOT 17 CLOSED

Hi @aravindpreshant and @chwimmer. Sorry about the delay. The code for median filtering is here and the code for downsampling/denoising using the voxel filter is here. I hope this helps in some way.

from realsense-ros.

Hey @aravindpreshant. As I described on my earlier post, I first applied a median filter to the depth image, converted that to a point cloud and then applied the VoxelGrid filter form PCL 1.7.2 for which the setMinimumPointsNumberPerVoxel(...) was set to some number that made sense for our cloud (30 I think).

There are two things to note here: 1) The resulting cloud is very low resolution since the VoxelGrid filter only keeps Voxel centroids* and 2) All of this was done in ROS (so the depth image to point cloud conversion came from a nodelet that comes with the PCL ROS package).

Are you using ROS? If yes then I can share some code snippets that should help you get a better cloud.

*Note that a) you don't have to use the VoxelGrid step since the median filter takes care of much of the noise and b) it should be possible to implement your own similar filter that removes the points associated with sparsely populated voxels and keeps the remaining points. Note that the method essentially does very simplistic clustering and does it fast by bypassing any nearest neighbor searches. A better result may be achieved using either an EM based fast clustering routine or by finding regions in the depth image and removing them if they don't meet some size criteria. I would probably do the latter if I had more time.

from realsense-ros.

@reaganlo We have a solution that works for our use case, but is not ideal as there is a lot of information loss due to downsampling. As stated before, there are many use cases where having a noise free point cloud is important, in particular 3d navigation + motion planning.

It would be a lot more helpful for the ROS community if it works out-of-the-box -- so I don't feel it should be closed -- unless #106 fixes it. We haven't had the chance to try the new feature yet but will soon.

If #106 doesn't fix it -- my recommendation is that you guys add as part of the launch file an optional nodelet that can apply a median filter to the depth image.

from realsense-ros.

@mirzashah The point cloud with "ghost points", are you referring to
-- the one that is generated by the RealSense nodelet itself
OR
-- the one that is generated by the "depth_image_proc/point_cloud_xyzrgb" nodelet in rgbd_launch using the native depth and color streams from the RealSense nodelet.

Just wanted to confirm.

from realsense-ros.

@reaganlo The ghost points appear in both -- the default topics are /camera/depth/image (what the RealSense nodelet produces) and /camera/depth/points (which depth_image_proc/point_cloud_xyzrgb produces).

You can compare them in rviz by using the DepthCloud and PointCloud displays respectively. To me, this is an issue in the depth image generated by the RealSense. For example, if you point the sensor at the ground, it clearly sees it and generates a nice point cloud covering the ground. However, it also generates these ghost points between the floor and the camera and well below the floor as well particularly as you get further from the camera.

The depth image needs to be cleaned, one approach that comes to mind is doing a nearest neighbor search on the depth image and removing cells that are not at similar depths to neighboring cells or in isolation. I couldn't find any out-of-the-box filters that work directly on depth image sin ROS.

Just confirming, do you also see this phenomenon? If you have no idea what I'm talking about, I'll take a screen capture video and send you a link here. It's the middle of the night here, will take the capture tomorrow. I can show you the difference between a clean point cloud generated by the statistical outlier filter and the one the RealSense is generating.

Note that I'm using the latest firmware and using everything stock out-of-the-box with your driver. This phenomenon happens at 640x480 as well as 320x240. I tried playing with the controls you guys exposed with dynamic_reconfigure (e.g. camera gains) to see if those things could clean the noise -- no luck.

from realsense-ros.

@mirzashah we have an open investigation task for this. Stay tuned.

from realsense-ros.

@ghindman Awesome thanks! We've been working on it ourselves...one of our engineers @safijari has had luck with an OpenCV median filter directly on the depth image. Computationally much more efficient but we're not able to remove all noise yet.

Also fyi - if you look at the depth image in greyscale in image_view, anywhere you see these white speckles flash -- those are the noise.

from realsense-ros.

Just an update on @mirzashah's application: Some of the noise appears to be salt and pepper noise. A median filter on the depth image takes care of most of that. There is still other noise present that appears to be near the edges/transitions between two depth levels. For our application we were able to do some voxel based downsampling using PCL and remove any voxels that contain less than 15 points (that number is highly empirical). If someone needs to efficiently clean a point cloud (albeit at low resolutions) we can share how that's accomplished.

from realsense-ros.

@ghindman @reaganlo @mdhorn

Here is a video showing you the difference between the filtered and unfiltered point clouds on our robot -- hope it makes the problem clearer.

https://www.dropbox.com/s/47k1rbrvfie5gpd/realsense_filter.mp4?dl=0

The video shows a "downsample depth image -> median filter depth image -> project point cloud -> voxel filter point cloud" chain approach. Note that this approach works for our use case of robot navigation and is computationally efficient -- however it comes at the cost of losing a lot of detail in the point cloud. Somebody doing 3d mapping, 3d object recognition, gesture recognition, or anything that requires more detailed point clouds will not be happy with this approach. I wouldn't close this issue until a better approach is found.

from realsense-ros.

@safijari : could you please share how you cleaned up the point cloud?

from realsense-ros.

Thanks a lot for the input @safijari. Yes we are using ROS. It would be great if you could share the code snippet.

from realsense-ros.

I'm facing the same problem currently, so I would also be interested in the code snippet.
For me the highest computationally intensive part was the conversion from a PointCloud2 to a pcl::PointCloud - Type. So I wrote my own conversion and its working considerably better, but because of using three cameras and merging the pointclouds its still not good enough. But perhaps its also for you a hint how you can improve your performance.

from realsense-ros.

@chwimmer would you consider sharing your conversion? Perhaps posting about it in the PCL mailing list :) ?

from realsense-ros.

hi @mirzashah
I am trying to improve the image quality of DEPTH for R200 in Linux with Librealsense. Any clue? thanks

from realsense-ros.

@l3s777 that's kind of vague -- quality of what? We were able to filter out the noise as described above. If you're using librealsense directly and not the ROS driver, you should post your question at github.com/intelrealsense/librealsense

from realsense-ros.

@mirzashah I see that you came up with a solution for this issue. Can this be closed?

from realsense-ros.

@reaganlo @I0x0I -- #106 fixes the problem! Thank you guys so much :) Closing this issue

from realsense-ros.