pointcept / segmentanything3d Goto Github PK
View Code? Open in Web Editor NEW[ICCV'23 Workshop] SAM3D: Segment Anything in 3D Scenes
Home Page: https://arxiv.org/abs/2306.03908
License: MIT License
[ICCV'23 Workshop] SAM3D: Segment Anything in 3D Scenes
Home Page: https://arxiv.org/abs/2306.03908
License: MIT License
Hi there,
Thank you for this excellent work, your code is very clear for something this new. Congrats to the team!
I apologize if this was a lot to read, I just want to be through with my question.
I ran your code on scene0000_00
from the scannet dataset, and the resulting segmentation looks much worse than your results on your github repo. In my results, the segmentation is very messy and oversegmented, leading me to think I am doing something wrong.
See here, where I'm plotting the original colors, sam3d segmentation, instance_gt, semantic_gt20, semantic_gt200.
What's the difference in your results between the SAM3D and the SAM3D Merged output? How is each one produced?
I'm just trying to figure out what I might be doing wrong.
I had to comment out this line
Line 62 in fc8eaca
Otherwise, the code would crash due to a dimension mismatch error between the masks and the image. Why are you resizing the image? Could this be related to my results?
Another thing, since I cannot download the whole scannet dataset, I'm downloading only individual scenes, and that doesn't download/create any intrinsic_depth.txt
file. I am creating it manually from the scans/scene0000_00/scene0000_00.txt
text file, which has content like this:
axisAlignment = 0.945519 0.325568 0.000000 -5.384390 -0.325568 0.945519 0.000000 -2.871780 0.000000 0.000000 1.000000 -0.064350 0.000000 0.000000 0.000000 1.000000
colorHeight = 968
colorToDepthExtrinsics = 0.999973 0.006791 0.002776 -0.037886 -0.006767 0.999942 -0.008366 -0.003410 -0.002833 0.008347 0.999961 -0.021924 -0.000000 0.000000 -0.000000 1.000000
colorWidth = 1296
depthHeight = 480
depthWidth = 640
fx_color = 1170.187988
fx_depth = 571.623718
fy_color = 1170.187988
fy_depth = 571.623718
mx_color = 647.750000
mx_depth = 319.500000
my_color = 483.750000
my_depth = 239.500000
numColorFrames = 5578
numDepthFrames = 5578
numIMUmeasurements = 11834
sceneType = Apartment
Here is the code I am using to make my plots. I noticed you don't have any plotting code. Maybe there is a mistake in mine?
import torch
import open3d as o3d
import numpy as np
# Define the file paths
pcd_filepath = '/pcd_gt_data/train/scene0000_00.pth' # Replace with your file path
pcd_seg_filepath = '/sam_3d_out/scene0000_00.pth' # Replace with your file path
# Load point cloud data
pcd_data = torch.load(pcd_filepath)
seg_data = torch.load(pcd_seg_filepath)
# Get the coordinates
coordinates = pcd_data['coord'].astype('float64') # convert to float64 for o3d
# Get the unique labels in seg_data
unique_labels = np.unique(seg_data)
# Generate random colors for each label
color_map = {label: np.random.rand(3) for label in unique_labels}
# Create an array for colors using the color_map
colors = np.array([color_map[label] for label in seg_data])
# Create a PointCloud object
pcd = o3d.geometry.PointCloud()
# Assign coordinates to the PointCloud object
pcd.points = o3d.utility.Vector3dVector(coordinates)
# Assign colors to the PointCloud object
pcd.colors = o3d.utility.Vector3dVector(colors)
# Visualize the point cloud
o3d.visualization.draw_geometries([pcd])
Thank you for your time and help in advance!
Hello!
During debugging, I found that the shape of the "mask" array here is (240, 320), but this line of code changed the shape of the "color_image" data to (640, 480), making the "color_image[mask]" have the wrong dimension. By annotating this line of code, we can avoid this error. So is it redundant?
Hi,
Thanks for your SAM3D. I have a question about "org_path" in sam3d.py/pcd_ensemble. Could you please explain what the "org_path" parameter in this function refers to?
Thank you
The actual datasets too big to download and run for learning purpose.
Thanks in advance!
I want to use SAM3D for my own dataset which contains outdoor scenes. Is that possible? If it is how can I do that, can you help me with that?
Is this method work on outdoor Scenes? like KITTI or NuScenes dataset
when run sam3d will report intrinsics/intrinsic_depth.txt not found
i check code def export_intrinsics(self, output_path): in SegmentAnything3D/scannet-preprocess/prepare_2d_data/SensorData.py will save intrinsics/intrinsic_depth.txt. but not find prepare_2d_data call this function
Hi,
First of all, thank you for the incredible work on your project!
I have a question regarding the process described in your paper where masks generated from 2D images are projected into 3D space. I'm curious about how this method handles complex scenarios, specifically if there is a pillar standing in the center of the room. Could you please explain how the system manages such cases, and how it handles the projection of such obstacles?
Thank you!
Can someone take a picture of the folder structure after it has been prepared ?
Hey everyone,
I'm diving into SAM3D as a beginner and successfully used it with a scanner dataset using the command:
python sam3d.py --rgb_path $RGB_PATH --data_path $DATA_PATH --save_path $SAVE_PATH --save_2dmask_path $SAVE_2DMASK_PATH --sam_checkpoint_path $SAM_CHECKPOINT_PATH
After running this, I'm unsure about the next steps, particularly with the .pth data, and how to apply it. Now, I'm excited to apply SAM3D to my personal point cloud obtained through photogrammetry.
Could someone guide me through adjusting parameters or any additional steps necessary to use SAM3D with my photogrammetry-generated point cloud? Any advice or assistance would be incredibly helpful!
Thanks a lot!
Hi, I have tried many times to reproduce this work but still not achieving the desired result.
Do you have any other contact information that I would like to ask you for further advice. I hope you can check my code.
Thank you very much!
Hi,
Thanks for developing SAM3D. I have a L515 LiDAR that generates both RGB and Depth frames, or point clouds. How can I use your codes to prepare my own input dataset? I saw a link in the ReadMe, but it is not clear for me. Could you please clarify a little more?
Thank you
Hello!
I would like to use this package to swgment my own pointcloud data. However, it does not contain RGB values. Also, I don't have a pre-segmented ground truth to evaluate the outcome. My questikn is twofold:
is it advisable to use the number of returns/intensity value as a substitute for RGB? Do I need to rescale the values to fall into EGB value ranges?
is there a metric to determine you have accumulated enough training imagesfrom different angles from your pointcloud? I was thinking something like a pointwise-conteibution metric, determining how often a point has been captured in an image? Or, across multiple trainings sequences, stop when segments become more stable across predictions?
Thanks for your excellent works!
I can get the point cloud with different colors in the end. How can I determine which color belongs to which class?
Hey! First thank you for exposing your work :)
Sorry I'm a little confused by what is required to run the script once prepared
Thank you
Hello, how do I get the fourth step in the pipeline results
Thank you for your excellent work!
I wonder how many 2D images are required at minimum for segmenting a 3D point cloud? Thanks!
Hi,
I would like to know more details about how to use SAM to get the segmentation masks on 2D frames and then map them into the 3D space via depth information.
When I use it, SAM sets the following parameters
mask_generator = SamAutomaticMaskGenerator( model=sam, points_per_side=64, pred_iou_thresh=0.86, box_nms_thresh= 0.7, stability_score_thresh=0.92, crop_n_layers=1, crop_n_points_downscale_factor=2, min_mask_region_area=200, # Requires open-cv to run post-processing )
But I can't get the same segmentation effect as you, how can I set the parameters and use the depth map to project to 3D space。
Hope to get your answer.
Thanks.
scannetv2 dataset is too large, do we need to download it all? Or just download these four?
python 1.py -o scannet/ --type _vh_clean_2.ply
python 1.py -o scannet/ --type _vh_clean_2.labels.ply
python 1.py -o scannet/ --type _vh_clean_2.0.010000.segs.json
python 1.py -o scannet/ --type .aggregation.json
Hello
I have requested access to the ScanNet database since last week and I didn't get any answer till now
how can I run the code in this case?
import open3d
leads to Illegal instruction (core dumped)
on some machines. Fix by installing open3d-python
instead of just open3d
Hi There.
Thanks for this excellent repo.
I have a question about the pipeline used. How do you manage to map the masks into the 3D space via depth information?
Do you have any questions about this step? Could you take a look for me? Is it the dataset? I only got fourpython 1.py -o scannet/ --type _vh_clean_2.ply
python 1.py -o scannet/ --type _vh_clean_2.labels.ply
python 1.py -o scannet/ --type _vh_clean_2.0.010000.segs.json
python 1.py -o scannet/ --type .aggregation.json.
python scannet-preprocess/prepare_2d_data/prepare_2d_data.py --scannet_path dataout/train --output_path data/scannetv2_images --export_la
bel_images
Namespace(export_label_images=True, frame_skip=20, label_map_file='D:\Desktop\dianyun\SegmentAnything3D-main\data\scannetv2\scannetv2-labels.combined.tsv', label_type='label-filt
', output_image_height=240, output_image_width=320, output_path='data/scannetv2_images', scannet_path='dataout/train')
Traceback (most recent call last):
File "scannet-preprocess/prepare_2d_data/prepare_2d_data.py", line 118, in
main()
File "scannet-preprocess/prepare_2d_data/prepare_2d_data.py", line 71, in main
label_map = util.read_label_mapping(opt.label_map_file, label_from='id', label_to='nyu40id')
File "D:\Desktop\dianyun\SegmentAnything3D-main\scannet-preprocess\prepare_2d_data\util.py", line 42, in read_label_mapping
if represents_int(mapping.keys()[0]):
TypeError: 'dict_keys' object is not subscriptable
I am getting this error for line 78 in sam3d.py script
color_image = np.reshape(color_image[mask], [-1,3])
IndexError: boolean index did not match indexed array along dimension 0; dimension is 480 but corresponding boolean dimension is 240
Hi,
I'm kind of a rookie here. Could someone please tell me something about the lib "pointops"? I searched it on google but got insufficient results. I guess it is some tool for point operations and saw "pointops_cuda" when searching, but I wonder where is its source?
Thanks a lot.
I would like to ask you, in the fourth step of Pipeline, in the command ". /segmentator input.ply [kThresh=0.01] [segMinVerts=20]", what is the optimal value to set for the parameters "kThresh" and "segMinVerts"?
After running sam3d.py, what should I do next?
Hey, thanks for your work.
I reproduce the results in scene0000_00, I follow the default setting in your repo, but I get about 817 instances in an scene, which is seemed not reasonable.
I think maybe there are something wrong when I prepare the dataset. So I want to check that whether the frames_skip is 20 when prepare the rgbd data, and did your rgbd frames are come from .sens file? Or maybe other issues that I haven't noticed...
Thank you!
Hello,
Amazing work! Would it be possible for you to add a license file to the repo? (preferably MIT)
Best,
Florian
When running the setup.py I am getting the following error:
Traceback (most recent call last): File "setup.py", line 9, in <module> flag for flag in opt.split() if flag != '-Wstrict-prototypes' AttributeError: 'NoneType' object has no attribute 'split'
On further investigation, it seems that "get_config_vars('OPT')" is returning 'None'. Now I can't find what 'OPT' configuration is expected. Please help me with this.
I am using Windows 11, Cuda 11.7, and Python 3.8 (as recommended)
Reference: https://github.com/ChaoningZhang/MobileSAM
Our project performs on par with the original SAM and keeps exactly the same pipeline as the original SAM except for a change on the image encode, therefore, it is easy to Integrate into any project.
MobileSAM is around 60 times smaller and around 50 times faster than original SAM, and it is around 7 times smaller and around 5 times faster than the concurrent FastSAM. The comparison of the whole pipeline is summarzed as follows:
Best Wishes,
Qiao
Hi,
Thank you for this nice repo.
How can we get/download/prepare ground truth segmented 3d point cloud data for all scenes to compare segmentation output performance?
Many thanks.
So when trying to install Sam3D, i ran into an error on the second part of the installation, when running the python setup.py install
, i ran into this issue at the bottom:
(Having Python 3.8.10 installed, CUDA 11.8, and pytorch installed through their official website, last version)
It gives an error on the typing of that specific line, that i assume it should be a string.
I tried to run the requirements.txt, but lots of them require a higher python version, while the tensorflow-io-gcs-filesystem specifies one version that doesnt exist for the highest version of the python to which the requirements don't give any trouble (which i found to be Python 3.10).
I also noticed that the package "torch.utils.cpp_extension" is not found at the installation, if this gives more insight on the problem.
I tried with multiple CUDA and Python versions, but none of them seem to work, so I'm stuck here.
Thanks in advance if someone responds :)
Hi, if I want to change the dataset (e.g. S3DIS), how should I generate the over-segmentation result in step 4?
When I process 2d images with the following command, I found instrinc_depth.txt missing when generate results with sam3d.py
python scannet-preprocess/prepare_2d_data/prepare_2d_data.py --scannet_path data/scannetv2 --output_path data/scannetv2_images --export_label_images
While preparing PointCloud data 3 different folders are created. but only the train folder got one .pth. Is it ok? Or am I mistaken? Please let me know about the point cloud data format after preparing it.
And RGBD preparation occurs File not found data/scannetv2.
Hello, how can I solve this problem?
Traceback (most recent call last):
File "sam3d.py", line 283, in
voxelize, args.th, train_scenes, val_scenes, args.save_2dmask_path)
File "sam3d.py", line 239, in seg_pcd
indices, dis = pointops.knn_query(1, gen_coord, offset, scene_coord, new_offset)
AttributeError: module 'pointops' has no attribute 'knn_query'
Why does "sam3d.py" run as a ".pth" file? Normally, shouldn't the result of semantic segmentation be a tag file or a point cloud file?
".pth" file is the final training result? If so, how should I test it?
When I preprocess the RGBD data in the data preprocessing step, I processed it according to the prepare_2d_data.py file, and each scene only generated four folders of color, depth, label, and pose, but when I ran the sam3d.py file, it prompted me that I could not find the intrinsics.txt file under the intrinsics folder.This error is reported on the second line of the get_pcd function in the sam3d.py file, how can I generate intrinsics.txt data for each scene?
Thank your for your excellent work!
I was considering the following approach: What if we integrate an object detection module into 2D images, and then map the detection results onto a 3D point cloud? This way, we could determine the classes of each group within the point cloud segmentation results. Do you think this approach would be feasible for the project? Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.