dinglufe / segment-anything-cpp-wrapper Goto Github PK

Hello , you are write a good project of SAM. I'd like it. But i met a problem: mask has Y offset.
can you help me ?

platform

• ubuntu 18

model

• vit_h

c++ envs:

• onnx runtime: 1.15.1
• opencv: 4.7.0

py envs:

• segment-anything 1.0
• onnx 1.14.1
• onnxruntime-gpu 1.16.0

I meet the a problem , when i run the decoder in C++ ，my masks output shape is [1,4,0,0] , Like this
facebookresearch/segment-anything#294
masks out shape: 1 4 0 0
iou_predictions out shape: 1 4
low_res_masks out shape: 1 4 256 256
Do you know how to solve it?
Thanks

bash ./vcpkg install opencv:x64-windows gflags:x64-windows onnxruntime-gpu:x64-windows
An error occurred mounting one of your file systems. Please run 'dmesg' for more details.

你好，请问您了解这是因为啥吗

I want to export mobile sam model to a new size 1024*xx. I hope xx can be a smaller number than 720. to speed up the inference speed.

But In export_pre_model.py, error coours at " import mobile_sam as SAM". Where is the mobile_sam.

help!!! I use it as this,but get error:not enough space: expected 4194304, got 0. My memory is enough.What's wrong?
thresholdMat = sam.getMask({500,400});

if change cuda ，loading model is fail，what's wrong？my cuda version is 11.7. GPU 1050TI

system windows
visual studio 2017

Can you show all the test project code.I cann't run in my compute after I create a new project and and test.cpp into project.

hi there, what are the main differences right now for the automatic mode vs the original segment anything demo?

it seems that the effect of mobile sam is significantly inferior to original sam , fastSAM maybe a better choice.

CMake Error at build/vcpkg/scripts/buildsystems/vcpkg.cmake:855 (_find_package):
Could not find a package configuration file provided by "OpenCV" with any
of the following names:

OpenCVConfig.cmake
opencv-config.cmake

Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set
"OpenCV_DIR" to a directory containing one of the above files. If "OpenCV"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
CMakeLists.txt:6 (find_package)

大佬能否留个联系方式

Can you add efficeientsam?

Thank you for providing such good learning materials. I tried to use the export_pre_model.py file you provided to export the onnx model, and found that the input tensor format is uint8 instead of float32. Use the same method to convert the encoder on the official SAM model. , the input format of the obtained onnx model is float32, is this correct?

I want that the model segments based on an input string, e.g. "red cars".
It seems that this is not yet supported in this implementation.

If I find time, I could try to add this. But I need a starting point.
Can you give some advice?

Thanks for sharing this repo. It really helps.

After compile , I dont have cuda dlls,
Did I compile wrong? btw compiled in vs2019.
May you show the compile way or param setting, and the cuda&cudnn version you used.
Thank you very much !

MobileSAM & FastSAM are "fast", but there is also another direction: https://github.com/SysCV/sam-hq — it would be wonderful if you could add support for it as well :)

Hello, I'm trying to generate the onnx preprocessing models and I'm having the following error:

File "export_pre_model.py", line 84, in forward
    return self.predictor.get_image_embedding(), torch.stack(self.predictor.interm_features, dim=0)
AttributeError: 'SamPredictor' object has no attribute 'interm_features'

I tested with the vit_b and vit_h models from https://github.com/facebookresearch/segment-anything/ and the error appear for both.
I installed the segment_anything library direct from the Meta github with pip.

On the original demo, there is a method to turn a box into a mask. I think would be a nice addon here as well.

Sorry for offtop. I was trying to find a windows cli to run stable diffusion onnx models and accidentally found your repo.

How to change the image size, still only 1024x720?I see that the recognition result is still different from the demo on the official website, I wonder if it's because of this reason?

I have downloaded the sam hq models and executed this exporter script. This error occurs because the original segement_anything is imported.
If I change the code of the exporter to segement_anything_hq, the script runs without error, but does not export any models.

Could you please upload the exported models to github?

s:\Sources\Master\segment-anything-cpp-wrapper>python export_pre_model.py
Traceback (most recent call last):
File "s:\Sources\Master\segment-anything-cpp-wrapper\export_pre_model.py", line 56, in
sam = SAM.sam_model_registrymodel_type
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\znoopy2k\AppData\Local\Programs\Python\Python311\Lib\site-packages\segment_anything\build_sam.py", line 15, in build_sam_vit_h
return _build_sam(
^^^^^^^^^^^
File "C:\Users\znoopy2k\AppData\Local\Programs\Python\Python311\Lib\site-packages\segment_anything\build_sam.py", line 106, in _build_sam
sam.load_state_dict(state_dict)
File "C:\Users\znoopy2k\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Sam:
Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

调用option.AppendExecutionProvider_CUDA(options);函数报错

使用的onnx版本为onnxruntime-win-x64-gpu-1.15.1

Inference by cpu,the image size is 2000*2000,It takes a long time to run "sam.loadImage(image)",about3minutes，why is it happen?How can I improve the running speed？thanks for your answer!

Progress: 0% 2023-07-26 17:04:50.4252918 [E:onnxruntime:test, cuda_call.cc:119 onnxruntime::CudaCall] CUDNN failure 4: CUDNN_STATUS_INTERNAL_ERROR ; GPU=0 ; hostname=MSI ; expr=cudnnFindConvolutionForwardAlgorithmEx( GetCudnnHandle(context), s_.x_tensor, s_.x_data, s_.w_desc, s_.w_data, s_.conv_desc, s_.y_tensor, s_.y_data, 1, &algo_count, &perf, algo_search_workspace.get(), max_ws_size);
2023-07-26 17:04:50.4346841 [E:onnxruntime:, sequential_executor.cc:494 onnxruntime::ExecuteKernel] Non-zero status code returned while running Conv node. Name:'/mask_downscaling/mask_downscaling.0/Conv' Status Message: CUDNN failure 4: CUDNN_STATUS_INTERNAL_ERROR ; GPU=0 ; hostname=MSI ; expr=cudnnFindConvolutionForwardAlgorithmEx( GetCudnnHandle(context), s_.x_tensor, s_.x_data, s_.w_desc, s_.w_data, s_.conv_desc, s_.y_tensor, s_.y_data, 1, &algo_count, &perf, algo_search_workspace.get(), max_ws_size);

It takes too long to load pictures in sam.loadImage(image). Is there any way to solve it? At present, it takes 1-2 seconds. Can it be reduced?

I want to run the mobile sam at resolution 320x240. Any suggestions?

with export_pre_model.py, it still require one side to be 1024.

Every time I import a new image into the code, it loads particularly slowly, especially when executing the sam.getMask() function. But when I select different points on the same image, it is processed quickly. I wonder why? Is it because the code has to execute the decoder model every time a new image is loaded?
compare: When I enter the same bounding box but use a different photo, the code takes about 11200ms to process it；
When I use the same photo but different points，the code takes about 50ms to process it。
Preprocess device:cpu（i5-13400） Sam device:cpu（i5-13400）
When both I use CPUs for Preprocess and Sam, I wanted to ask if there was any way to reduce processing time？

I was trying to run the program in VsCode and followed the steps as specified. I have installed vcpkg and the opencv folders are already in it.
But, when I run test.cpp, the following error is shown
"opencv2/opencv.hpp: No such file or directory".

Can you please advise how to solve?
Thank you,
Mareeta

I've export onnx model ''mobile_sam.onnx" use script provided in mobile sam project,call it use v1.4.1,error msg as follows:

Now click on the image (press q/esc to quit; press c to clear selection; press a to run automatic segmentation) Ctrl+Left click to select foreground, Ctrl+Right click to select background, Middle click and drag to select a region 2023-07-12 10:52:46.2316412 [E:onnxruntime:, sequential_executor.cc:368 onnxruntime::SequentialExecutor::Execute] Non-zero status code returned while running Reshape node. Name:'Reshape_197' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\cpu\tensor\reshape_helper.h:41 onnxruntime::ReshapeHelper::ReshapeHelper gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,6,256}, requested shape:{1,10,8,32}

Thanks for your Segment Anything CPP Wrapper.

Here is the Segment Anything CPP Wrapper for macOS.
This code is originated from Segment Anything CPP Wrapper and implemented on macOS app RectLabel. We customized the original code so that getMask() uses the previous mask result called as low_res_logits and retain the previous mask array for undo/redo actions.

Please let us know your opinion.

How can we support multiple points, and negative points?

When running the Run function in the loadImage function, it is always slow, is there any solution? Is the process a must every time a new picture is loaded? Can it be reused to reduce pre-treatment time?

@dinglufe

What should I do if the input size of the modified image is 640x480, and the obtained mask cannot coincide with the original image?

Hello, thank you for your efforts. Is it possible for you to support EdgeSam?

Thank you in advance!