Comments (15)
The current
ms_deform_attn
cuda kernel only supports GPU. However, there is a pure PyTorch implementation here:Mask2Former/mask2former/modeling/pixel_decoder/ops/modules/ms_deform_attn.py
Lines 116 to 119 in 012049c
Maybe you can try
ms_deform_attn_core_pytorch
instead ofMSDeformAttnFunction.apply
when using CPU.
Thanks for your help! But I tested it on a random input and found that there is a significant error between using ms_deform_attn_core_pytorch
under CPU mode and using MSDeformAttnFunction.apply
under GPU mode.
Result obtrained by using ms_deform_attn_core_pytorch:
tensor([[[[ 1.8436, 2.2416, 2.6247, ..., 2.7736, 2.7426, 2.1027],
[ 2.5010, 3.0443, 3.3008, ..., 2.9831, 2.9631, 2.3863],
[ 2.9538, 3.5157, 3.5276, ..., 3.2951, 2.8759, 2.1722],
...,
[ 2.8571, 3.7810, 3.2169, ..., 3.1239, 3.6230, 3.1102],
[ 2.9726, 3.7958, 3.5246, ..., 3.7332, 4.0925, 3.1939],
[ 2.9812, 3.2457, 3.2969, ..., 2.9720, 3.0884, 2.6482]],
[[-3.9825, -3.8449, -4.4274, ..., -4.3323, -4.5814, -3.9630],
[-3.8496, -4.2992, -4.9432, ..., -4.3094, -4.3040, -4.0190],
[-4.4084, -4.7840, -4.8061, ..., -4.6905, -4.1271, -3.5728],
...,
[-4.4807, -5.3218, -4.5969, ..., -4.8324, -5.4793, -4.2268],
[-4.7795, -5.3368, -4.9663, ..., -5.6892, -6.0757, -4.5740],
[-4.5650, -4.6115, -4.4746, ..., -4.4084, -4.3251, -4.2097]],
[[-3.0353, -3.5850, -3.8838, ..., -4.6120, -4.3296, -3.6653],
[-3.7541, -4.4487, -4.5815, ..., -4.4298, -4.8221, -3.9200],
[-4.0843, -5.0629, -5.0538, ..., -4.8248, -4.4019, -3.8944],
...,
[-3.9654, -5.2787, -4.6193, ..., -4.8375, -5.6787, -5.3515],
[-4.3268, -5.5459, -5.3988, ..., -5.7265, -6.5075, -5.4388],
[-5.0671, -5.2403, -5.4108, ..., -4.9114, -5.2505, -4.6957]],
[[-4.2067, -4.4303, -4.8076, ..., -4.4385, -5.2699, -5.5285],
[-4.0580, -4.6923, -5.0086, ..., -4.0468, -4.5669, -4.8352],
[-5.1472, -5.4393, -5.1526, ..., -4.5857, -4.1159, -4.2404],
...,
[-4.9072, -5.6090, -4.4477, ..., -4.1471, -5.1054, -4.6439],
[-5.2460, -5.6856, -4.9846, ..., -5.4047, -6.0951, -5.2036],
[-6.4418, -5.6409, -5.5475, ..., -4.7478, -5.0343, -5.0288]]]])
Result obtained by using MSDeformAttnFunction.apply:
tensor([[[[ 1.8438, 2.2418, 2.6247, ..., 2.7730, 2.7421, 2.1017],
[ 2.5010, 3.0440, 3.3010, ..., 2.9818, 2.9627, 2.3849],
[ 2.9538, 3.5151, 3.5274, ..., 3.2940, 2.8751, 2.1706],
...,
[ 2.8567, 3.7798, 3.2173, ..., 3.1232, 3.6217, 3.1084],
[ 2.9719, 3.7946, 3.5238, ..., 3.7322, 4.0911, 3.1923],
[ 2.9801, 3.2443, 3.2952, ..., 2.9713, 3.0874, 2.6471]],
[[-3.9825, -3.8448, -4.4270, ..., -4.3318, -4.5808, -3.9619],
[-3.8493, -4.2982, -4.9436, ..., -4.3074, -4.3032, -4.0174],
[-4.4081, -4.7829, -4.8064, ..., -4.6886, -4.1262, -3.5713],
...,
[-4.4809, -5.3203, -4.5975, ..., -4.8315, -5.4775, -4.2247],
[-4.7791, -5.3360, -4.9657, ..., -5.6872, -6.0733, -4.5714],
[-4.5640, -4.6103, -4.4728, ..., -4.4075, -4.3242, -4.2080]],
[[-3.0350, -3.5848, -3.8832, ..., -4.6107, -4.3282, -3.6638],
[-3.7525, -4.4475, -4.5803, ..., -4.4276, -4.8206, -3.9177],
[-4.0838, -5.0615, -5.0520, ..., -4.8226, -4.3999, -3.8922],
...,
[-3.9630, -5.2760, -4.6183, ..., -4.8337, -5.6740, -5.3469],
[-4.3242, -5.5429, -5.3960, ..., -5.7220, -6.5034, -5.4351],
[-5.0641, -5.2365, -5.4068, ..., -4.9085, -5.2471, -4.6935]],
[[-4.2063, -4.4301, -4.8070, ..., -4.4376, -5.2689, -5.5274],
[-4.0575, -4.6916, -5.0092, ..., -4.0446, -4.5659, -4.8326],
[-5.1466, -5.4383, -5.1529, ..., -4.5832, -4.1145, -4.2369],
...,
[-4.9071, -5.6081, -4.4490, ..., -4.1483, -5.1057, -4.6425],
[-5.2457, -5.6848, -4.9844, ..., -5.4051, -6.0948, -5.2022],
[-6.4400, -5.6390, -5.5436, ..., -4.7483, -5.0334, -5.0283]]]])
Can you give me some suggestions to make up for this error?
from mask2former.
Have you restarted runtime after installation?
from mask2former.
Yes I pressed the 'Restart runtime' button that appeared at the bottom of the cell. But I tried it again and this time I think I restarted after the running the second installation cell and it works for me now.
from mask2former.
@bowenc0221 also get this issue when running in colab on cpu
RuntimeError Traceback (most recent call last)
in ()
9 cfg.MODEL.MASK_FORMER.TEST.PANOPTIC_ON = True
10 predictor = DefaultPredictor(cfg)
---> 11 outputs = predictor(im)
16 frames
/content/Mask2Former/mask2former/modeling/pixel_decoder/ops/functions/ms_deform_attn_func.py in forward(ctx, value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights, im2col_step)
35 ctx.im2col_step = im2col_step
36 output = MSDA.ms_deform_attn_forward(
---> 37 value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights, ctx.im2col_step)
38 ctx.save_for_backward(value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights)
39 return output
RuntimeError: Not implemented on the CPU
from mask2former.
The current ms_deform_attn
cuda kernel only supports GPU. However, there is a pure PyTorch implementation here:
Mask2Former/mask2former/modeling/pixel_decoder/ops/modules/ms_deform_attn.py
Lines 116 to 119 in 012049c
Maybe you can try ms_deform_attn_core_pytorch
instead of MSDeformAttnFunction.apply
when using CPU.
from mask2former.
@bowenc0221 thanks could the colab be updated to support cpu? I am currently testing it to add the models to https://huggingface.co/spaces as a web demo, similar to
github: https://github.com/facebookresearch/omnivore
Spaces: https://huggingface.co/spaces/akhaliq/omnivore
Spaces is completely free, and I can help setup a Gradio Space. Here are some getting started instructions if you'd prefer to do it yourself: https://huggingface.co/blog/gradio-spaces
from mask2former.
@AK391 thanks for adding it to huggingface space!
I've updated the code to support inference on CPU (16c3bee). Could you try it again?
from mask2former.
@bowenc0221 thanks working on it now, also the models can be added to the facebook organization on Huggingface: https://huggingface.co/facebook
Would you be interested in becoming part of the Facebook organization on the hub? This allows you to have write access to all model repos of Facebook, which means you can easily upload new checkpoints, write model cards, etc.
Once a model is added it can be made into a Gradio space with a few lines of code
for example in xm_transformer
https://huggingface.co/spaces/facebook/xm_transformer_600m
gr.Interface.load("huggingface/facebook/xm_transformer_600m-es_en-multi_domain").launch()
can join here: https://huggingface.co/welcome
from mask2former.
@bowenc0221 inference on cpu is working in colab thanks, I setup the space here https://huggingface.co/spaces/akhaliq/Mask2Former but I get this error
Traceback (most recent call last):
File "setup.py", line 76, in
ext_modules=get_extensions(),
File "setup.py", line 52, in get_extensions
raise NotImplementedError('CUDA_HOME is None. Please set environment variable CUDA_HOME.')
NotImplementedError: CUDA_HOME is None. Please set environment variable CUDA_HOME.
from mask2former.
Installing the package requires CUDA (GPUs are not needed). Is CUDA available?
You can use the following command:
FORCE_CUDA=1 CUDA_HOME=/path/to/cuda python setup.py build install
from mask2former.
@bowenc0221 thanks, yes I tried that step, see the code here: https://huggingface.co/spaces/akhaliq/Mask2Former/blob/main/app.py#L16
however, cuda is not available
from mask2former.
Hey @AK391, sorry for the late reply. I think there is an easy solution to avoid installing the custom op. Could you try to do the following:
- Define an environment variable, e.g.,
HUGGINGFACE_DEMO
- Do not import the following when
HUGGINGFACE_DEMO=1
.
For example:
if os.environ.get('HUGGINGFACE_DEMO', 0):
MSDeformAttnFunction = None
else:
from ..functions import MSDeformAttnFunction
from mask2former.
@bowenc0221 thanks, I opened a PR here #52 for the Huggingface demo
from mask2former.
@bowenc0221 thanks, I opened a PR here #52 for the Huggingface demo
Excuse me, have you resolved the question above?
from mask2former.
does anybody have error free python code or worked on it for this mask2former demo?
@AK391 @lianzheng-research @bowenc0221 @bh-cai @blnfb , I need this very bad. I started it recently it would be very helpful if you guys could share it to my mail [email protected]
from mask2former.
Related Issues (20)
- Installation expects CUDA_HOME on Apple Silicon Macs HOT 1
- How to understand the output of different tasks
- Using ground truth masks instead of the predicted ones
- No module named 'MultiScaleDeformableAttention', Please compile MultiScaleDeformableAttention CUDA op HOT 2
- As for training, how long does it take?
- HAVE ANYONE MEET SUCH ERROR WHEN TRAINING ON OWN DATASET HOT 1
- batch_size doesn't affect evaluation
- how use custom pre-trained backbone in mask2former HOT 1
- why swin accept different input size
- loading swintransformer
- Ambiguous checkpoint key error when running train_net.py HOT 1
- difference among different mode
- Prebuilt wheels provided via 3rd party repository
- Using COCO for the dataset, what is the appropriate adjustment for learning rate if using a single GPU
- Run in colab seems that there's a ModuleNotFoundError related to the MultiScaleDeformableAttention module.
- Poor Output image quality
- Mask loss with soft labels
- Custom dataset registration to use a model trained on Cityscapes for semantic segmentation.
- How should I fix the input size during testing? HOT 3
- Could you please let me know if anyone has successfully trained using the YouTube VIS 2021 dataset? How should the dataset be formatted?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mask2former.