foundationvision / glee Goto Github PK

In paper, "query denoising and "Hybrid matching" are kept to accelerate convergence and improve performance". Does Hybrid matching refer to the H-DETR paper: https://arxiv.org/pdf/2207.13080.pdf

是否能像SAM或CLIP那样直接调用

例如：from sam import SAM
model = SAM(....)
.....

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory projects/GLEE/clip_vit_base_patch32. 缺这么多文件怎么说

您好，我试用了GLEE，非常棒的工作！想请教一下GLEE是否支持跨图的检测呢，具体来说，就是在第一张图像上给出scribble或者bbox，然后在另一张图像上检测第一张图上的所指目标。我看到视频有类似功能，请问是否也支持静态图像呢

Do you plan to add a VIDEO panoptic benchmark?
https://github.com/yoxu515/VIPOSeg-Benchmark

Hello, thank you for the outstanding work. Could you please make the inference code and weight files about the TAO data set public? Thank you very much.

Hello, GLEE is a wonderful work. I saw that the RVOS code part is not finished yet. Can you update the GLEE.py for RVOS inference?

github界面只给了图片任务的R50和SwinL2个版本的模型，然后我在huggingface上demo的files里面看到了视频任务的R50版本（visual prompt，GLEE_vos_r50.pth），想问下作者能不能开源一下视频任务的SwinL版本，是不是因为huggingface上使用的GPU跑不动所以才没放SwinL版本？
此外，关于使用的体验，我发现模型对于没学过的语言提示词效果很差，比如用custom-list不认识人头（head），输入human head才有可能给出比较差的结果。

您好，我使用了您的模型对 COCO 数据集进行测试，发现我得到的评估指标与paper的指标有差距。我想了解一下，现在提供的demo和完整测试过程之间是否存在某些差异？

Try to test using the Hugging Face demo. The demo page is just showing a error:

Hi! I've read README and tried to run the demo on my server, but I think there're a lot of code that is out of sync or missing. And the guides are incomplete.

Here are my steps:

1. I installed the dependencies according to INSTALL.md
2. I ran the app.py in this repo
3. A lot of models are missing:
   • I need GLEE models so I downloaded them on huggingface demo directory
   • Misc models are missing, then I found the ones mentioned in TRAIN.md and downloaded them
4. There are bugs in app.py on lines like (outputs,_) = GLEEmodel(...), which should be ((outputs, _), _, _) = GLEEmodel(...)
5. Then I ran the app.py in this repo again, but the results are just random like below.

Did I do anything wrong? Should I just clone the huggingface repo instead?

作者您好，非常感谢您的工作，我在本地运行时，报了缺少文件的错误：
Config '/mnt/yrfs/userdata/hsp/projects/GLEE/app/GLEE/configs/R50.yaml' has no VERSION. Assuming it to be compatible with latest v2.
Traceback (most recent call last):
File "app.py", line 90, in
GLEEmodel_r50 = GLEE_Model(cfg_r50, None, device, None, True).to(device)
File "/mnt/yrfs/userdata/hsp/projects/GLEE/app/GLEE/glee/models/glee_model.py", line 67, in init
self.text_encoder = CLIPTextModel.from_pretrained('GLEE/clip_vit_base_patch32')
File "/home/hsp/anaconda3/envs/GLEE/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3206, in from_pretrained
raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory GLEE/clip_vit_base_patch32.
请问我需要在哪里下载对应的文件呢？

GLEE在训练过程中，每个iter采样的数据都是来自相同数据集的吗？我发现task总是取batched_inputs[0],

运行时出现如下问题，请问如何解决：
File "E:\hezt\vis\lib\site-packages\gradio\queueing.py", line 527, in process_events
response = await route_utils.call_process_api(
File "E:\hezt\vis\lib\site-packages\gradio\route_utils.py", line 270, in call_process_api
output = await app.get_blocks().process_api(
File "E:\hezt\vis\lib\site-packages\gradio\blocks.py", line 1847, in process_api
result = await self.call_function(
File "E:\hezt\vis\lib\site-packages\gradio\blocks.py", line 1433, in call_function
prediction = await anyio.to_thread.run_sync(
File "E:\hezt\vis\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "E:\hezt\vis\lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "E:\hezt\vis\lib\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "E:\hezt\vis\lib\site-packages\gradio\utils.py", line 805, in wrapper
response = f(*args, **kwargs)
File "E:\tool\GLEE-new\app.py", line 169, in segment_image
(outputs,_) = GLEEmodel(infer_image, prompt_list, task="coco", batch_name_list=batch_category_name, is_train=False)
ValueError: too many values to unpack (expected 2)

Why you have excluded from the VOS benckmark at table 8 some top performing memory networks?

https://github.com/yoxu515/aot-benchmark
https://github.com/yamy-cheng/DMAOT-VOTS2023

Have you tried to finetune the stage 2 model on COCO dataset and see the limit of the COCO performance?

请教一下GLEE-Pro和GLEE-Plus的运行速度是多少

Hi， I reproduced the inference script via the gradio script (app.py). But I get different object detection results with the same parameter.

In the todo list you have a bullet point related to "zero-shot testing or fine-tuning new datasets".

Do you plan to also cover single modality dataset fine tuning?

你好：我在windows下运行Python app.py时，遇到GLEE\app.py", line 18, in
from projects.GLEE.glee.models.glee_model import GLEE_Model
ModuleNotFoundError: No module named 'projects.GLEE'
的错误，明明projects就在当前文件夹下，怎么就加载不了的呢？盼回复。
主要错误如下：Traceback (most recent call last):
File "E:\tool\GLEE\app.py", line 18, in
from projects.GLEE.glee.models.glee_model import GLEE_Model
ModuleNotFoundError: No module named 'projects.GLEE'
原码app.py部分如下：mport gradio as gr
import numpy as np
import cv2
import torch

from detectron2.config import get_cfg
import sys
#sys.path.insert(0, 'E:/tool/GLEE-main')
#sys.path.append('E:\tool\GLEE\projects\')
from projects.GLEE.glee.models.glee_model import GLEE_Model
from projects.GLEE.glee.config import add_glee_config

in the hugging face demo, the expression prompt mode only output one object, even if there are multi same objects?

Hi, there. I believe GLEE is a great work, thanks for open source!
I have a question about object detection: what's the input to the decoder when used as a object detector?
Does it need to input object query including box position from anchor boxes?
If I'm not wrong, in MaskDINO, it will input box position from anchor and mask as object query.
So, what's the object query like in GLEE when used as object detector?
Looking forward for your reply, thanks a lot!

您好，GLEE是一个很棒的工作。同时，关于算法的一些细节，我有一些疑问想像您请教，如果您有空了，可以回复一下，感谢！

1. 我使用points作为视觉提示词，GLEE是否支持负点击？能否像SAM一样使用多次点击来对一个目标进行微调？
2. 我看您的代码实现，似乎会将points变为一个box作为提示，为什么要这么做？我没有在您的论文中找到相关的解释。
3. 视觉提示词返回的topk_instance是否只能为1？它能否分割出一个被遮挡目标的多个部分？
   感谢！

Hi, Thanks for the solid work. Could you let me know when you'll release the training code?