dawn-lx / vidsgg-big Goto Github PK
View Code? Open in Web Editor NEWPytorch implementation of our paper Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs, which is accepted by CVPR2022
Pytorch implementation of our paper Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs, which is accepted by CVPR2022
Hi, how did these .npy generate in prepared_data folder? pred_bias_matrix_vidvrd.npy and vidvrd_EntiNameEmb.npy. I did not find the corresponding code for these. Thank you.
When I loaded all the cached VidOR data and dataloader started to fork, the program crashed due to error OSError: [Errno 12] Cannot allocate memory
. I tried lower number of threads, but it did not work.
I am wondering whether my memory is too low (189GB, occupied memory is about 150+GB before crashing). Could you please provide your device information?
Thank you!
Hi, could you introduce the structure of the pre-prepared cache data for VidOR, for example, MEGAv7_VidORtrain_freq1_part01_th_15-180-200-0.40.pkl
?
Besides, when I trainning grounding stage, I have a questions about the code of DEBUG.
In models/grd_model_v5.py
, line 268, does the inter_dura
means the time slot that subject and object are all appearing in the trajectory? And what's the meaning of the index_map
?
Thank you very much!
Hello,
The link for downloading the tracklet data VidVRD_test_every1frames is invalid. Could you please provide the new one?
Thank you very much.
According to tools_draft/extract_classme.py
, I run the tools_draft/construct_CatName2vec.py
first. But no file named vidor_CatName2vec_dict.pkl
is generated. Could you help me?
Thank you!
Thank you for your sharing the source code!
I would like to know how we can run experiments from Figure 6 in the paper.
And about the VidVRD Dataset, can we get qualitative results as visualization results?
I find this snippet in grd_model_v5.py, which is used for DEBUG model to predict time boundaries. But, in regression head, why the left (right) offset is mapping to [0, 1] by sigmoid ? And how do you transform time boundaries to video frame feature sequences ?
temp = nn.Sequential(
DepthWiseSeparableConv1d(self.dim_hidden,self.dim_hidden,3),
nn.ReLU()
)
temp2 = [copy.deepcopy(temp) for _ in range(4)] \
+ [DepthWiseSeparableConv1d(self.dim_hidden,self.num_bins,3)]
temp3 = [copy.deepcopy(temp) for _ in range(4)] \
+ [DepthWiseSeparableConv1d(self.dim_hidden,2*self.num_bins,3),nn.Sigmoid()]
self.cls_head = nn.Sequential(*temp2)
self.conf_head = copy.deepcopy(self.cls_head)
self.regr_head = nn.Sequential(*temp3)
Hello, could you provide your extracted tracklet data VidVRD_test_every1frames
? Thank you very much.
Hello, when I download the prepared data, I find the VidVRD test data from zju needs the permission, and the pku data is no longer available. Could you please update the link? Thanks a lot.
Hi, could you introduce the structure of the pre-prepared cache data for VidOR, for example, MEGAv7_VidORtrain_freq1_part01_th_15-180-200-0.40.pkl
?
Besides, when I trainning grounding stage, I have a questions about the code of DEBUG.
In models/grd_model_v5.py
, line 268, does the inter_dura
means the time slot that subject and object are all appearing in the trajectory? And what's the meaning of the index_map
?
Thank you very much!
Hi, when I evaluate grounding stage on VidOR by running eval_vidor.py
, I find inference result has been load before evaluation, e,i,, VidORval_infer_results_topk3_epoch60_debug.pkl'
. Could you introduce the structure of the data of the file?
Thank you very much!
Hi, I find that the cat_ids of the trajectory proposal is different from the traj_cat_ids of the paired gt_graph.
For example:
proposal.cat_ids: tensor([ 4, 4, 24, 4, 31, 31]), gt_graph.traj_cat_ids: tensor([11, 11, 7, 35, 35])
There is no common trajectory categories between each video-gt_graph pairs. But I think there should be the same category.
So I want to know how to obtain the concrete catetory of a trajectory proposal or gt_graph.
Thanks!
Tips from @Dawn-LX :
This problem originates from
VidSGG-BIG/dataloaders/dataloader_vidor.py
Lines 488 to 508 in eaf7578
Here, we notice that tracking results for each box at one specific frame consist of a 6-dim
vector or a (12+dim_boxfeature)-dim
vector.
6-dim
vector appears, corresponding box will be viewed as background.12-dim
of box_info
, which consists of frame_id
, tracklet_id
, 4-dim bbox coordinates
, confidence
, category_id
, 4-dim bbox coordinates
, will be used to determine the final location of bbox.The first 4-dim bbox coordinates (box_info[2:6])
is generated by tracker, and the second one box_info[8:12]
is generated by our video obeject detector. The reason why box shift is that we calculate an average bbox coordinates by the two mentioned one. Because detected object location maybe inconsistent with current tracklet, and the tracker-generated one is more precise, so this averaging manner may merge two boxes to a background one.
Specifically, box generated by tracker is much more precise since it considers boxes in previous frames, current detected box, and visual similarity. But box from video object detector maybe wrongly linked to current tracklet (which does not mean it is a background box itself). So this averaging manner is not strictly correct in these cases and that is why we only use track-generated one (box_info[2:6])
in
VidSGG-BIG/dataloaders/dataloader_vidor_v3.py
Lines 414 to 421 in eaf7578
However, tracklet_mAP does not improve by switching from averaging manner to unique manner. The reasons maybe
Hi!
I found some negative values in the bounding boxes of proposal. Does it have some special meanings?
Thank you!
Hello, where can I get the file prepared_data/vidvrd_EntiNameEmb_pku.npy
and prepared_data/pred_bias_matrix_vidvrd_pku.npy
? I find no way to generate them by myself. Than you very much!
Hello! I tried to download the tracklets with features from the author's website http://www.muyadong.com/publication.html
under this paper
Beyond Short-Term Snippet: Video Relation Detection with Spatio-Temporal Global Context, but the link has expired.
Are there any ways to get the data? i have read that you trained your own RCNN to get the tracklet with features for one file that was missing. Should I do the same? Maybe you have the downloaded dataset of features saved somewhere?
Hi, I have a one question.
Could you please explain the choice that only applying classification stage on VidVRD dataset?
Thank you!
您好,我想做一些轨迹方面的消融实验,能提供一下您当时使用deepSORT方法时所用到的参数吗,另外还有提取物体的deepSORT feature时所用到的模型链接
refer to #2 (comment)
Hello, when I try to look the structure for "MEGAv7_VidORtrain_freq1_part01_th_15-180-200-0.40.pkl", and I try pickle.load, but it got an error as "_pickle.UnpicklingError: pickle data was truncated". I want to know how can I use these prepared data and the structure of it. Thank a lot.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.