[ICLR2024] The official implementation of paper "VDT: General-purpose Video Diffusion Transformers via Mask Modeling", by Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, Mingyu Ding.
Is the network model framework in the inference code you provide consistent with that during training? Are there inconsistencies between the training and inference codes?
Great job and looking forward to your reply, thanks in advance.
Fantastic work! Since a few month have passed and the paper is accepted by ICLR(congrats!), would you please release the training code? Also some instructions on how to prepare the dataset would be great!
Congratulations on the impressive paper. When I tried running inference with your pre-trained physion model, the results significantly degraded as the number of condition frames was reduced. For example, using 4 condition frames (rather than the default of 8) produces only noise - see attached image.
Does this match your expectation? It seems at odds with the paper's discussion, which states "our VDT can still take any length of conditional frame as input and output consistent predicted features".
Thank you!
Edit: I see in Figure 8 that you tried using more than 8 conditional frames, but not less. Do you have a sense how well the forward prediction can perform with only 1 conditioning frame using VDT? Would the model need to be trained with only 1 conditioning frame?
Thanks for the great work! One question I have is about the evaluation results on Physion data. In both the paper and code, there seems to be only results and model checkpoint for collision. Wondering if VDT is evaluated on 7 other Physion scenarios? If there is, it would be great if both the evaluation results and checkpoints could be shared. Thanks in advance