Under this section Finetune, it's written that during test, you consider multiple segments and multiple crops

    --test_num_segment 2 \
    --test_num_crop 3 \

But while calculating accuracy, we don't aggregate accuracy scores over all these segments/crops--
https://github.com/MCG-NJU/VideoMAE/blob/main/engine_for_finetuning.py#L180
https://github.com/rwightman/pytorch-image-models/blob/master/timm/utils/metrics.py#L25

acc1, acc5 = accuracy(output, target, topk=(1, 5))
Instead while calculating accuracy we should have done something like this--

scores=pd.dataFrame({ "id" : ids, "outputs" : outputs, "labels" : labels })
scores=scores.groupby([ "ids" , "labels" ]).aggregate({"outputs": lambda x : max(x)})

Hi VideoMAE team :)

I've implemented VideoMAE as a fork of 🤗 HuggingFace Transformers, and I'm going to add it soon to the library (see huggingface/transformers#17821). Here's a notebook that illustrates inference with it: https://colab.research.google.com/drive/1ZX_XnM0ol81FbcxrFS3nNLkmn-0fzvQk?usp=sharing

The reason I'm adding VideoMAE is because I really like the simplicity of it, it was literally a single line of code change from ViT (nn.Conv2d -> nn.Conv3d).

As you may or may not know, any model on the HuggingFace hub has its own Github repository. E.g. the VideoMAE-base checkpoint fine-tuned on Kinetics-400 can be found here: https://huggingface.co/nielsr/videomae-base. If you check the "files and versions" tab, it includes the weights. The model hub uses git-LFS (large file storage) to use Git with large files such as model weights. This means that any model has its own Git commit history!

A model card can also be added to the repo, which is just a README.

Are you interested in creating an organization on the hub, such that we can store all model checkpoints there (rather than under my user name)?

Let me know!

Kind regards,

Niels
ML Engineer @ HuggingFace

Thanks for you excellent work. With only self-supervised learning, the VideoMAE can perform surprisely well. Besides that, I think the (self-supervised + full-supervised) models are also very meaningful, especially for the downstream tasks/datasets.

@yztongzhan Could you please providing a model/checkpoint that is the best pretrained model you can offer for learning video representation towards any unknown downstream video datasets?

For example, a model pretrained on the Kinetic-700 with VideoMAE self-supervisly, and then pretrained on Kinetic-700 supervisely.

Such a model should be very useful. One can easily fine-tune it on any downstream datasets for many possible purposes. Just like the ResNet-50 pretrained on ImageNet.

Hi,
First of all, thank you very much for your clean code and impressive results.
Second, is it possible for you to share the weights of the pretrained MoCoV3 baseline?

Best regards,
Sepehr

Hi,

Thank you for sharing the exciting work!

Could you please release the pre-trained models on ImageNet1K(with input_size 224x224 or 320x320)?

Many many thanks!

Hello.
First, thank you for this repository.

I'm trying to preprocess.

I already looked at install.md and installed it.
Now look at data.md and try to follow it.
However, I already have the kinetics-400 downloaded before, so I just need to pre-process the data.

i) Download the dataset from the official website.

ii) Preprocess the dataset by resizing the short edges of the video to 320px. You can refer to the MMAction2 data benchmark for TSN and SlowOnly.

iii) Create the annotations required by the dataloader ("<path_to_video> <video_class>" in the annotations). Comments usually include train.csv, val.csv, and test.csv (here val.csv is equivalent to test.csv). The file format *.csv is:

I'm on the second of this course you mentioned, and I'm not sure how to do it. How do I pre-process?

Thank you for read it!

When I set a high mask ratio, some unpredictable content will be roughly predicted.

Hi, congratulate on your great work!
Could you release label files (train.csv val.csv test.csv) for ssv2?
I tried to generate these files following the guidance in DATASET.md and use them for evaluation on ssv2 by running run_class_finetuning.py with --eval, but the accuracy is abnormal(too low), and I guess maybe the csv files I generated were wrong.
Could you release the csv files of K400 and ssv2 for everyone to download? Thanks!

Thanks for the excellent work!

Could you provide the training configuration for the AVA dataset? I tried to reproduce the results but observed overfitting.

Is there a script or details available for linear probing? The original MAE paper suggests theres a substantial difference between setups for finetuning and linear probing for MAE models and wanted to be able to reproduce results.

Thanks!

Hello, the author! I read your DATASETS.md for ssv2. But I have no ideo how to run the data_clean.py for original ssv2 videos? Why are .txt files used in data_clean.py? Could you kindly give one or two instructions to run data_clean.py for ssv2?

Hi, thank you for the nice work.

I have a question.
Does batch-size matter when I train a model?
What if someone only has two 24Gb GPUs, then what is the good choice of batch-size for the case?
Can I apply the equation appeared in Appendix B to compute the learning rate for small batch-size, such as 2 or 4, even 1?

Hi, I was wondering if there are any available resources on the preprocessing for UCF101 similar to the given data_clean.py script for SSV2?

Hi, thank you for sharing this amazing work.

I know you have used 64 GPUs spreading in 8 nodes for pretraining. However, I don't have so many GPU resources, I wonder whether I could pretrain this model in 8 GPUs or less.

Whether there are large margin performance decrease?

Thanks for the great work! I am wondering whether the demo/inference code has been released so that we can quickly test out the model on customized videos?

Hi,
Thanks for your work!
When I reproduce the finetuning experiments on K400 without deepspeed backend (by applying your released pretrained checkpoint), I only got 78.0 with VIT-B, which is 2.7% lower than the number (80.7) stated in the paper. Do you know the reason for that? Does the different backend is the main reason?
Thanks!

Nice. I'm reproducing it.

When the code for training and testing will be released?

Like the tile, I found that the videos are preprocessed twice, refer to: dataset and the train_one_epoch. can you help check it ?

Thank you very much for your code and contributions. Is it possible to use tensorRT to speed up

How do I use a pre-trained model to complete my inference?

Hi author:
Thanks for your impressive work, I run your code "run_mae_pretraining.py" and it comes an error that "engine_for_pretraining.py", line 79, in train_one_epoch
loss_scale_value = loss_scaler.state_dict()["scale"]" , as shown below; p.s. I used the pretrained ssv2's checkpoint;

Thanks for the great work! But I have one question that was not discussed in the paper, which is the inference speed.

I understand that during training the speed is fast, since you masked out 90% of the patches. However, during test time, I assume you retain all the patches. Since the attention in ViT is quadratic in cost, does it mean you will need 10*10 = 100 times the flops compared to the pretrain phase?

If that is the case, how much time does it take to do a single forward pass?

Hi Author:
I am studying your paper and trained a mae on kinetic dataset, and I have two questions; I would appreciate that if u could give a feedback;
1. Now I run "run_class_finetuning.py" based on a pretrained model, and warning shows below, is this correct ?
2. finetuning shows that the GPU occupation is 18G while batchsize are 2, could I freeze the part of foward_feature and only train the
classification head?

i train the model on UCF101, pretrain （100 epochs,finetune（100 epochs）, the accuracy is 67-68,but your accuracy is above 90， Is it related to parameter settings? can you tell me the parameter settings

Hi,
thanks for you sharing this code.

I have one question that you use the nn.Conv3d operation in L145 in PatchEmbed class to mitigate the patching process, which gets the patches in temporal and spatial dimensions. After this operation, you use x_vis = x[~mask].reshape(B, -1, C) # ~mask means visible to get the unmasked patches.

However, I am curious about whether this learnable nn.Conv3d module could extract the original masked image features, and resulting in the masked patches information leaking? Since the respective field of this 3D convolution may contain the masked patches in the original images.

I hope you could solve my confusion.

Fantastic job! If I try to finetune the model without deepspeed, will the performance drop? Thanks!

Hi, Thank you for your work!

I read your 'DATASET.md'.

Are there two key points in processing sthv2 datasets: the first is to change the suffix to MP4, the second is to reduce the short side to 320p? (and can only videos with an original height of 240p be zoomed out?)
中文：处理sthv2数据集是不是就2个核心要义：第一个是将后缀改为.mp4，第二个是将短边放缩至320p？（而且是不是只有原始高度为240p的视频才能被选中，然后再去放缩？）

Hi

What is the type of the inputs of the model? frames or videos?

Hi, thanks for your solid works!

Would you mind sharing some pretraining time statistics? Like how long does it take to pretrain on Kinetics400 using 64 V100s.

Hi, I have tried to reproduce the VideoMAE performance on SSV2 dataset. I run the experiments on four A100 machines (each includes eight 80G GPUs) and modify --nnodes=4, batch_size 64, such that our total batch size is the same. However, the performance is not consistent with the reported one. I checked the log you provided and notice that the learning rate is different. It seems that your learning rate is not 1.2e-3 (1.5e-4*2048/256) after the warm-up stage. Thanks a lot and I am looking forward to your reply.

I encontered missing keys error while finetunening, I resume from model pretrained with pretrain model and using it while finetune.
how to solve this problem?

Hello, thanks for your sharing again. I have downloaded the kinetic_400_vitl_epoch_1600 pretrained weight and trying to do visualization on videos. The vis.sh I run is same as the official except for the path. However, timm raise the error "RuntimeError: Unknown model (pretrain_videomae_base_patch16_224)" while loading the model.

I have tried to modify pretrain_videomae_base_patch16_224 to vit_base_patch16_224, then another error raised "TypeError: init() got an unexpected keyword argument 'decoder_depth'". Subsequently, I comment the keyword "deocder_path" in line80 of run_videomae_viz.py but the error occurs again "AttributeError: 'VisionTransformer' object has no attribute 'encoder'".

I really appreciate if you can help it. Thanks.

Could you release the preprocess code for ssv2?
That's will help me a lot.

Hi,
Thank you for awesome work.
I got a problem about the batch size. To be specific, when pre-training VIT-B on K400, the script sets the batch_size as 32, which means 32 videos per GPU. If one video clip consists of 16 frames (as set), one gpu will need to process 32*16=512 frames.
Is this right? Or do I misunderstand something?
Another problem, your paper reports the batch_size as 1024 when pre-training VIT-B, which is inconsistent with your scripts here.

Hi! I finetuned model on my dataset, but I'd like to resume from saved checkpoint .pt. But if I start the finetuning it always begins from 0 epoch.
My finetune.sh:

Set the path to save checkpoints

OUTPUT_DIR='/home/jovyan/people/Murtazin/VideoMAE/output_ckpts/eval_lr_1e-3_epoch_55'
# path to Kinetics set (train.csv/val.csv/test.csv)
DATA_PATH='/home/jovyan/datasets/sign_language/WLASL/WLASL_kinetic_hardcode'
# path to pretrain model
MODEL_PATH='/home/jovyan/people/Murtazin/VideoMAE/ckpts/checkpoint.pth' 
PT_PATH='/home/jovyan/people/Murtazin/VideoMAE/output_ckpts/eval_lr_1e-3_epoch_100/checkpoint-45/mp_rank_00_model_states.pt' 

# batch_size can be adjusted according to number of GPUs
# this script is for 64 GPUs (8 nodes x 8 GPUs)
OMP_NUM_THREADS=1 python -m torch.distributed.launch --nproc_per_node=8 \
    run_class_finetuning.py \
    --model vit_large_patch16_224 \
    --data_set WLASL \
    --nb_classes 2000 \
    --data_path ${DATA_PATH} \
    --resume ${PT_PATH} \
    --log_dir ${OUTPUT_DIR} \
    --output_dir ${OUTPUT_DIR} \
    --batch_size 2 \
    --num_sample 2 \
    --input_size 224 \
    --short_side_size 224 \
    --save_ckpt_freq 10 \
    --num_frames 32 \
    --sampling_rate 2 \
    --opt adamw \
    --lr 2e-3 \
    --opt_betas 0.9 0.999 \
    --weight_decay 0.05 \
    --epochs 55 \
    --dist_eval \
    --test_num_segment 5 \
    --test_num_crop 3 \
    --enable_deepspeed \

Hi, thanks for sharing your work, it's a brilliant job！

Even though you mentioned training and fine-tuning configurations on UCF101 and HMDB51 in the appendix of the paper, I would appreciate if you could provide pre-trained models for these two benchmarks, or provide scripts for training and fine-tuning.

Thanks.

Hi Zhan,

Thank you for your excellent work! We are surprised by VideoMAE's data efficiency (paper section 4.3), especially on small datasets like HMDB51. We are trying to use your code to reproduce your experiment on HMDB51. However, we cannot achieve the same finetune accuracy as the table2 reported (61.1 %):

Our experiments show that the model is converged after 2400 pretraining epochs. We are using eight Tesla V100 32GB GPUs. Also, we changed the batch size, learning rate and temporal stride as the appendix described.

I wonder if you can also share your complete experiment configurations for UCF101 and HMDB51? Some settings like warmup epochs number may also be critical for reproducing.

Hello author,

I saw you mentioned there were 8 GPUs used for pretraining 9.5k UCF101 video clips. May I ask what the total pretraining time for pretraining UCF101 for 3200 epochs using batch size 192 (the setting in your paper) is?

Appreciate it!

Hi,

I am trying to use data_clean.py for Kinetics 400 but am unable to use petrel_client from the import statement:
from petrel_client.client import Client
I have installed petrel but that has not worked and I have not been able to find a petrel_client package online.

Looking in into the code, I can find a linear layer on top of the pre-trained encoder that looks as follows (for kinetics400):

      (11): Block(
        (norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
        (attn): Attention(
          (qkv): Linear(in_features=768, out_features=2304, bias=False)
          (attn_drop): Dropout(p=0.0, inplace=False)
          (proj): Linear(in_features=768, out_features=768, bias=True)
          (proj_drop): Dropout(p=0.0, inplace=False)
        )
        (drop_path): DropPath(p=0.10000000149011612)
        (norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
        (mlp): Mlp(
          (fc1): Linear(in_features=768, out_features=3072, bias=True)
          (act): GELU()
          (fc2): Linear(in_features=3072, out_features=768, bias=True)
          (drop): Dropout(p=0.0, inplace=False)
        )
      )
    )
    (norm): Identity()
    (fc_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)
    (head): Linear(in_features=768, out_features=400, bias=True)

What I don't understand is, how exactly the layer is connected to the encoder. The encoder should deliver N (sequence length) tokens with d dimensions (768 in this case). Did you employ a CLS token (I don't think so), are you taking the mean over the N tokens or how exactly do you connect the encoder to the classification head? I cannot find any information on that in your paper, I might be wrong though - sorry in this case :)

Hello,

Thank you for sharing your work. I am looking at the code for finetuning the model and trying to understand how to apply mixup and cutmix to videos. The train dataloader seems to provide a batch of size (B, C, T, H, W). However, the mixup function from timm requires a batch of size (B, C, H, W). I couldn't find the code for reshaping the batch before sending it to mixup function. Am I missing something? Should we reshape the batch from (B, C, T, H, W) to (B, C*T, H, W) or (B*T, C, H, W)? Which is the correct way?

Thank you.

In modeling_finetune.py Line 82, why you set the bias of k to be 0 (requires_grad=False), which means the model only learns the bias parameters of q and v. The official timm code for image learns all the three biases of q, k, and v. Is there something special in videos?

BTW, why should the qkv be written in 84-85 lines instead of the 83 line which is commented out. Thanks!

def forward(self, x):
    B, N, C = x.shape
    qkv_bias = None
    if self.q_bias is not None:
        qkv_bias = torch.cat((self.q_bias, torch.zeros_like(self.v_bias, requires_grad=False), self.v_bias))
    # qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
    qkv = F.linear(input=x, weight=self.qkv.weight, bias=qkv_bias)
    qkv = qkv.reshape(B, N, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)
    q, k, v = qkv[0], qkv[1], qkv[2]   # make torchscript happy (cannot use tensor as tuple)

    q = q * self.scale
    attn = (q @ k.transpose(-2, -1))

    
    attn = attn.softmax(dim=-1)
    attn = self.attn_drop(attn)

    x = (attn @ v).transpose(1, 2).reshape(B, N, -1)
    x = self.proj(x)
    x = self.proj_drop(x)
    return x

Good job! After reading your VideoMAE, I feel this work should compare with MaskFeat [https://arxiv.org/abs/2112.09133] regarding to both method and performance. MaskFeat, the concurrent work with MAE (ImageMAE), also performs the masked image modelling for video classification task. The main difference between the MaskFeat and VideoMAE is that the reconstructed target is different (HOG feature vs. RGB image). Interest not relevant. It is just my personal opinion.

Hello, thanks for your effort on coding VideoMAE. I have followed the installation guide completely, then I selected two categories from UCF-101 to do downstream training (video binary classification). The data is preprocessed including resized the short edge to 240px or 320px as well as saved in .mp4 format. The train.csv, val.csv, test.csv are all prepared also. However, how could the training loss /accuracy are nearly contstants 0.69/0.5 throughout the training?

The dataset I specified is UCF-101 and the nb_classes is set as 2. The only other thing I modfied from the official script is the learning rate due to working on single GPU non-distributed environment. I have tried different multiple setting of learning rate from 64x to 0.015x, but all of them leads to constant losses. Could you please help me, thank you a lot.

@yztongzhan @wanglimin Thanks for your great works!

Could you please tell me how to extract features using the pre-trained model? Could you please provide a script as finetune and pretrain?

Thanks for your nice help!

And can you provide the train.csv file in the form of dataset_root/video_1.mp4 label_1 during the Data Preparation stage

Thank you very much

ModuleNotFoundError: No module named 'timm.optim.novograd'

it seems like timm just gives you nvnovograd if you ask for novograd

https://github.com/rwightman/pytorch-image-models/blob/master/timm/optim/optim_factory.py

elif opt_lower == 'novograd' or opt_lower == 'nvnovograd':
    optimizer = NvNovoGrad(parameters, **opt_args)

According to the code to load data, it seems we should add the length of each video when preparing datasets. See https://github.com/MCG-NJU/VideoMAE/blob/main/DATASET.md for details. The generated annotation of video datasets should be like:
dataset_root/video_1.mp4 100 label_1, where 100 means the length of video_1.mp4.