Maomao Li, Yu Li, Tianyu Yang, Yunfei Liu, Dongxu Yue, Zhihui Lin, Dong Xu

Reconstruction comparison between DDIM and STEM inversion. DDIM inversion in existing video editing methods usually exploits 1-frame or 2-frame context to invert each frame. Thus, we design a more radical inflated DDIM inversion that uses all-frame context as reference. Here, we use the typical DDIM reconstruction method to provide a video reconstruction comparison, where both our STEM inversion and the radical inflated DDIM one can explore context from the entire video, while the resource-consuming latter yields inferior performance.

TL; DR: STEM inversion is a efficient video inversion method for text-guided video editing.

The illustration of the proposed STEM inversion method. We estimate a more compact representation (bases $\mathbf{\mu}$) for the input video via the EM algorithm. The ST-E step and ST-M step are executed alternately for R times until convergence. The Self-attention (SA) in our STEM inversion are denoted as STEM-SA, where the $\rm{Key}$ and $\rm{Value}$ embeddings are derived by projections of the converged $\mathbf{\mu}$.

• 2023.12.11 Paper is released!
• 2024.05.01 The code based on TokenFlow editing is released!

• Release the STEM inversion code
• Release the code based on FateZero editing

Prepare the Conda environment using the following commands:

git clone https://github.com/STEM-Inv/stem-inv
cd STEM-Inv
cd TokenFlow-Edit
conda create -n stem-tf python=3.9
conda activate stem-tf
pip install -r requirements.txt

We provide demo source videos in the data folder. The corresponding config for STEM Inversion and Editing is in the configs folder. Below are the instructions for performing video editing on the provided source videos. You can run the following command to perform inversion and editing process at once:

bash run_editing.sh

The inversion results are saved in Stem_Inv_Latents/base_256_iter_5, and the editing results are saved in STEM_TF_results.

If you are only interested the reconstruction results of STEM inversion, please run:

bash run_inversion.sh

Note that our default setting is to use 256 bases to represent the whole input video, where 5 iterations are applied for EM algorithm convergence. You can also consider other configurations by modifying the values of “num_bases” and "n_iters" in line 88 of tokenflow_utils.py.

Source prompt: A man is playing tennis.      Target prompt: Spider-Man is playing tennis.

@article{li2023video,
  title={A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing},
  author={Li, Maomao and Li, Yu and Yang, Tianyu and Liu, Yunfei and Yue, Dongxu and Lin, Zhihui and Xu, Dong},
  journal={arXiv preprint arXiv:2312.05856},
  year={2023}
}

This is official code of STEM Inversion. All the copyrights of the demo images and audio are from community users. Feel free to contact us if you would like remove them.

The code is built upon the below repositories, we thank all the contributors for open-sourcing.

• TokenFlow