Introducing an AnimateDiff retrain over the sd-inpainting model in order to take an input image

Input	Prompt	Output
	Man on a Beach
	Taylor Swift
	Woman on a Beach

1. Download Motion Module

   • Motion module from HuggingFace
   • Place it in the motion_module folder.

2. Configuration Keys:

   • base: Set this to "models/StableDiffusionInpainting". This should point to the diffuser's inpainting model available here.
   • vae: Use "models/StableDiffusion". This must link to the original 1.5 stable diffusion model due to a diffusers issue. Get the vae from here.
   • path: Specify something like "models/DreamBooth_LoRA/realisticVisionV20_v20.inpainting.safetensors". It must be your Dreambooth model and needs to be an inpainting model. Note: You can convert existing models to inpainting models by adding the difference from the inpainting model and the standard model to any custom model.

3. Execution:

   python -m scripts.animate --config "configs/prompts/5-RealisticVision.yaml" --image_path "/images/image.png" --W 256 --H 256

• This model currently only works at roughly 256x256 resolutions. Retraining it to 512x512 didn't work for some reason, so you'd be best just upscaling with comfyui for now.

• In terms of making it work well, consider the prompts are not you telling the model what you want with this, you're guiding the generation for what is in the input image, if the input image and the prompts do not align, it will not work.

• You may have to try a few seeds per generation to get a nice image, it's a tiny bit unreliable.

Special thanks to the developers behind AniamteDiff. https://github.com/guoyww/AnimateDiff.