What is AnimateDiff?
AnimateDiff is an Ai text to video technique to combine generated stationary images with motion dynamics. The paper is here.
What can AnimateDiff do?
AnimateDiff can create short animations from your prompts. There are presets of models and prompts (in yaml files) you can try. In my opinion, the animations of human movement are not ready for practical use. But some of scenery animations are stunning.
How to generate animation of scenery using AnimateDiff?
1. Go to Google colab for AnimateDiff.
2. Comment out the line to install toonyou_beta3 model. Uncomment the lines to install CounterfeitV30_v30 and lora_Ghibili_n3 models.
3. Run the first block to setup the environment. Click “Run anyway” to close the popup. Wait for the green check mark.
4. In the left panel, click folder icon to expand it. A new directory “animatediff” is created.
5. Under the “models\Motion_Module”, double check two models “mm_sd_v14.ckpt” and “mm_sd_v15.ckpt” are there.
6. Under the “models\DreamBooth_LoRA”, double check two models “CounterfeitV30_v30.safetensors” and “Pyramid%20lora_Ghibli_n3.safetensors” are there. Rename the second one to “lora_Ghibli_n3.safetensors.”
7. Under “configs\prompts”, there are 8 yaml files. Since we are going to generate scenery, we use “8-GhibliBackground.yaml”. Click to open this file, change base to “CounterfeitV30_v30.safetensors”.
8. In the second block of the code, comment out the first line which uses toonyou yaml file. Uncomment the last line to use GhibliBackground yaml file.
9. Run the second block. While it is running, check the left file directory. A “samples” directory is created.
10. Expand the directory. When a new animation is generated, it shows as a gif file. Click the file to see the result at right side of the screen.
Screenshot
