face  animation using ai

What is Thin-Plate Spline Motion Model?

Thin-Plate Spline Motion Mode is an Ai model to add movement to a still image based on the driving video. The paper is here. One of its usage is to do face animation using Ai technology.

How to create face animation using the Ai model from GitHub?

1. First download the code from Thin-Plate-Spline-Motion-Model GitHub repository. Open a dos prompt, in the directory where you want to install, type the command:
>git clone https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model.git
2. Still in the dos prompt, run the following commands
>cd Thin-Plate-Spline-Motion-Model
>mkdir checkpoints
>pip3 install wldhx.yadisk-direct
>curl -L $(yadisk-direct https://disk.yandex.com/d/i08z-kCuDGLuYA) -o checkpoints/vox.pth.tar
This will download vox.path.tar in the “checkpoints” directory.
3. Now it is time to configure a virtual environment to run the code. If you haven’t installed Anaconda3, go to install Anaconda3.
4. Then you setup a conda environment by the instruction. If you run into problems when you setup, you can download a pre-configured pytorch_env at Gumroad (compatible with CUDA 11.8). Unzip it and put it at your anaconda3 installation under “envs” directory.
5. Prepare your input files. The first one is a short video mp4 file with a face movement. The dimension is 512 x 512. Name it as “driving.mp4.” Put it at “assets” directory.
6. The second file is an image with a face that you want to animate. Note the mouth of the character should be open to show a little bit upper teeth. The dimension is 512 x 512. Name it as “source.jpg” and put it at “assets” directory.
7. Download main.py and put it at “Thin-Plate-Spline-Motion-Model” directory.
8. Open an Anaconda Prompt. Run command:
>conda activate pytorch_env
9. Still in the Anaconda prompt, go to the directory “Thin-Plate-Spline-Motion-Model” and run command:
>python main.py
10. When it finishes, the new video is called “generated.mp4” under “Thin-Plate-Spline-Motion-Model” directory.
11. (Optional) You can enhance resolution of the animation using Real-ESRGan.