AnimateDiff is a tool that allows you to animate your personalized text-to-image diffusion models without specific tuning. When integrated with ComfyUI, it can be used to create a variety of animations and visual effects. In the vast landscape of digital creativity, AnimateDiff emerges as a game-changing tool, bringing life to text-to-image diffusion models without the hassle of intricate adjustments. Picture a world where personalized animations and visual effects come to life effortlessly, thanks to the seamless integration of AnimateDiff Evolved with ComfyUI. In this article, we will learn how we can use AnimateDiff Evolved with ComfyUI to create different gifs, animations or videos.
Table of Contents
What is AnimateDiff?
AnimateDiff is a tool that adds limited motion to Stable Diffusion generations, which are primarily text-to-image and image-to-image transformations. It was developed based on a research paper by Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. AnimateDiff can be used with Stable Diffusion to generate animations from images, with no fine-tuning required. It can also be used in conjunction with ControlNet for video-to-video transformations.
Features of AnimateDiff
1. txt2img & img2img Support
AnimateDiff supports both text-to-image and image-to-image transformations, adding limited motion to the generated images.
2. video2video
It also has a video-to-video implementation that works in conjunction with ControlNet.
3. Customizability
AnimateDiff allows you to experiment with various text prompts to generate different animation outcomes.
4. Integration with Stable Diffusion
It can be integrated with AUTOMATIC1111 Stable Diffusion WebUI and ComfyUI, enabling you to generate GIFs in the same way as you create images.
How to Install AnimateDiff with ComfyUI? Step-by-Step Guide
To use AnimateDiff with ComfyUI, you need to follow these steps:
1. Install ComfyUI
First, install ComfyUI. Here is our step-by-step guide to help you install ComfyUI and use it with Stable Diffusion.
Stable Diffusion SDXL v1.0 and ComfyUI: How to Install and Use
2. Download ComfyUI Manager
Next, install the ComfyUI manager to download extensions and custom nodes. You can find it on Civit AI, along with useful instructions. Save the file in your ComfyUI directory, extract it, and run the batch file to install the ComfyUI manager. If you have an Nvidia GPU, use the Run Nvidia GPU batch file; otherwise, go with the CPU option.
3. Install AnimatedDiff
To install AnimateDiff, open the ComfyUI manager, click on “Install Custom Nodes,” and search for AnimateDiff. Install the one by Kosinkadink. After downloading, restart ComfyUI.
4. Download the Motion Modules
You need to get the motion models from Hugging Face page.
Download the following checkpoints/Motion Modules (You will need at least 1) :
- mm_sd_v14.ckpt
- mm_sd_v15.ckpt
- mm_sd_v15_v2.ckpt (Use this as it is latest one)
You can also download these from Google Drive, CivitAI, or Baidu NetDisk. You can also download stabilized finetunes of mm_sd_v14, mm-Stabilized_mid, and mm-Stabilized_high, and finetunes of mm_sd_v15_v2, mm-p_0.5.pth, and mm-p_0.75.pth, all by manshoety, from HuggingFace. For higher resolution finetuning, you can use temporaldiff-v1-animatediff by CiaraRowles from HuggingFace.
Optionally, you can use Motion LoRAs to influence movement like v2_lora_PanLeft.ckpt . Download the MotionLoRA models available at Google Drive / HuggingFace / CivitAI ) and save them to the models/MotionLoRA folder.
Motion LoRA only supports V2 motion modules.
5. Place the Models
Now paste these checkpoints in the models section of ComfyUI using the path
- ComfyUI_windows_portable > ComfyUI > custom nodes > ComfyUI-AnimatedDiff-Evolved > models.
6. Download and Place VAE File
Finally, head over to this page to download the VAE file. Go to the respective Hugging Face page. Make sure to pick the safe Tensor version.
Go to Comfy UI models. Open the VAE folder and save the file there.
ComfyUI_windows_portable > ComfyUI > models > vae
5. Restart ComfyUI
Now, restart Comfy UI. Close the previously opened tab and the terminal, then run the batch file again to start Comfy UI in your browser.
6. Import the AnimateDiff Workflow
You can find a good starting workflow in the AnimateDiff guides or download the workflow from GitHub page. Drag and drop the image into your interface to instantly load the workflow.
7. Adjust Settings
Let’s go through the AnimateDiff settings. The first node allows you to select the AI model to load. Choose the model that suits your preferences. You can change the seed value and output dimension. Adjust the batch size, steps, CFG value, sampler, and scheduler. Set the frame rate and click on the Queue prompt button to start creating animations.
8. Provide Prompts
Prompting is a crucial part of the generative process. Modify the prompt to suit your animation project. Green represents the positive prompt, while red signifies the negative prompt. Customize these prompts to align with your creative vision.
9. Download Your Output
Click on the Queue prompt button to start generating animations. Each animation is safely stored in the output folder. You can download from there and use them in your videos or animations.
10. Get Creative
Experiment with different settings and workflows. If it works for normal image generation, it probably will work for AnimateDiff generations. You can try latent upscales, stacking ControlNets, masking the conditioning of ControlNets, and more.
Benefits of Using AnimateDiff Evolved with ComfyUI
1. Animation
AnimateDiff can be used to generate animations from images, making it useful for creating GIFs.
2. Video Editing
It can be used for video-to-video transformations, which can be useful for video editing tasks.
3. Educational Purposes
It can be used to teach about AI and machine learning, especially in the context of image and video generation.
Challenges & Limitations
1. System Requirements
AnimateDiff requires more RAM and GPU capabilities for optimal performance.
2. Limited Motion
While AnimateDiff has improved considerably since its inception, the motion it can add to images is still limited.
3. Quality of Outputs
The quality and smoothness of the outputs produced by AnimateDiff may not always be perfect.
4. Complexity
Depending on the method used to run AnimateDiff, it may require various amounts of effort to get working. For example, running AnimateDiff on Google Colab may require careful monitoring of VRAM usage.
5. Learning Curve
While AnimateDiff can be used by users with different levels of experience, it may have a steep learning curve for beginners.
Final Verdict
As we conclude our exploration of AnimateDiff Evolved and its dynamic partnership with ComfyUI, it’s evident that a new realm of creative expression awaits. Whether you’re crafting mesmerizing GIFs, engaging animations, or diving into the realm of video transformations, AnimateDiff opens doors to endless possibilities. While challenges like system requirements and a learning curve exist, the payoff is a canvas where your imagination knows no bounds. Embrace the art of animated storytelling and redefine your creative horizons with AnimateDiff and ComfyUI.
If you like this article, kindly let us know your thoughts in the comment section below.
| Also Read:
