Modern AI has reached incredible heights in generating photorealistic images, animations and videos from simple text prompts. However, one area that still poses challenges is controlling the precise motions and trajectories of objects depicted in synthesized media. Enter Boximator – a novel approach that brings unprecedented control over video synthesis through intuitive box constraints. This is developed by researchers at ByteDance, the parent company that owns TikTok.
Table of Contents
Introducing Boximator by ByteDance Research
Boximator introduces the novel concept of using box-shaped constraints to define objects and control their movement across frames. It functions as a plug-in for existing video generator models like PixelDance and ModelScope, preserving their abilities while enabling unprecedented control over motions. It enables users to define rich motions simply by drawing boxes around objects or tracing motion paths. This grants all kinds of creators a powerful and easy-to-use way to craft videos according to their exact vision.
Example Videos Generated by Boximator
How Boximator Works?
At its core, Boximator utilizes two types of boxes – hard and soft. Hard boxes represent objects and their precise positions/shapes at keyframes. Soft boxes, on the other hand, define loose regions where entities can flexibly move within over time.
In frames without user boxes, Boximator infers approximate motion paths using algorithm-generated softboxes for guidance. It associates each box with a unique object ID, allowing multiple objects to be tracked and manipulated individually over time.
Through techniques like interpolation and constraint relaxation, Boximator intelligently places soft boxes between hard constraints to achieve controllable yet natural-looking animations. This grants both flexibility and stability – an important balance for producing believable motions.
Self-Supervised Pretraining Approach
Boximator adopts a self-supervised pretraining approach. It lets models generate visible bounding boxes around objects in every frame. This decomposes the problem into easier sub-tasks of detecting objects and controlling box motions separately. Once pre-trained, the generated boxes can then be removed, leaving behind powerful spatial-temporal reasoning abilities tuned for motion control. Extensive ablation studies proved this approach integral to Boximator’s success.
Boximator vs Other Methods
This approach offers several benefits over prior motion control methods. Boximator uses visually-grounded boxes for object selection and motion control. This avoids the need for verbose text descriptions. Other methods rely on text descriptions to specify objects and regions. Empirical studies have shown that this AI achieves state-of-the-art video quality scores, surpassing two base models. The incorporation of box constraints further enhances video quality and motion control.

The bounding box alignment metric demonstrates the robust motion controllability provided by this model. Moreover, user evaluation has indicated that users prefer the results generated by the Boximator over those produced by the base models. Moreover, ByteDance’s Tiktok Boximator leaves behind Pika 1.0 and Gen-2 in motion control, as described in the prompt. Check the example below:
A Game-Changer in the World of Video Synthesis
With the release of Boximator, TikTok has taken a giant leap forward in fine-grained motion control for AI-generated videos. This revolutionary technology empowers users to manipulate and control motions with unprecedented precision and ease. The combination of flexibility, user-friendliness, and state-of-the-art video quality makes Boximator a game-changer in the world of video synthesis.
| Also Read: Unbelievable! TikTok’s AI Learns Like a Human – Without Any Help!







One Response