Type "a red kite over a grey sea" into a text-to-video AI tool and a few seconds later you get moving footage. Nothing in that sentence tells you what happened in between, and the in-between is where every limitation of these tools comes from: the odd hands, the object that melts after two seconds, the clip that cannot be longer than a few seconds.
This is the path one prompt takes, built on how the public papers describe it. The closed models do not publish their details, so where a step is general practice rather than a confirmed fact about a given product, it says so.
Before: how a still image is made
Image generators such as Stable Diffusion start from random noise and remove it step by step until a picture matches the text. The Latent Diffusion Models paper (Rombach et al., CVPR 2022) made this cheap by running diffusion in the latent space of a pretrained autoencoder rather than on pixels, and by adding cross-attention layers so text can steer the result. That paper's abstract says the approach significantly reduces computational requirements compared with pixel-based diffusion.
A video generator reuses that recipe and adds one hard demand: every frame has to agree with the others.
What the video step adds
The list below is what differs from still images, in the order a reader usually hits it.
- Compression covers time. OpenAI's February 15, 2024 report on Sora says videos are compressed into a lower-dimensional latent space and cut into spacetime patches, which act as the tokens of a transformer. Generation happens in that compressed space, and a decoder maps it back to pixels. The report also says model and implementation details are not included.
- Noise is removed across frames together. Denoising each frame alone would give flicker. Attention that looks across frames lets the model keep a subject in place from one frame to the next.
- Motion is learned from video. AnimateDiff takes a different route: its paper describes a plug-and-play motion module, trained once, that can be inserted into personalized text-to-image models built on the same base model. A companion, MotionLoRA, adapts it to new motion styles such as camera moves.
- Data matters as much as architecture. The Stable Video Diffusion paper names its training stages as text-to-image pretraining, video pretraining and high-quality video finetuning, and argues that a curated dataset, with captioning and filtering, is needed for good video.
Sora, for the record, was shut down by OpenAI in 2026, and this site is independent and not affiliated with OpenAI. The report remains the clearest public description of the approach, and our look at what the shutdown taught later tools covers the product side.
Where it still fails, and why
Time consistency is a learned habit, not a rule the model enforces. The model has no explicit object that must persist, only patterns of how pixels tend to change, so objects drift, merge or vanish as a clip lengthens. Treat any single demo reel as a best case selected by the developer.
For local work the same logic shows up as memory. Video models hold many frames at once, so VRAM becomes the limit before speed does, and our VRAM guide lists the figures the projects publish. If you want to see the AnimateDiff idea as an actual node graph, the ControlNet plus AnimateDiff walkthrough shows one.
Where to go from here
To see what a prompt does, a hosted tool is the short route. To steer pose, camera or style, the local route with a motion module is where that control lives, at the price of setup and memory.