World Models: The Next Frontier of Artificial Intelligence
How AI is evolving from predicting patterns to understanding, simulating, and acting in the world.
Introduction
Ask a modern AI to write an essay about gravity and it will produce something elegant. Ask a robot arm to catch a falling glass, and it will probably watch it shatter.
That gap is the story of the last two years in AI. Talking about a world and understanding one turn out to be different skills. A three-year-old who has never read a physics textbook knows a ball rolling behind a sofa will come out the other side. She has an internal picture of how things behave, and she runs little simulations in her head before she acts.
That picture has a name in AI research: a world model. Yann LeCun left Meta in November 2025 to found a startup dedicated to it, raising $1.03 billion by March 2026, and Google DeepMind, NVIDIA, Waymo and Wayve have all shipped world models as flagship products. Here is what they are, how they work, and why many think they are what comes after chatbots.
What Are World Models?
A world model is an AI system that builds an internal representation of an environment and predicts how it changes over time in response to actions.
Three things are packed into that sentence:
Internal representation. It compresses what it sees into a compact description of the situation — not pixels, but state: "a cup near the table edge, tilting."
Over time. It is about dynamics, not snapshots. What happens next?
In response to actions. Nudge the cup and one future unfolds; leave it and another does.
The useful analogy is a flight simulator the AI builds for itself, from experience, and can then fly inside. Once you have one, you can try an action a thousand times in imagination before committing to it once in reality.
The idea isn't new — Schmidhuber proposed networks that predict future world states in 1990. What changed is scale: we now have the video data, the compute and the architectures to make these simulators good.
World Models vs Traditional AI
Most AI you have used is reactive: it maps an input to an output — image in, label out; prompt in, text out. A world model is predictive about consequences instead.
The practical consequence is planning. A reactive system picks whatever looked good in similar training data. A world-model agent rolls a candidate plan forward in imagination, sees it fail, and picks another — without paying the real cost of the failure. For a robot or a self-driving car, that is the difference between learning from accidents and learning instead of having them.
How Do World Models Work?
Strip away the engineering and the loop is simple, repeating dozens of times a second:
Perceive. Compress the raw observation — a camera frame, a lidar sweep — into a compact internal state.
Predict. Given that state and a candidate action, predict the next state. Repeat to roll the future forward.
Evaluate. Score the imagined futures. Did the plan reach the goal? Did anything break?
Act. Execute the first step of the best plan, then correct the model where prediction and reality diverged.
That last step is the engine of learning: the model improves precisely where it was surprised.
The crucial design decision is where prediction happens. Predicting video pixel-by-pixel burns capacity on carpet texture when all that matters is that the cup is about to fall. Predicting in a compressed latent space is what makes the approach tractable.
Technical Architecture of a World Model
The architecture can be understood as a closed-loop AI system that observes the world, builds an internal representation, imagines possible futures, chooses an action, and then learns from the result.
1. Real World — Where the data originates
The system starts with the environment it needs to understand.
This could be:
A physical environment for a robot
A road for an autonomous vehicle
A game environment
A hotel or pricing ecosystem
A manufacturing plant
A financial market
A digital application
The environment continuously produces signals such as images, transactions, events, sensor readings, user behavior, and external information.
2. Perception Layer — Turning observations into information
The Perception Layer converts raw signals into information that the AI can process.
It can receive:
Vision: images and video
Audio: speech and environmental sounds
Text: documents, conversations and logs
Sensors: IoT, robotics and telemetry
Structured data: APIs, databases and event streams
For example, an autonomous vehicle may receive camera images, LiDAR data, GPS coordinates and radar signals.
The objective is:
Raw world → Machine-readable observations
3. Representation Encoder — Creating the internal state
Raw observations are still too complicated to reason over directly. The Representation Encoder compresses them into a meaningful latent state.
We can represent this as:
The representation can contain:
Entities
Objects
Relationships
Context
Historical information
Temporal information
Uncertainty
For example, instead of remembering every pixel in an image, an autonomous-driving system might represent:
Car A is 20 meters ahead, moving at 40 km/h, while a pedestrian is approaching the crossing.
This compressed representation is much more useful for reasoning.
4. World Model — The Core of the Architecture
This is the most important component. The World Model learns how the environment changes over time. A simplified formulation is:
This means:
Given the current state zt and an action at, what is the probability distribution of the next state z(t+1) ?
For example:
Current state
Hotel occupancy = 70%Price = $100Competitor price = $95Large concert tomorrow
Action
Increase price to $120
World model predicts
Demand may decrease
Competitors may change their prices
Occupancy may evolve
Revenue may increase or decrease
The market may behave differently depending on demand elasticity
The world model therefore learns dynamics, rather than simply predicting a single target.
5. Imagination / Future Simulation
This is where world models become particularly powerful. Instead of immediately taking an action, the AI can ask:
"What might happen if I do this?"
The model can simulate multiple possible trajectories.
For example:
Current State
│
├── Action A ──► Future A1 ──► Future A2
│
├── Action B ──► Future B1 ──► Future B2
│
└── Action C ──► Future C1 ──► Future C2
These are essentially internal simulations. The AI does not necessarily need to perform every action in the real world to understand its potential consequences. This is sometimes called planning in imagination.
6. Planning / Decision Making
The simulated futures are then evaluated by the planning layer. Different techniques can be used here:
Model Predictive Control — MPC
Continuously:
Predict → Evaluate → Act → Observe → Re-plan
Monte Carlo Tree Search — MCTS
Explore many possible action sequences and estimate their outcomes.
Reinforcement Learning
Use predicted future states and rewards to improve the policy.
Optimization
In business applications, incorporate explicit constraints such as:
Revenue
Cost
Risk
Capacity
Availability
Business rules
This layer answers:
"Given everything the world model predicts, what should I do?"
7. Policy / Agent — Selecting the Action
The Policy/Agent converts the planning result into an actual decision.
For example:
Robot: Move forward.
Car: Brake.
Game agent: Attack.
Pricing agent: Increase price by 8%.
Business agent: Launch a targeted promotion.
The agent may also use external tools such as:
APIs
Search
Databases
LLMs
Business systems
Robotic actuators
This is where the world model becomes connected to an agentic system.
8. Action — Affecting the Real World
The selected action is executed. For example:
World Model
↓
Prediction
↓
Planning
↓
Policy
↓
Action
↓
Real World
But the process doesn't stop here. The environment changes because of the action. That new state becomes the next observation.
9. Feedback Loop — The Most Important Part
The architecture is fundamentally a closed loop.
OBSERVE
↓
UNDERSTAND
↓
IMAGINE
↓
EVALUATE
↓
ACT
↓
LEARN
↺Suppose the model predicted that increasing a hotel price from $100 to $120 would reduce occupancy by 3%. After executing the action, actual occupancy might fall by only 1%. That difference becomes valuable feedback. The system can update its understanding of:
Price → Demand → Occupancy → Revenue
Over time, the world model becomes better calibrated to the environment.
10. Memory System
The architecture also contains a dedicated Memory System.
Memory can operate at different timescales:
Short-term memory
What happened recently?
Long-term memory
What has historically happened?
Experience replay
Which previous situations and actions were useful or unsuccessful?
For an autonomous agent, memory can prevent the system from repeatedly making the
same mistakes.
11. Knowledge & Tools
A world model doesn't need to contain all possible knowledge internally. It can connect to external systems.
For example:
World Model
│
├── LLM
├── Search
├── APIs
├── Databases
├── Knowledge Graph
└── Domain-specific toolsThis is particularly important for modern agentic AI architectures. The world model handles environment dynamics while external tools provide fresh information and capabilities.
12. Monitoring & Evaluation
A production world model needs continuous evaluation.
Important metrics can include:
Prediction accuracy
Future-state prediction error
Calibration of uncertainty
Planning quality
Reward achieved
Action success rate
Safety violations
Distribution shift
One particularly important problem is:
Prediction error accumulation
If the model makes a small error at every timestep:
the error can compound as the prediction horizon increases. This is one of the fundamental challenges of world models.
13. Continuous Learning
The final component closes the loop. New observations and outcomes can be used to update the model. The system therefore becomes:
Experience → Learning → Better World Model → Better Simulation → Better Decisions
This is fundamentally different from a static predictive model that is trained once and deployed.
Where Do LLMs Fit in World Models?
LLMs already have world models.
The strongest evidence is interpretability work. Othello-GPT — a small transformer trained only on move sequences, never shown a board or told the rules — turned out to encode the board state internally. Researchers could linearly probe it out, and crucially, edit it: flip a piece in the activations and the model's next move changes accordingly. That's not a correlation artifact; that's a causal internal representation. Similar board-state structure shows up in chess-playing models, and estimates of latent variables (like player skill) emerge too. The argument: predicting text well enough, at scale, forces some model of the generating process into the weights. You can't compress well without understanding.
They have nothing of the sort.
The sharpest counter-evidence is Vafa et al.'s Manhattan taxi study (NeurIPS 2024). They trained a model on turn-by-turn routes, and it gave near-perfect directions — above 99% accuracy. Then they reconstructed the street map its internal model implied. It was a city with streets that don't exist and flyovers crossing over other roads at impossible angles. The model had learned a large bag of route fragments that happened to work, not a map. Detour-robustness collapsed the moment they closed a few streets. High task performance, incoherent underlying model.
They have fragmentary ones.
This is where most researchers have landed, and I think it's right. Both results are true. LLMs build locally coherent patches of world structure — enough to be genuinely useful — that don't stitch into one consistent global model. Ask about physics in the abstract and you get a good answer; ask a question that requires composing three facts into a
novel physical situation and the patches don't line up.
How they actually get combined in practice?
Three patterns are in production right now:
1. LLM as planner, world model as simulator. The language model decomposes a goal into subgoals — "clear the table, don't touch the laptop" — and the world model handles the physical rollout of each step. Clean separation: semantics upstairs, dynamics downstairs. Most robotics stacks work this way today.
2. Fused multimodal architectures. NVIDIA's Cosmos 3 uses a mixture-of-transformers that pairs reasoning and generation in one model — it reasons about a scene, predicts future states, and emits robot actions. The language-like reasoning and the physical prediction share representations rather than passing messages across an interface.
3. Vision-Language-Action (VLA) models. A VLM backbone provides semantic grounding and instruction-following; an action head is trained on robot demonstrations. The LLM contributes "what is a mug and what does 'tidy' mean," the action layer contributes "what torque sequence."
The pattern across all three: the LLM supplies the ontology, the world model supplies the dynamics. Language is how you name goals, objects, and constraints. It's not how you predict trajectories.
Why the Latent Space Matters
If you take one technical idea from this article, make it this one.
A latent space is a compressed, learned coordinate system for "situations." Instead of two million pixel values, you describe a scene with a few hundred numbers capturing only what is predictively relevant: positions, velocities, materials, relationships. Three reasons it is the crux:
Imagination becomes cheap. Rolling 500 candidate futures through a pixel generator is prohibitive; through a small latent dynamics model it is fast enough for a robot's control loop. Dreamer 4 runs interactive world-model inference in real time on a single GPU.
It filters out the unpredictable. Video generators waste capacity on detail nobody can predict — the exact shape of every leaf. A JEPA-style model is free to be vague about the leaves and precise about the branch that is about to snap.
It is where abstraction lives. A good latent space clusters "a cup sliding toward an edge" regardless of the cup's colour or the table's grain, which is what lets a model trained in one kitchen work in another. A 2026 formal result showed a JEPA-style objective provably recovers true variables like position and velocity — though the guarantee proved fragile in practice.
Training World Models
Training runs in three stages, and the data requirements are forgiving:
1. Self-supervised pretraining. The model watches enormous quantities of unlabelled video and learns to predict what comes next. No annotation needed; the future is its own label. This is why video, not text, is the raw material of choice.
2. Action conditioning. A tiny amount of data pairing observations with actions taken teaches the model to answer counterfactuals. V-JEPA 2 needed just 62 hours of robot data atop a million hours of passive video: the physics comes from watching, the controllability from a thin layer of interaction.
3. Training the agent inside the model. Once the simulator is good, the policy learns by reinforcement learning entirely in imagination. The landmark result is Dreamer 4 (2025): the first agent to obtain diamonds in Minecraft purely from offline data, with no environment interaction — over 20,000 correct actions from raw pixels — using roughly 100× less data than the previous offline approach. That is the thesis in miniature: learn a simulator from passive data, then practise inside it for free.
Key Technical Challenges
The honest state of play, as of late 2026:
Physics is still shaky. On IntPhys 2, which asks models to tell plausible scenes from physically impossible ones, humans score 85–95%. Leading video models perform at or near chance.
Brittleness. A 2026 evaluation platform found an agent succeeding ~51% of the time on a manipulation task, dropping to ~12% when the agent's colour changed and ~6% on a background shift. It predicted frames accurately while planning badly — latching onto appearance instead of geometry.
Long-horizon drift. Per-step errors compound. Genie 3 holds consistency for minutes and remembers about a minute back; beyond that, worlds quietly stop being the same world.
Cost. Training on video is orders of magnitude more expensive per useful bit than training on text.
Evaluation is unsolved. "Looks realistic" is not "is correct."
Causality vs correlation. Predicting what follows is not knowing what causes what, and only the latter survives intervention.
Applications of World Models
Robotics. The flagship case. Robots cannot afford to learn by breaking things, and real-world data collection is brutally slow. Training in a learned simulator turns a hardware problem into a compute problem — the aim of NVIDIA's open-weight Cosmos 3 (May 2026), with early adopters including Agile Robots and Samsung Electronics.
Autonomous driving. The highest-stakes deployment today. Waymo's World Model (Feb 2026), built on Genie 3, generates synchronised camera and lidar data and lets engineers rewrite a real recorded drive: change the weather by text prompt, alter the road layout, or ask "what if we had braked half a second earlier?" Wayve's GAIA series does the same for safety validation. Dangerous scenarios nearly impossible to capture on real roads can be manufactured on demand.
Industry, media and science. Any physical system where mistakes are expensive — factory lines, warehouses, grids, supply chains — is a candidate for simulating an intervention before making it. Genie-class models also generate playable worlds from a sentence, doubling as an unlimited training ground for agents, and world models increasingly serve as learned surrogates for expensive physical simulations.
The Future of World Models
Hybrid systems win. Not LLMs replaced by world models, but stacks where a language model handles goals and abstract reasoning while a world model handles grounded prediction. Most serious robotics platforms already work this way.
World models become infrastructure. Just as nobody trains an image encoder from scratch anymore, few teams will train a world model from scratch. Open foundation models like Cosmos point to fine-tuning a general physical prior on your own domain.
Evaluation gets serious. The field is shifting from "does the video look plausible?" to formal tests of whether a model's internal state recovers real, manipulable structure — a sign of maturity, and a large share of current results are bluntly unflattering.
Underneath it all is LeCun's bet: that scaling text prediction alone eventually hits a ceiling, and that intelligence grounded in physical prediction is the more durable path. Testing it will take years — AMI Labs raised over a billion dollars with no expected product for roughly five.
Conclusion
A world model is the machine equivalent of the simulation you run in your head before stepping off a kerb. It compresses what it observes into a compact state, predicts how that state evolves under your actions, and lets you rehearse consequences before paying for them. That unlocks what language alone cannot: planning under genuine novelty, safe learning where mistakes are expensive, and reasoning about what would have happened rather than only what was written down.
The technology is real but early. Today's world models generate stunning interactive video and can train an agent to mine diamonds it has never mined — and they still fail physics tests a toddler passes, and get confused when you change the colour of a wall. Both are true at once.
If the last four years were about teaching machines to describe the world, the next few look like teaching them to predict it. A much harder problem, and a far more useful one.
References
Ha, D. & Schmidhuber, J. — World Models (2018). worldmodels.github.io
Wikipedia — World model (artificial intelligence). en.wikipedia.org
Google DeepMind — Genie 3: A new frontier for world models (Aug 2025). deepmind.google
Meta AI — Introducing the V-JEPA 2 world model and new benchmarks for physical reasoning (2025). ai.meta.com
Hafner, D., Yan, W. et al. — Dreamer 4: Training Agents Inside of Scalable World Models (2025). danijar.com · arXiv:2509.24527
Hafner, D. et al. — Mastering diverse control tasks through world models (DreamerV3), Nature (2025). nature.com
NVIDIA — Cosmos World Foundation Model Platform for Physical AI. arXiv:2501.03575
NVIDIA Newsroom — NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI (May 2026). nvidianews.nvidia.com
Waymo — The Waymo World Model: A New Frontier For Autonomous Driving Simulation (Feb 2026). waymo.com
Wayve — GAIA-3: Scaling World Models to Power Safety and Evaluation. wayve.ai
Li, K. et al. — Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (Othello-GPT). arXiv:2210.13382
Vafa, K., Chen, J., Rambachan, A., Kleinberg, J. & Mullainathan, S. — Evaluating the World Model Implicit in a Generative Model, NeurIPS (2024). arXiv:2406.03689 · MIT News
Klindt, D., LeCun, Y. & Balestriero, R. — When Does LeJEPA Learn a World Model? (May 2026).
Maes, L. et al. — stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation (May 2026).
Wikipedia — Advanced Machine Intelligence Labs. en.wikipedia.org
TechCrunch — Yann LeCun's AMI Labs raises $1.03B to build world models (Mar 2026). techcrunch.com
Comments