Meta JEPA: Non-Generative World Models

Hi, this is Alex from DEV STORIES DOT EU. Meta JEPA: Non-Generative World Models, episode 1 of 4. If you want to teach a neural network what a forest is, asking it to memorize the exact location of every single leaf is the worst way to do it. Yet, when we train models to understand images and video by predicting missing parts, that is exactly the trap we fall into. The solution lies in understanding The Limits of Pixel Prediction. To build models that comprehend visual data, a standard approach is to hide sections of an input and ask the network to fill in the blanks. Masked Autoencoders do exactly this. They take an input, mask out patches, and train the model to reconstruct the missing pixels. If the model can rebuild the image perfectly, the assumption is that it understands the scene. However, this has a cost. Generative methods must spend significant capacity and compute on low-level pixel detail. Consider a video of a car driving down a tree-lined street. The car follows a predictable path. The leaves on the trees, however, flutter in the wind. Their exact pixel values in the next frame are chaotic and fundamentally unpredictable. When a model operates in pixel space, its loss function penalizes it for getting the background texture wrong. It is forced to treat the predictable car and the unpredictable leaves as equally important reconstruction tasks. The network wastes heavy computational resources trying to generate the exact texture and position of every single leaf. This is where the Joint-Embedding Predictive Architecture, or JEPA, diverges. People sometimes confuse Masked Autoencoders with JEPA because both rely on masking to train the model. But the contrast is entirely in where the prediction happens. While masked autoencoders predict in pixel space, JEPA predicts missing information in a learned representation space. In a JEPA, the network never outputs an actual image or video frame. Instead, a component called a target encoder processes the unmasked ground-truth data and produces abstract targets. Because this target encoder operates strictly in representation space, unnecessary pixel-level details are potentially eliminated. The model is trained simply to match these abstract representations. By predicting representations rather than pixels, the network is no longer burdened with rendering unpredictable background noise. Returning to our car on the tree-lined street, the target encoder can effectively ignore the flickering leaves. This allows the model to focus on the predictable aspects of the scene, such as the car's trajectory. It learns the structure of the world without being forced to paint it. The key takeaway is that predicting raw pixels forces a model to spend its budget on chaotic, irrelevant details. Moving the prediction to an abstract representation space allows the network to capture the semantic reality of a scene. If you are enjoying this series and want to support the show, you can find us by searching for DevStoriesEU on Patreon. That is all for this one. Thanks for listening, and keep building!

Hi, this is Alex from DEV STORIES DOT EU. Meta JEPA: Non-Generative World Models, episode 2 of 4. Many self-supervised vision models rely on aggressively cropping and color-jittering images to force a neural network to learn what matters. But what if a model could build a deep understanding of the visual world just by playing peek-a-boo with its own internal representations? That approach is the focus of I-JEPA: Learning Without Pixel Decoding. The full name is the Image-based Joint-Embedding Predictive Architecture. Going forward, we will just say I-JEPA. To understand why this design exists, look at standard masked image modeling. Usually, you hide patches of an image and ask the model to reconstruct the exact missing pixels. This pushes the network to spend capacity on high-frequency noise and fine details, like the exact texture of grass or the pattern of shadows. That level of detail is irrelevant for actually understanding what is in the scene. I-JEPA solves this by abandoning pixel reconstruction entirely. Instead, it predicts missing information purely in latent space. The architecture relies on three distinct neural networks working together: a context encoder, a target encoder, and a predictor. All three are built using Vision Transformers. The process starts by splitting an image into patches. The system selects one large block of patches to serve as the context, and a few other separate blocks from the same image to serve as the targets. The context encoder processes only the context block, after any patches that overlap the targets have been removed from it. It outputs a latent representation of what it can see. Meanwhile, the target encoder processes the whole, unmasked image, and the target blocks are then picked out of its output. The paper stresses that the masking is applied to the output of the target encoder, not to its input, and calls that distinction crucial for getting targets with a high semantic level. Do not confuse this target encoder with a decoder. A decoder would take a latent representation and try to paint the original image. The target encoder does the opposite: it turns real pixels into the representations the predictor has to match. This is where the predictor comes in. The predictor takes the latent representation from the context encoder, plus a positional token indicating exactly where the missing target block is located in the original image. Its job is to output a vector that matches what the target encoder produced. Picture an image of a dog. You feed the context encoder a block showing the dog's body. The target encoder sees the whole image, and the part of its output that covers the dog's head becomes the target. The predictor takes the encoded body, alongside the position of the head, and outputs a latent vector. The system calculates the loss by comparing the predictor's output directly to that target. The network learns to predict the representation of the dog's head, completely skipping the busywork of generating actual pixels for fur and whiskers. There is a mechanical trap here. If both encoders update freely based on the loss function, the network will quickly find a shortcut. It will suffer from collapse, meaning the encoders learn to produce a constant output whatever the input is. If every image patch encodes to the exact same vector, the predictor always guesses correctly, and the loss drops to zero without the model learning anything useful. I-JEPA avoids collapse through an asymmetric design. The target encoder does not learn through direct gradient descent. Instead, it is an exponential moving average of the context encoder. You can think of it as a slowly updated copy of the context encoder. Because the target encoder weights trail behind, it provides a stable, evolving target for the predictor to hit, keeping the training process honest. Predicting missing information in latent space lets a network filter out irrelevant visual noise and learn the semantic structure of a scene, instead of memorizing its pixels. That is all for this one. Thanks for listening, and keep building!

Hi, this is Alex from DEV STORIES DOT EU. Meta JEPA: Non-Generative World Models, episode 3 of 4. Can a model learn how things move just by watching video, with no labels, and without ever drawing a single pixel? This episode covers V-JEPA: Masked Prediction Across Space and Time. V-JEPA extends the Joint-Embedding Predictive Architecture to video. Instead of predicting the exact next frame from the frames before it, V-JEPA looks at a whole video clip and predicts the parts of it that are hidden. It does this entirely in representation space. The model uses no language, no text pairings, no negative examples, and zero human annotations to guide its learning. It relies completely on the physical structure of the video itself. Take a short video of someone throwing a ball. The model treats this video as a three-dimensional block of data, spanning spatial width, spatial height, and time. It cuts this spatiotemporal volume into small patches. Then it masks most of them out. V-JEPA does not hide the middle frames of the clip. If it did, the model could copy the hidden content from the neighboring frames. Instead, it samples large spatial blocks and hides the same region in every frame, from the first to the last. On average, about ninety percent of the video is masked. In our example, a large part of the yard is hidden for the whole clip, and the flight of the ball passes through that hidden region. Three neural networks handle this setup: the context encoder, the target encoder, and the predictor. The context encoder only looks at the visible, unmasked patches. It creates an abstract representation of what it can currently see. The target encoder looks at the full, original, unmasked video to generate the target representations for the hidden patches. Finally, the predictor takes the context encoder's output and tries to predict those exact targets. A common problem with embedding models is representation collapse, where the network simply outputs a constant vector for everything to perfectly match its own predictions with zero effort. The hard masking task alone does not prevent this. Collapse is prevented by the target encoder's design. The target encoder does not update via gradient descent. Instead, its weights are an exponential moving average of the context encoder's weights. A stop-gradient ensures no learning signal flows backward through the target encoder. This asymmetric design forces the predictor to do actual work and stops the model from collapsing into a trivial solution. This entire process forms a mask-denoising objective operating across time and space. The model must predict the latent representation of the masked patches based on the surrounding spatiotemporal context. Returning to the thrown ball, the predictor has to output the representation of the hidden region the ball flies through. It does not generate an image of the ball against the sky. It outputs abstract features of that part of the scene. When researchers decode these abstract predictions back into visual space to verify what the model learned, the decoded predictions show motion that stays consistent over time. Furthermore, some samples suggest a form of object permanence: objects stay consistent after being partly hidden. The core takeaway is that forcing a model to predict missing spatiotemporal representations forces it to capture the structure of a scene without generating a single pixel. That is all for this one. Thanks for listening, and keep building!

Hi, this is Alex from DEV STORIES DOT EU. Meta JEPA: Non-Generative World Models, episode 4 of 4. What if a robot could learn to manipulate objects in a room it has never seen, just by imagining the future in its latent space? The answer lies in V-JEPA 2-AC: Planning Robot Actions in Latent Space. V-JEPA 2 scales the video recipe to more than a million hours of internet video and an encoder of about one billion parameters. Thanks to this scale, the pretrained encoder learns video representations that support understanding and predicting the physical world. But a frozen video encoder alone does not know how to move a robot arm. That requires an action-conditioned predictor. The original V-JEPA mask predictor simply fills in missing spatial patches of a video. The action-conditioned predictor does something completely different. It is a roughly 300-million-parameter transformer trained on about 62 hours of unlabeled robot video from the Droid dataset. It acts as a time-causal world model. It predicts the next frame's representation from the current and previous frames, the arm's state, and the action. Here, an action is computed as the change in the arm's position, orientation, and gripper between frames. Consider a Franka robot arm tasked with picking up a cup and moving it. For a simple grasping task, the model is given a single goal image. But because planning over long horizons is still a limitation, for pick-and-place tasks like moving the cup, the researchers gave it two intermediate goal images plus the final goal image. The robot must figure out the sequence of physical actions required to match those visual goals zero-shot, without any task-specific training or reward. It solves this using Model Predictive Control combined with energy minimization. To find the right actions, the system uses the cross-entropy method. It samples many candidate action sequences and passes them to the predictor. The predictor imagines the outcomes in latent space. The system then measures the energy, which is the mathematical distance between the predicted representations and the goal image representations. The cross-entropy method keeps the best action sequences that minimize this distance and refines its sampling around them over several rounds. Round by round, it narrows down the most promising sequences of actions. Once a good sequence is identified, the robot executes the first action. Then it stops and runs the whole process again. Re-planning after each executed action lets it correct its course and adjust for small physical variations in the real world. Deployed zero-shot on Franka arms in two labs absent from its training data, pick-and-place with a cup succeeded in about 80 percent of trials, and grasping a box about a quarter of the time. This planning is not instant. Planning with the cross-entropy method on one RTX 4090 GPU takes about 16 seconds per action versus about 4 minutes for the Cosmos video-generation model, so it is not real-time. Even at 16 seconds per action, it shows that a world model which predicts only representations, never pixels, can plan for a real robot zero-shot. I encourage you to check out the official V-JEPA 2 repository to look at the post-training code and try these concepts hands-on. That is all for this one. Thanks for listening, and keep building!