Robots Will Be Trained on Video Games - Pim de Witte, General Intuition

Highlights

• Metal operates as a video game recorder that collects action data across tens of thousands of video games.

• Metal captures an estimated three billion videos per year, yielding over a trillion and a half action tokens per month.

• General Intuition builds vision-based models that take video frames and text prompts as input and output controller or keyboard actions.

• Video game action spaces align closely with robot teleoperation spaces because multiple robots ship with standard game controllers.

• General Intuition models control over ten different types of robot embodiments alongside video games using identical primitives.

Timeline

1. 00:12 - 01:55

   Parallels Between Video Games and Robotics

   Standard language models predict text tokens in a turn-based manner, which limits their effectiveness in spatial and temporal environments. In contrast, continuous observation allows systems to react dynamically while acting, much like animated characters or biological entities.

   • Intelligence architectures require continuous observation during action execution rather than purely auto-regressive token prediction.
   • Purpose-built models provide a better fit for spatial and temporal environments than standard language models.
2. 01:55 - 06:45

   Scaling Pixel Space via Video Game Data

   Human operators use identical mental frameworks and physical controllers for video games and robot teleoperation. By associating video frames with labeled game inputs and controller actions, vision models learn high-density action predictions without requiring direct motor torque calculations.

   • Metal records gameplay footage from users playing tens of thousands of distinct video games.
   • Video game controllers match the action spaces used in robot teleoperations.
3. 06:45 - 13:35

   Model Capabilities and World Models

   Scaling pixel space data enables the creation of general agents capable of solving tasks across computers and physical hardware. Operating in the native modality of human vision allows faster problem solving and high action-per-minute execution for complex tasks like drone piloting.

   • Metal collects data volumes equivalent to current robotics frontier labs' entire training sets every half day.
   • World models generate simulation environments for agents to explore and learn without relying on expensive digital twins.
4. 13:35 - 18:43

   Releases, Accessibility, and Hiring

   The models integrate with orchestrator systems like Gemini for long-term planning tasks. Future updates and upcoming hackathons provide developers hands-on access to test models across world modeling, policy execution, and environment generation teams.

   • Early access rollouts and hackathons are scheduled in New York City and London.
   • Computer-use capabilities assist users with physical accessibility needs by predicting keyboard actions based on mouse input.