Anthropic Research published findings on how language models, including Claude, perform in robotics tasks. The research explored whether the strengths of language models transfer to robotics, a domain requiring the synthesis of logical skills and precise 3D understanding.
Key Points
- Anthropic tested several language models, including Claude Opus 4.6 and Opus 4.5, on their ability to control a range of robot bodies.
- Robot bodies included classic control toys, a simulated quadruped and humanoid, a a robotic arm, and a real Unitree Go2 quadruped robot.
- Models were evaluated using four control interfaces: direct control (motor torques), programmatic control (writing Python code), policy control (issuing high-level commands to a pretrained policy), and reinforcement learning supervision (training a policy).
- Performance varied significantly based on the control method; models largely failed when directly driving joints but succeeded in navigation and manipulation tasks when supervising a pretrained controller or using simple orientation tools.
- Tests covered classic control problems (e.g., balancing a pendulum), locomotion and navigation, and manipulation (e.g., grasping objects).
- Claude Opus 4.6 and Opus 4.5 demonstrated generational improvement across classic control benchmarks, particularly in code control and reinforcement learning.
- The improvements are attributed to a better ability to adapt strategies based on prior outcomes.
Context
According to Anthropic Research, the study aimed to determine if language models could perceive a scene, understand a robot's state, and issue actions to effect physical change. The research utilized the Mujoco simulation engine for evaluations, pausing the simulator between LLM calls to isolate model capability from real-time latency issues. To ensure generalizability, the evaluation included a novel task, TwinFlipper, designed to be a chaotic, dynamic system not present in pretraining corpora.
Why It Matters
This research provides insights into the current capabilities and limitations of language models in robotics. It highlights that while frontier models cannot yet control humanoid robots without a pretrained policy, newer models show substantial gains in direct manipulation and high-level policy control. This suggests a closing gap in reliability for general-purpose chat models to perform complex robotic tasks, impacting the future development and deployment of AI in physical systems.