Coding agents
Each agent writes the controller for a whole-body humanoid from one demonstration video, with the same 101,000 environment steps per session. Mean success over 9 tasks × 3 independent sessions, 100 hidden episodes per final program.
Every agent had the same 101,000 environment steps per session; tokens and dollars are what each spent on its own.
Pareto frontier; the shaded area is beaten on both axes. Click two points to compare them
Each point is the mean of 3 sessions (100 hidden seeds each); All tasks is the mean over every session. The step line is the Pareto frontier: no agent both spends less and succeeds more than one on it, and the shaded area is beaten on both; with 3 sessions per task, near ties are within noise. Harnesses: Claude Code 2.1.280; Codex CLI 0.153.4 for GPT-6 Astra, 0.157.1 for GPT-6 Sol, 0.160.0 for GPT-6.1 Sol and 0.164.0 for GPT-6.1 Terra, each newer model being served only to a newer client; DeepSeek's own CLI (dsh 0.2.1-rc.2) for DeepSeek V4.1 Flash (V4.1-Flash-1017, max effort), priced at DeepSeek's peak rate, twice its off-peak rate; Claude Code 2.1.280 against Moonshot's endpoint for Kimi K3.1 Pro (kimi-1021-preview, max effort), priced at Kimi K3.1 Pro's list rate, rerun sessions in place of the four that timed out. Output tokens are everything the model generated, reasoning included; every harness reports them per request.
All 189 sessions
Success of each session and their mean (%), then API-equivalent cost ($) and output tokens per session (reasoning included). Last row: mean over all sessions, total cost and total output tokens.
| Task | Claude Opus 5.5 (high) | GPT-6 Astra (high) | GPT-6 Sol (high) | GPT-6.1 Sol (high) | GPT-6.1 Terra (high) | DeepSeek V4.1 Flash (1017) (max) | Kimi K3.1 Pro (max) | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sessions | Mean | $ | Tokens | Sessions | Mean | $ | Tokens | Sessions | Mean | $ | Tokens | Sessions | Mean | $ | Tokens | Sessions | Mean | $ | Tokens | Sessions | Mean | $ | Tokens | Sessions | Mean | $ | Tokens | |
| Dishwasher close | 0 / 0 / 100 | 33 | 20.3 | 246k | 100 / 100 / 100 | 100 | 11.7 | 28k | 0 / 0 / 0 | 0 | 6.9 | 88k | 0 / 0 / 0 | 0 | 2.5 | 78k | 0 / 0 / 0 | 0 | 0.9 | 31k | 0 / 0 / 0 | 0 | 0.7 | 265k | 0 / 0 / 0 | 0 | 16.3 | 149k |
| Dishwasher load cups | 83 / 1 / 1 | 28 | 27.9 | 317k | 18 / 16 / 30 | 21 | 24.4 | 68k | 2 / 0 / 3 | 2 | 10.1 | 93k | 36 / 2 / 15 | 18 | 2.1 | 62k | 0 / 0 / 36 | 12 | 0.7 | 21k | 0 / 0 / 0 | 0 | 0.9 | 321k | 4 / 0 / 0 | 1 | 17.3 | 163k |
| Drawer close | 100 / 100 / 100 | 100 | 1.7 | 29k | 96 / 100 / 100 | 99 | 1.6 | 5k | 98 / 100 / 100 | 99 | 1.2 | 22k | 100 / 100 / 100 | 100 | 0.3 | 7k | 70 / 88 / 81 | 80 | 0.3 | 9k | 99 / 91 / 99 | 96 | 0.4 | 180k | 100 / 22 / 47 | 56 | 2.8 | 56k |
| Drawer open | 99 / 98 / 100 | 99 | 11.0 | 143k | 100 / 97 / 100 | 99 | 6.6 | 22k | 55 / 66 / 36 | 52 | 4.7 | 57k | 40 / 100 / 34 | 58 | 0.9 | 24k | 100 / 0 / 27 | 42 | 0.6 | 21k | 0 / 0 / 0 | 0 | 0.6 | 258k | 0 / 84 / 16 | 33 | 14.4 | 147k |
| Move plate | 21 / 15 / 23 | 20 | 27.8 | 273k | 36 / 2 / 4 | 14 | 18.8 | 61k | 1 / 7 / 5 | 4 | 4.2 | 56k | 7 / 0 / 17 | 8 | 0.9 | 31k | 3 / 0 / 1 | 1 | 0.6 | 24k | 0 / 0 / 0 | 0 | 0.9 | 292k | 0 / 0 / 0 | 0 | 9.8 | 115k |
| Move two plates | 8 / 15 / 22 | 15 | 23.0 | 313k | 0 / 38 / 9 | 16 | 12.3 | 45k | 0 / 0 / 0 | 0 | 6.2 | 65k | 0 / 8 / 0 | 3 | 2.0 | 63k | 0 / 0 / 5 | 2 | 0.9 | 26k | 0 / 0 / 0 | 0 | 0.7 | 236k | 0 / 0 / 0 | 0 | 17.1 | 128k |
| Pick box | 44 / 99 / 71 | 71 | 10.6 | 149k | 64 / 85 / 12 | 54 | 12.5 | 34k | 0 / 0 / 0 | 0 | 5.9 | 73k | 0 / 52 / 0 | 17 | 1.5 | 40k | 3 / 8 / 0 | 4 | 0.9 | 25k | 0 / 0 / 0 | 0 | 0.5 | 204k | 0 / 0 / 0 | 0 | 12.9 | 138k |
| Reach dual | 71 / 99 / 65 | 78 | 10.2 | 159k | 100 / 93 / 75 | 89 | 6.0 | 24k | 33 / 58 / 20 | 37 | 3.8 | 62k | 82 / 61 / 66 | 70 | 0.8 | 30k | 67 / 34 / 34 | 45 | 0.6 | 26k | 0 / 20 / 35 | 18 | 0.6 | 299k | 0 / 7 / 48 | 18 | 12.3 | 143k |
| Reach multi-modal | 100 / 100 / 100 | 100 | 7.7 | 102k | 75 / 97 / 95 | 89 | 4.6 | 20k | 73 / 86 / 99 | 86 | 1.2 | 29k | 81 / 97 / 100 | 93 | 0.9 | 30k | 86 / 84 / 46 | 72 | 0.5 | 20k | 0 / 10 / 1 | 4 | 0.5 | 228k | 100 / 59 / 87 | 82 | 6.3 | 96k |
| All 9 tasks | 27 sessions | 60.6 | 420 | 5.2M | 27 sessions | 64.5 | 296 | 921k | 27 sessions | 31.2 | 133 | 1.6M | 27 sessions | 40.7 | 36 | 1.1M | 27 sessions | 28.6 | 18 | 610k | 27 sessions | 13.1 | 17 | 6.8M | 27 sessions | 21.3 | 328 | 3.4M |
Learned policies and coding agents
Success rate on the same 9 tasks and hidden seeds, 100 evaluation episodes per checkpoint or program.
| VLA | Imitation learningIL | Demo-driven RLRL | Coding agentAgent | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Task | π0.5 | ACT | DP | CQN-AS | DrQ-v2+ | DEAS | Opus 5.5OpusstrictClaude Opus 5.5Claude Codev2.1.280 | GPT-6 AstraAstrastrictGPT-6 AstraCodex CLIv0.153.4 | DeepSeek V4.1 Flash (1017)DeepSeekstrictDeepSeek V4.1 Flash (1017)DeepSeek CLI (dsh)v0.2.1-rc.2 | Kimi K3.1 ProKimistrictKimi K3.1 ProClaude Codev2.1.280 |
| Pick box | 19 | 61±4 | 10±3 | 48±2 | 0±0 | 14±3 | 71±16 | 54±22 | 0±0 | 0±0 |
| Reach multi-modal | 90 | 81±1 | 84±1 | 19±16 | 0±0 | 79±2 | 100±0 | 89±7 | 4±3 | 82±12 |
| Reach dual | 58 | 30±0 | 27±0 | 3±1 | 0±0 | 34±1 | 78±10 | 89±7 | 18±10 | 18±15 |
| Drawer close | 100 | 100±0 | 100±0 | 100±0 | 99±0 | 100±0 | 100±0 | 99±1 | 96±3 | 56±23 |
| Drawer open | 99 | 92±0 | 98±0 | 89±2 | 0±0 | 79±4 | 99±1 | 99±1 | 0±0 | 33±26 |
| Move plate | 65 | 60±1 | 62±4 | 48±3 | 0±0 | 27±3 | 20±2 | 14±11 | 0±0 | 0±0 |
| Move two plates | 68 | 39±0 | 18±4 | 21±3 | 0±0 | 2±1 | 15±4 | 16±11 | 0±0 | 0±0 |
| Dishwasher close | 79 | 93±5 | 51±2 | 78±1 | 0±0 | 27±3 | 33±33 | 100±0 | 0±0 | 0±0 |
| Dishwasher load cups | 92 | 56±3 | 51±0 | 47±7 | 0±0 | 6±2 | 28±27 | 21±4 | 0±0 | 1±1 |
| Mean | 75 | 68 | 56 | 50 | 11 | 41 | 61 | 65 | 13 | 21 |
Success rate (%) on nine representative tasks, in the current scene lit by one light that follows the robot. The best result in each row is bold.
How each number is measured
ACT, DP, CQN-AS, DrQ-v2+ and DEAS: mean ± SE across three training runs, each averaging its last five checkpoints at 100 episodes. π0.5: one training run per task, pooled over its last five checkpoints at 50 episodes, so no SE. Coding agents: three independent development sessions each under the strict interface (84×84 onboard views, direct joint actions, no kinematics tools), 100 episodes per frozen program, mean ± SE across sessions. Kimi K3.1 Pro (kimi-1021-preview, max effort) runs in Claude Code against Moonshot's endpoint; four of its 27 sessions failed on API timeouts before submitting and were rerun.