BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation

TL;DRBiGym 2.0 is a household loco-manipulation benchmark: 20 tasks on the Unitree G1, with 60 human VR demonstrations each. From one demonstration, GPT-6 Astra and Claude Opus 5.5 write controllers that beat every demo-driven RL baseline on the nine-task mean and rival imitation learning, but learned policies still lead on precise object rearrangement.

Human demonstrations

One from each of the 20 tasks, recorded in VR through the whole-body controller the policies are scored with.

**BiGym → BiGym 2.0.** The H1's legs were replayed under a commanded pelvis; the G1's are driven by a frozen GR00T-WBC controller, the same one every demonstration below was recorded through.

Move plateMove the plate between two draining racks.
Move two platesMove two plates simultaneously from one draining rack to the other.
Flip cupFlip the upside-down cup to an upright position.
Flip cutleryTake the cutlery from the holder, flip it, and place it back.
4×
Stack blocksMove blocks across the table and stack them in the target area.
2.5×
Dishwasher closePush back all trays and close the door of the dishwasher.
Dishwasher load cupsMove the cups from the table into the dishwasher's upper tray.
1.5×
Dishwasher load cutleryMove the cutlery from the holder on the table into the dishwasher's cutlery basket.
1.5×
Dishwasher load platesMove the plates from the rack into the dishwasher's lower tray.
Drawer top openOpen the top drawer of the kitchen cabinet.
Drawer top closeClose the top drawer of the kitchen cabinet.
2×
Pick boxPick up the box from the side table and place it on the counter.
1.5×
Put cupsPick up the cups from the table and put them into the closed wall cabinet.
2×
Saucepan to hobTake the saucepan from the closed cabinet and place it on the hob.
1.5×
Sandwich removeRemove the sandwich from the frying pan.
Wall cupboard openOpen the doors of the wall cabinet.
Wall cupboard closeClose the doors of the wall cabinet.
Reach target singleReach the target with the left wrist.
Reach target multi modalReach the target with either wrist.
Reach target dualReach the two targets, one with each wrist.

Learned policies and coding agents, scored on the same seeds

Success rate on 9 representative tasks, 100 hidden evaluation episodes per checkpoint or program. An episode counts only once the goal state has held for one second.

Success rate (%) on nine representative tasks, in the current scene lit by one light that follows the robot. The best result in each row is bold. ACT, DP, CQN-AS, DrQ-v2+ and DEAS: mean ± SE across three training runs, each averaging its last five checkpoints at 100 episodes. π0.5: one training run per task, pooled over its last five checkpoints at 50 episodes, so no SE. Coding agents: three independent development sessions each under the strict interface (84×84 onboard views, direct joint actions, no kinematics tools), 100 episodes per frozen program, mean ± SE across sessions.
Task	π0.5	ACT	DP	CQN-AS	DrQ-v2+	DEAS	Opus 5.5Opusstrict	GPT-6 AstraAstrastrict
	VLA	Imitation learningIL		Demo-driven RLRL			Coding agentAgent
Pick box	19	61±4	10±3	48±2	0±0	14±3	71±16	54±22
Reach multi-modal	90	81±1	84±1	19±16	0±0	79±2	100±0	89±7
Reach dual	58	30±0	27±0	3±1	0±0	34±1	78±10	89±7
Drawer close	100	100±0	100±0	100±0	99±0	100±0	100±0	99±1
Drawer open	99	92±0	98±0	89±2	0±0	79±4	99±1	99±1
Move plate	65	60±1	62±4	48±3	0±0	27±3	20±2	14±11
Move two plates	68	39±0	18±4	21±3	0±0	2±1	15±4	16±11
Dishwasher close	79	93±5	51±2	78±1	0±0	27±3	33±33	100±0
Dishwasher load cups	92	56±3	51±0	47±7	0±0	6±2	28±27	21±4
Mean	75	68	56	50	11	41	61	65

Success rate (%) on nine representative tasks, in the current scene lit by one light that follows the robot. The best result in each row is bold.

How each number is measured

ACT, DP, CQN-AS, DrQ-v2+ and DEAS: mean ± SE across three training runs, each averaging its last five checkpoints at 100 episodes. π0.5: one training run per task, pooled over its last five checkpoints at 50 episodes, so no SE. Coding agents: three independent development sessions each under the strict interface (84×84 onboard views, direct joint actions, no kinematics tools), 100 episodes per frozen program, mean ± SE across sessions.

A program written from one demonstration

A cold-start coding agent gets a learned policy’s inputs and the online learners’ 101,000-step budget, with no IK, no calibration, no external camera, and writes the controller itself.

Every program, every seed

Two ways to touch a sphere: Astra works from pixels alone; Opus 5.5 recalls the G1’s link offsets from Unitree’s public description and solves inverse kinematics with them.

GPT-6 Astrasession 2 · 71 lines · solves 97 of 100 hidden seeds

Servoes in the image: it finds the red pixels in a wrist camera and nudges the arm toward them. No model of the camera or the arm.

Finds the target by colour13

m=(z[:,:,0]>65)&(z[:,:,0]>z[:,:,1]*1.8)&(z[:,:,0]>z[:,:,2]*1.8)

Moves the shoulder in proportion to the pixel offset
54
```
self.a[i]+=np.clip((y-42)*.0012,-.03,.03)
```
Extends the elbow once the target is centred
63
```
self.a[i+3]+=.023
```

Claude Opus 5.5session 3 · 416 lines · solves 100 of 100 hidden seeds

Rebuilds the tools the strict interface withholds: a camera model, the arm’s kinematics recalled from Unitree’s published description, and an IK solver.

Models the head camera it was never given21

F = 72.746          # focal length in px (fovy 60 deg, 84 px)

Recalls the G1 shoulder offset from Unitree’s URDF120

p = TORSO + np.array([0.0039563, s * 0.10022, 0.23778])

Solves inverse kinematics numerically170

r = least_squares(res, s0, bounds=(lo, hi))

Success on 100 hidden seeds of each coding-agent session; choose a cell to open its program.
Session	Pick box	Reach multi-modal	Reach dual	Drawer close	Drawer open	Move plate	Move two plates	Dishwasher close	Dishwasher load cups
Opus 5.5session 1
session 2
session 3
Astrasession 1
session 2
session 3

Success on 100 hidden seeds of each coding-agent session; choose a cell to open its program.
Task	s1	s1
	Opus 5.5	Astra
Pick box
Reach multi-modal
Reach dual
Drawer close
Drawer open
Move plate
Move two plates
Dishwasher close
Dishwasher load cups

Success (%) on the 100 hidden seeds; darker is higher. Pick a cell to read its program.

Reach multi-modal · Opus 5.5 · session 3

policy.py416 lines · solves inverse kinematicsRaw

"""Touch the red sphere with either hand (Unitree G1, scripted).

Pipeline
  1. Locate the sphere in the head camera: fit a circle (in ray space) to the
     red blob boundary -> direction + angular radius -> 3D point, using a
     pinhole model of the head camera calibrated from the floor checkerboard.
  2. Store the target in world coordinates (pelvis odometry) and walk the base
     so the target sits at a comfortable spot in front of one shoulder.
  3. Re-observe, then reach with an analytic G1 arm model + numerical IK:
     sweep the hand forward through the estimate; the sphere lights up while
     touched, so the lit stretch of the sweep gives the touch-zone centre
     (then z / y sweeps if needed), and the hand holds there.
The arms hang down (out of the head camera view) until the reach starts.
"""

import cv2
import numpy as np
from scipy.optimize import least_squares

# ---------------------------------------------------------------- camera model
F = 72.746          # focal length in px (fovy 60 deg, 84 px)
CX = CY = 41.5
CAM_PITCH = 1.093   # rad below horizontal
CAM_OFF = np.array([0.07, 0.0, 0.40])   # head camera position in pelvis frame
R_SPHERE = 0.047    # effective sphere radius for the boundary detector


def cam_R(th, psi=0.0):
    """Camera->frame rotation (camera x right, y down, z forward)."""
    z = np.array([np.cos(th) * np.cos(psi), np.cos(th) * np.sin(psi), -np.sin(th)])
    x = np.array([np.sin(psi), -np.cos(psi), 0.0])
    y = np.cross(z, x)
    return np.stack([x, y, z], 1)


R_CAM = cam_R(CAM_PITCH)


def red_mask(im):
    im = im.astype(np.int32)
    r, g, b = im[..., 0], im[..., 1], im[..., 2]
    return (r > 40) & (r > 2 * g + 8) & (r > 2 * b + 8)


def bright_count(im):
    im = im.astype(np.int32)
    r, g, b = im[..., 0], im[..., 1], im[..., 2]
    return int(((r > 215) & (g < 110) & (b < 110)).sum())


def fit_sphere(im):
    """Fit the red blob. Returns (unit ray in camera frame, angular radius, n boundary pts)."""
    m = red_mask(im)
    if m.sum() < 4:
        return None
    n, lab, stats, _ = cv2.connectedComponentsWithStats(m.astype(np.uint8), connectivity=8)
    k = 1 + int(np.argmax(stats[1:, cv2.CC_STAT_AREA]))
    mk = (lab == k).astype(np.uint8)
    cs, _ = cv2.findContours(mk, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
    pts = np.concatenate([c.reshape(-1, 2) for c in cs], 0).astype(float)
    h, w = m.shape
    keep = (pts[:, 0] > 0) & (pts[:, 0] < w - 1) & (pts[:, 1] > 0) & (pts[:, 1] < h - 1)
    pts = pts[keep]
    if len(pts) < 6:
        return None
    ys, xs = np.nonzero(mk)
    rays = np.stack([(pts[:, 0] - CX) / F, (pts[:, 1] - CY) / F, np.ones(len(pts))], 1)
    rays /= np.linalg.norm(rays, axis=1, keepdims=True)

    def dirf(p):
        d = np.array([p[0], p[1], 1.0])
        return d / np.linalg.norm(d)

    x0 = [(xs.mean() - CX) / F, (ys.mean() - CY) / F, np.sqrt(mk.sum() / np.pi) / F]

    def res(p):
        return np.arccos(np.clip(rays @ dirf(p[:2]), -1, 1)) - p[2]

    r = least_squares(res, x0)
    return dirf(r.x[:2]), r.x[2] + 0.5 / F, len(pts)


def locate(im):
    """Sphere centre in the pelvis frame (x fwd, y left, z up rel. pelvis)."""
    s = fit_sphere(im)
    if s is None:
        return None
    d, alpha, npts = s
    if alpha <= 0.01:
        return None
    D = R_SPHERE / np.sin(alpha)
    return R_CAM @ (D * d) + CAM_OFF, npts


# ------------------------------------------------------------------ arm model
def rx(a):
    c, s = np.cos(a), np.sin(a)
    return np.array([[1, 0, 0], [0, c, -s], [0, s, c]])


def ry(a):
    c, s = np.cos(a), np.sin(a)
    return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])


def rz(a):
    c, s = np.cos(a), np.sin(a)
    return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1]])


TORSO = np.array([-0.0039635, 0, 0.054])
Q16 = 2 * np.arcsin(0.139201)
TOOL = np.array([0.10, 0.0, 0.0])


def fk(q, side):
    """Tool point and wrist rotation (pelvis frame) for arm joints q (7)."""
    s = side
    R = np.eye(3)
    p = TORSO + np.array([0.0039563, s * 0.10022, 0.23778])
    R = rx(s * Q16) @ ry(q[0])
    p = p + R @ np.array([0, s * 0.038, -0.013831])
    R = R @ rx(-s * Q16) @ rx(q[1])
    p = p + R @ np.array([0, s * 0.00624, -0.1032])
    R = R @ rz(q[2])
    p = p + R @ np.array([0.015783, 0, -0.080518])
    R = R @ ry(q[3])
    p = p + R @ np.array([0.1, s * 0.00188791, -0.01])
    R = R @ rx(q[4])
    p = p + R @ np.array([0.038, 0, 0])
    R = R @ ry(q[5])
    p = p + R @ np.array([0.046, 0, 0])
    R = R @ rz(q[6])
    return p + R @ TOOL, R


IDX = [0, 1, 2, 3, 5]   # joints used by the IK: shoulder pitch/roll/yaw, elbow, wrist pitch


# sane working ranges (left arm) for sp, sr, sy, el, wp; roll/yaw mirrored for the right arm
W_LO = np.array([-1.9, 0.0, -0.9, -0.9, -1.0])
W_HI = np.array([0.5, 0.9, 0.9, 1.6, 1.0])


def limits(side):
    lo, hi = W_LO.copy(), W_HI.copy()
    if side < 0:
        for i in (1, 2):
            lo[i], hi[i] = -W_HI[i], -W_LO[i]
    return lo, hi


def ik(p, side, q0=None, approach=np.array([1.0, 0, 0]), w_ori=0.05):
    lo, hi = limits(side)
    if q0 is None:
        starts = [np.clip(np.array(s, float) * np.array([1, side, side, 1, 1]), lo + 1e-3, hi - 1e-3)
                  for s in [(-0.3, 0.1, 0, -0.5, 0), (-0.8, 0.1, 0, -0.3, 0), (-0.5, 0.2, 0, 0.3, 0),
                            (-1.2, 0.1, 0, 0.3, 0), (0, 0, 0, 0, 0)]]
    else:
        starts = [np.clip(np.asarray(q0)[IDX], lo + 1e-3, hi - 1e-3)]

    def res(x):
        q = np.zeros(7)
        q[IDX] = x
        tip, R = fk(q, side)
        return np.r_[(tip - p) * 30, w_ori * (R[:, 0] - approach), 0.003 * x]

    best = None
    for s0 in starts:
        r = least_squares(res, s0, bounds=(lo, hi))
        if best is None or r.cost < best.cost:
            best = r
    q = np.zeros(7)
    q[IDX] = best.x
    return q, float(np.linalg.norm(fk(q, side)[0] - p))


# --------------------------------------------------------------------- policy
def to_world(p_rel, pel):
    x, y, z, yaw = pel
    c, s = np.cos(yaw), np.sin(yaw)
    return np.array([x + c * p_rel[0] - s * p_rel[1], y + s * p_rel[0] + c * p_rel[1], z + p_rel[2]])


def to_rel(p_w, pel):
    x, y, z, yaw = pel
    c, s = np.cos(yaw), np.sin(yaw)
    dx, dy = p_w[0] - x, p_w[1] - y
    return np.array([c * dx + s * dy, -s * dx + c * dy, p_w[2] - z])


X_DES = 0.32      # desired target x in pelvis frame before reaching
Y_DES = 0.12      # desired |y| of target (in front of the reaching shoulder)
SX0, SX1 = -0.06, 0.08   # x sweep range (relative to the estimate)
SW = 0.05                # half range of the lateral (z, y) sweeps
S_SPEED = 0.0015         # sweep speed, m per control step
V_MOVE = 0.003           # repositioning speed, m per control step
KI = 0.05                # joint-space integral gain (gravity sag)
# x-sweep lines (dy, dz) tried in turn around the estimate
OFFSETS = [(0, 0), (0, 0.025), (0, -0.025), (0.025, 0), (-0.025, 0), (0.025, 0.025), (-0.025, 0.025),
           (0.025, -0.025), (-0.025, -0.025), (0, 0.05), (0, -0.05), (0.05, 0), (-0.05, 0)]
AXES = {"x": 0, "y": 1, "z": 2}
BASE0 = np.array([0.01, 0.0, 0.02])   # systematic offset of the touch zone from the estimate
# arms hang down while observing/walking so the grippers never hide the sphere
CLEAR_L = np.array([0.0, 0.15, 0.0, 1.3, 0.0, 0.0, 0.0])
CLEAR_R = np.array([0.0, -0.15, 0.0, 1.3, 0.0, 0.0, 0.0])
T_CLEAR = 30        # steps for the arms to get out of the view
SETTLE = 70         # steps standing still after the walk
SETTLE_OBS = 40     # observe from this step of the settle phase on


class Policy:
    """The control policy that is scored on the hidden seeds."""

    def reset(self, obs, tools):
        self.hold = np.asarray(tools.hold_action(), dtype=np.float64)
        self.phase = "observe"
        self.t0 = 0
        self.obs_buf = []
        self.target_w = None
        self.side = 1
        self.q = np.zeros(7)
        self.qi = np.zeros(7)
        self.log = []
        self.line = 0
        self.base = np.zeros(3)      # centre of the current search (offset from estimate)
        self.g = np.zeros(3)         # commanded tip offset from the estimate
        self.mode = None
        self.unlit = 0

    def pel(self, obs):
        return np.asarray(obs["low_dim_obs"][18:22], dtype=np.float64)

    def observe(self, obs, tools):
        im = tools.image("head")
        r = locate(im)
        if r is None:
            return None
        p, npts = r
        return to_world(p, self.pel(obs)), npts

    # ------------------------------------------------------------ search program
    def start_xsweep(self):
        dy, dz = OFFSETS[self.line % len(OFFSETS)]
        c = self.base + np.array([0.0, dy, dz])
        self.go(c + np.array([SX0, 0, 0]), ("sweep", "x", c[0] + SX1))

    def go(self, p, then):
        self.g_to = np.asarray(p, float)
        self.mode = "move"
        self.after = then

    def begin_sweep(self, axis, end):
        self.mode = "sweep"
        self.axis = axis
        self.end = end
        self.end0 = end
        self.lit_vals = []
        self.unlit = 0

    def sweep_done(self):
        ax = AXES[self.axis]
        if self.lit_vals:
            v = np.array(self.lit_vals)
            centre = 0.5 * (v.min() + v.max())
        else:
            centre = None
        self.log.append(("sweep", self.axis, None if centre is None else round(float(centre), 4),
                         len(self.lit_vals), self.base.round(3).tolist()))
        if self.axis == "x":
            if centre is None:
                self.line += 1
                self.start_xsweep()
                return
            dy, dz = OFFSETS[self.line % len(OFFSETS)]
            self.cur = np.array([centre, self.base[1] + dy, self.base[2] + dz])
            self.go(self.cur, ("hold", "x"))
        elif self.axis == "z":
            if centre is not None:
                self.cur[2] = centre
            self.go(self.cur + np.array([0, -SW, 0]), ("sweep", "y", self.cur[1] + SW))
        else:
            if centre is not None:
                self.cur[1] = centre
            self.go(self.cur, ("hold",))

    def step_search(self, lit, tip_off):
        m = self.mode
        if m == "move":
            d = self.g_to - self.g
            n = np.linalg.norm(d)
            if n <= V_MOVE:
                self.g = self.g_to.copy()
                if self.after[0] == "sweep":
                    self.begin_sweep(self.after[1], self.after[2])
                else:
                    self.mode = "hold"
                    self.hold_kind = self.after[1] if len(self.after) > 1 else "final"
                    self.unlit = 0
            else:
                self.g = self.g + d / n * V_MOVE
        elif m == "sweep":
            ax = AXES[self.axis]
            if lit:
                self.lit_vals.append(tip_off[ax])
                self.unlit = 0
            else:
                self.unlit += 1
            if lit and self.g[ax] >= self.end and self.end < self.end0 + 0.04:
                self.end += S_SPEED   # still lit at the end of the range: keep going
            if (self.lit_vals and self.unlit >= 3) or self.g[ax] >= self.end:
                self.sweep_done()
            else:
                self.g[ax] += S_SPEED
        elif m == "hold":
            self.unlit = 0 if lit else self.unlit + 1
            if self.hold_kind == "x" and self.unlit > 20:
                # x centre alone is not enough: centre z, then y
                self.go(self.cur + np.array([0, 0, -SW]), ("sweep", "z", self.cur[2] + SW))
            elif self.unlit > 40:
                # lost it: search again around the last centre
                self.base = self.cur.copy()
                self.line = 0
                self.start_xsweep()

    # ------------------------------------------------------------------- act
    def act(self, obs, tools):
        t = int(obs["t"])
        a = self.hold.copy()
        pel = self.pel(obs)
        low = np.asarray(obs["low_dim_obs"], dtype=np.float64)
        # both arms hang clear; the reaching arm is overwritten in the reach phase
        a[4:11] = CLEAR_L
        a[11:18] = CLEAR_R

        if self.phase == "observe":
            r = self.observe(obs, tools) if t >= T_CLEAR else None
            if r is not None and r[1] >= 20:
                self.obs_buf.append(r[0])
            self.log.append(("obs", t, pel.round(3).tolist(), None if r is None else (r[0].round(3).tolist(), r[1])))
            if len(self.obs_buf) >= 8:
                self.target_w = np.median(np.array(self.obs_buf), 0)
                self.first_w = self.target_w.copy()
                rel = to_rel(self.target_w, pel)
                self.side = 1 if rel[1] >= 0 else -1
                self.phase = "walk"
                self.t0 = t
            elif T_CLEAR + 10 < t < T_CLEAR + 160:
                a[0] = 0.12   # sphere not well visible: creep forward
            return a

        rel = to_rel(self.target_w, pel)

        if self.phase == "walk":
            ex = rel[0] - X_DES
            ey = rel[1] - self.side * Y_DES
            if abs(ex) < 0.02 and abs(ey) < 0.02 or t - self.t0 > 250:
                self.phase = "settle"
                self.t0 = t
                self.obs_buf = []
                self.log.append(("walked", t, pel.round(3).tolist(), rel.round(3).tolist()))
            else:
                vx = np.clip(1.5 * ex, -0.25, 0.25)
                vy = np.clip(1.5 * ey, -0.25, 0.25)
                sp = np.hypot(vx, vy)
                if sp < 0.07:
                    k = 0.07 / max(sp, 1e-6)
                    vx, vy = vx * k, vy * k
                a[0], a[1] = vx, vy
            return a

        if self.phase == "settle":
            if t - self.t0 >= SETTLE_OBS:
                r = self.observe(obs, tools)
                if r is not None and r[1] >= 20:
                    self.obs_buf.append(r[0])
                self.log.append(("settle", t, pel.round(3).tolist(), None if r is None else (r[0].round(3).tolist(), r[1])))
            if t - self.t0 >= SETTLE:
                if len(self.obs_buf) >= 5:
                    est = np.median(np.array(self.obs_buf), 0)
                    if np.linalg.norm(est - self.first_w) < 0.08:
                        self.target_w = est
                self.log.append(("target", t, self.target_w.round(3).tolist(), self.first_w.round(3).tolist()))
                self.phase = "reach"
                self.t0 = t
                self.line = 0
                self.base = BASE0.copy()
                dy, dz = OFFSETS[0]
                self.g = self.base + np.array([SX0, dy, dz])
                self.start_xsweep()
                self.q_from = (CLEAR_L if self.side > 0 else CLEAR_R).copy()
                rel = to_rel(self.target_w, pel)
            else:
                return a

        # ---------------- reach
        side = self.side
        sl = slice(4, 11) if side > 0 else slice(11, 18)
        qm = low[0:7] if side > 0 else low[9:16]
        tip_off = fk(qm, side)[0] - rel
        lit = bright_count(tools.image("head")) >= 3
        tau = t - self.t0
        if tau >= 40:
            self.step_search(lit, tip_off)
        q, err = ik(rel + self.g, side, None if tau == 0 else self.q)
        self.q = q
        if tau < 40:
            w = (tau + 1) / 40.0
            cmd = (1 - w) * self.q_from + w * q
        else:
            self.qi[IDX] = np.clip(self.qi[IDX] + KI * (q - qm)[IDX], -0.15, 0.15)
            cmd = q + self.qi
        a[sl] = cmd
        self.log.append(("reach", t, self.mode, self.line, self.g.round(4).tolist(), tip_off.round(4).tolist(),
                         rel.round(3).tolist(), round(err, 4), int(lit)))
        return a

No recording for this seed yet.

Seed 620000Success279 steps

100/ 100 hidden seeds solved

successfailureclick any seed to watch it in 3D

A score is a point on a compute curve

“As LLMs become more capable, benchmark performance is increasingly a function of test-time compute.”

Noam Brown, Implications of Large-Scale Test-Time Compute, June 2026

Success against compute

AgentSuccessCost per taskInput tokens per task

GPT-6 Astrahigh · Codex64.5%$11.07.7M

Claude Opus 5.5high · Claude Code60.6%$15.634.1M

GPT-6.1 Solhigh · Codex40.7%$1.36.3M

GPT-6 Solhigh · Codex31.2%$4.918.8M

Every agent had the same 101,000 environment steps per session; tokens and dollars are what each spent on its own.

Pareto frontier; the shaded area is beaten on both axes

Each point is the mean of 3 sessions (100 hidden seeds each); All tasks is the mean over every session. The step line is the Pareto frontier: no agent both spends less and succeeds more than one on it, and the shaded area is beaten on both; with 3 sessions per task, near ties are within noise. Harnesses: Claude Code 2.1.280; Codex CLI 0.153.4 for GPT-6 Astra, 0.157.1 for GPT-6 Sol and 0.160.0 for GPT-6.1 Sol; each newer model is served only to a newer client. Input tokens include cache reads (98.5% for Claude Opus 5.5, 97.8% for GPT-6 Astra, 98.4% for GPT-6 Sol, 97.6% for GPT-6.1 Sol); Claude Code also counts cache writes.

All 108 sessions

Success of each session and their mean (%), then API-equivalent cost ($) and input tokens per session. Last row: mean over all sessions, total cost and total tokens.

Task	Claude Opus 5.5 (high)				GPT-6 Astra (high)				GPT-6 Sol (high)				GPT-6.1 Sol (high)
Task	Sessions	Mean	$	Tokens	Sessions	Mean	$	Tokens	Sessions	Mean	$	Tokens	Sessions	Mean	$	Tokens
Dishwasher close	0 / 0 / 100	33	20.3	37M	100 / 100 / 100	100	11.7	9M	0 / 0 / 0	0	6.9	25M	0 / 0 / 0	0	2.5	11M
Dishwasher load cups	83 / 1 / 1	28	27.9	72M	18 / 16 / 30	21	24.4	18M	2 / 0 / 3	2	10.1	42M	36 / 2 / 15	18	2.1	11M
Drawer close	100 / 100 / 100	100	1.7	3M	96 / 100 / 100	99	1.6	1M	98 / 100 / 100	99	1.2	4M	100 / 100 / 100	100	0.3	1M
Drawer open	99 / 98 / 100	99	11.0	26M	100 / 97 / 100	99	6.6	5M	55 / 66 / 36	52	4.7	18M	40 / 100 / 34	58	0.9	4M
Move plate	21 / 15 / 23	20	27.8	51M	36 / 2 / 4	14	18.8	13M	1 / 7 / 5	4	4.2	16M	7 / 0 / 17	8	0.9	4M
Move two plates	8 / 15 / 22	15	23.0	58M	0 / 38 / 9	16	12.3	8M	0 / 0 / 0	0	6.2	25M	0 / 8 / 0	3	2.0	10M
Pick box	44 / 99 / 71	71	10.6	26M	64 / 85 / 12	54	12.5	9M	0 / 0 / 0	0	5.9	21M	0 / 52 / 0	17	1.5	8M
Reach dual	71 / 99 / 65	78	10.2	24M	100 / 93 / 75	89	6.0	4M	33 / 58 / 20	37	3.8	14M	82 / 61 / 66	70	0.8	3M
Reach multi-modal	100 / 100 / 100	100	7.7	12M	75 / 97 / 95	89	4.6	3M	73 / 86 / 99	86	1.2	4M	81 / 97 / 100	93	0.9	3M
All 9 tasks	27 sessions	60.6	420	921M	27 sessions	64.5	296	208M	27 sessions	31.2	133	508M	27 sessions	40.7	36	169M

Summary

BiGym 2.0: 20 household tasks on the Unitree G1 under whole-body control, with 60 human VR demonstrations per task.
Four kinds of policy on the same inputs and hidden seeds: vision-language-action fine-tuning, imitation learning, demo-driven reinforcement learning, and coding agents that write a program from one demonstration.
Coding agents handle spatial variation well but struggle to keep physical contact; learned policies show the reverse, and multi-object transport stays hard for all of them.

household tasks: 20; reaching, table-top, dishwasher, kitchen counter
human VR demonstrations: 1,200; 60 per task, full simulator state per frame
hidden evaluation seeds: 100; bit-exact replay, one protocol for every method
step budget: 101k; shared by online learners and the coding agents

BibTeX

citation.bib

@article{zhang2026bigym2,
  title   = {BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation},
  author  = {Zhang, Zexi and Zhu, Zecheng and Chen, Zidong and Tuya, Zulkhuu and James, Stephen},
  journal = {arXiv preprint arXiv:2610.07594},
  year    = {2026}
}

BiGym 2.0