PLUMB-Bench · A physically verified manipulation benchmark

A placement counts only when the object is released, at rest, and within millimetres of its goal.

Selim Sarowar1, Physical Superintelligence Research Team1,2 1Yeungnam University  ·  2Physical Superintelligence

PLUMB-Bench is a MuJoCo benchmark of 210 instruction-accurate manipulation tasks in seven domains, from shaft seating and electronics assembly to burger building, espresso service, and centrifuge loading. Every instruction compiles to an ordered conjunction of physical goal predicates, and a placement counts only when the body is released, supported, at rest, and within 2–8 mm of a goal pose calibrated by letting the body settle in its target, or wholly inside its target region. Five paired levels shift layout, appearance, friction, and mass, and matched Base/Dynamic rollouts displace objects mid-execution. A privileged closed-loop oracle certifies every condition, and random and null policies solve none.

Six PLUMB-Bench workcells (assembly jig, electronics bench, burger station, espresso station, laboratory bench, sorting table), each shown at the start of a multi-goal task and after the oracle completed it, with the task instruction below.
One multi-goal task per workcell, at its start and after the privileged oracle completed it (arm returned to its ready posture for visibility).
01

physical success

Success is defined by physics, not proximity.

Common success checks accept an object whose centre lies within centimetres of its target, whether or not it was released or came to rest. PLUMB-Bench checks every placement physically.

01

Released

No finger touches the body.

02

Supported

A non-robot body carries it, after it was lifted at least 2 cm.

03

At rest

‖v‖ < 0.02 m/s and ‖ω‖ < 0.2 rad/s.

04

Upright

Tilted at most 5° on precision workcells, 15° elsewhere.

05

Within 2–8 mm

Of the settled goal pose at a bore, slot, or rack hole, or with its whole footprint inside a region.

Ordered goals. Each instruction compiles to goals g1, …, gk with a partial order. A goal counts only when it becomes true after every goal ordered before it, so a button pressed too early or a part already in place does not count. Besides placements (On), goals can require a latching button pressed (Pressed), a drawer or vise jaw closed (Closed), a body slid without lifting (Slid), a body held at a clearance above a target (HeldOver), a hover or touch held for consecutive steps (GripperAbove, Touching), or a body raised (Lifted).

Attainable by construction. Goal poses are not set by hand. When a task is built, every goal body is lowered into its target and simulated to rest; the settled pose becomes the goal pose, and a pose that drifts more than 5 mm while settling rejects the task. There is no attachment, snapping, or teleport: grasping is frictional and force-limited (about 35 N per finger), so a body that slides, tips, or binds on a rim fails exactly as it would physically.

02

task taxonomy

210 tasks in seven domains over six workcells.

D1–D4 are single-goal skills; D5–D7 chain 2–6 ordered goals across workcells and precision tiers. Every object, mechanism, and target an instruction names exists in its workcell.

Seven domains of 30 tasks (27 object types, 6 latching buttons, a drawer, and a vise jaw)
DomainWorkcellsExample instructionGoal predicatesGoalsTol. (mm / °)
D1 Industrialassembly jig, electronics“Seat the transmission shaft in the jig bore.”On, Pressed, Closed, Lifted, GripperAbove12 / 5
D2 Culinaryburger, espresso“Crown the burger: put the top bun on the tomato slice.”On, Pressed, Slid, Touching18 / 15
D3 Domesticsorting table“Stack the green cylinder on the red block.”On, Closed, Slid, Touching1–28 / 15
D4 Laboratorylaboratory bench“Put the lid on the centrifuge.”On, HeldOver, Touching, Pressed, Lifted, GripperAbove, Slid12 / 5
D5 Long-horizonfive workcells“Build the burger on the bottom bun on the cutting board: patty, then cheese, then tomato, then the top bun.”On, Pressed, HeldOver, Slid, Closed2–52–8 / 5–15
D6 Precision chainjig, electronics, lab“Seat the shaft in the jig bore, slide the bearing ring over it, then close the vise jaw.”On, Pressed, HeldOver, GripperAbove, Closed, Touching2–42 / 5
D7 Service chainburger, espresso, sorting“Put the mug on the drip tray, press the brew button, then put the mug on the saucer.”On, Pressed, Closed, Slid2–68 / 15

Tolerances are (εxy, εθ) of placement goals; two loose fits (a bearing ring over a shaft, a centrifuge lid) use 6 mm, and chains take the tolerance of their workcell.

Task-set statistics: goals per task by domain, goal predicate counts over 343 goals, and placement tolerances of the 262 placement goals.
Task-set statistics. (a) Goals per task, stacked by domain. (b) Goal predicates over all 343 goals. (c) Placement tolerance εxy of the 262 placement goals (log scale): every point or stacking target allows at most 8 mm (69 allow 2 mm), and region targets (bins, trays, marked squares) require the whole footprint inside the region.
03

generalization

Static shifts on paired seeds, and events after the robot acts.

Suites are paired on seeds, so a change in success measures the shift rather than a new scene. Every shifted parameter is restored before the next sample.

Matched mid-execution events

A deployed policy must also absorb changes after it has started acting. Following LIBERO-MAX, an event case pairs a Base rollout with a Dynamic rollout that shares the task, seed, initial state, and executed action prefix and differs by one event. When the hand first arrives above the event body (within 3 cm horizontally and 15 cm above it, not yet grasped), the Dynamic rollout displaces the body by 3 cm along ±x or ±y at rest and hands control back to the policy. A displacement counts only if the body lands on a support within 3 mm of its original height and penetrates nothing by more than 1 mm.

(1, 1)

preserved success

(1, 0)

regression

(0, 1)

recovery

(0, 0)

persistent failure

Reported per policy: the paired change ΔSR and the conditional regression rate CRR = n10 / (n11 + n10).

04

related benchmarks

PLUMB-Bench against Meta-World and the LIBERO family.

Nine design dimensions; every cell is taken from the cited papers and code.

Comparison cards for Meta-World, LIBERO, LIBERO-PRO, LIBERO-Plus, LIBERO-X, LIBERO-MAX, and PLUMB-Bench along scale, robot and scenes, goal structure, placement success, distribution shift, physical shift, mid-execution events, observations, and demonstrations.
Design comparison. Circles mark whether placement success requires the body to be released and at rest, whether friction and mass are shifted, and whether events occur during execution (filled: yes; half: on a subset; open: no). Bars give the number of tasks on a log scale (LIBERO-Plus: task variants; LIBERO-MAX: matched pairs built from LIBERO-Plus and LIBERO-PRO cases). Meta-World distance thresholds are task-specific.
05

feasibility ceilings and floors

Every condition is solvable, and none by chance.

The oracle reads poses that policies do not observe, so its success bounds solvability rather than predicting learned performance.

Success (%) over all 210 tasks, 5 episodes per task and condition
ConditionOracleRandomNull
L1 Nominal99.9[99.7, 100.0]0 / 6300 / 210
L3 Visual99.9[99.7, 100.0]0 / 6300 / 210
Color99.9[99.7, 100.0]0 / 6300 / 210
L4 Physical98.9[98.1, 99.5]0 / 6300 / 210
L2 Spatial97.7[96.4, 98.9]0 / 6300 / 210
L5 Compound95.6[94.0, 97.1]0 / 6300 / 210

95% task-bootstrap intervals; 6,300 oracle, 3,780 random, and 1,260 null episodes. Every domain stays at 89.3% or above in every condition.

Heatmap of oracle success per domain and condition, all between 89.3 and 100 percent.
Feasibility ceilings per domain and condition; shading marks the shortfall from 100%.
Audit. Every task builds, so every goal pose settles stably in its target. Over 4,200 audit resets, no goal held at reset, no start fell back to a default layout, and every region start varied with a standard deviation of at least 0.99 cm. In a validation run the oracle completed 2,096 of 2,100 nominal episodes and every task at least 9 of 10; the lift requirement, the release-and-rest condition, and goal ordering leave no goal satisfiable by chance.
06

what PLUMB-Bench measures

Success requires closed-loop precision, not recall.

Three probes with the privileged oracle, on seeds disjoint from data generation and evaluation.

trajectory memorization

8.9%vs100%

open-loop replay · closed-loop oracle

Replaying a successful trajectory, as a perfect memorizer would, reproduces all 629 episodes on the same seed but succeeds in only 8.9% of new layouts from the same nominal distribution.

millimetre sensitivity

50.9%

oracle on 2 mm tasks with a 3 mm goal error

A fixed goal-localization error of 3 mm halves oracle success on the 55 tasks whose tightest goal allows 2 mm; both point tiers reach 0% at 12 mm, an error a 3 cm predicate would still accept.

mid-execution events

1.4%vs97.5%

Dynamic SR · open loop · oracle

Over 921 matched 3 cm displacements on 154 tasks, each Dynamic prefix bit-identical to its Base rollout, the oracle re-plans its grasp (CRR 2.4%), while continuing the Base actions regresses in 907 of 920 solved cases.

Three panels: open-loop replay versus closed-loop oracle success on same and new layouts; oracle success against goal-localization error by tolerance tier; paired outcomes of 921 mid-execution displacements.
(a) Open-loop replay of a successful nominal oracle trajectory versus the closed-loop oracle on the same seeds: same layout, friction and mass shifted on that layout (L4; 621 seeds), and a new layout under L1 and L2 (629 source episodes, 3 seeds per task). (b) Oracle success when its estimate of every placement goal is offset by a fixed horizontal error, by tolerance tier (tasks in parentheses; dashed lines at 2 and 8 mm). (c) Paired outcomes of the 921 matched mid-execution displacements.
07

evaluation protocol

How learned policies are evaluated.

Policies train on 50 oracle demonstrations per task and are evaluated with 50 episodes per task and condition on disjoint seeds, plus the 921 event pairs.

Metrics. Success rate with 95% task-bootstrap intervals, goal completion (the fraction of goals reached), the physical robustness index PRI = SR(𝒫) / SR(L1), ΔSR and CRR for events, and the peak contact force on the robot as logged telemetry.

Reference policies. Behavior cloning, Diffusion Policy, ACT, DP3, and fine-tuned VLAs (OpenVLA, Octo, π0, SmolVLA), each trained with three seeds; 63,000 rollouts per policy over the static conditions. The benchmark is designed to test four hypotheses:

08

open benchmark

Install, run the oracle, generate data.

One privileged oracle solves all 210 tasks through the same controller and action space as a learned policy. It generates robomimic HDF5 demonstrations with both cameras, proprioception, the full simulator state, instructions, and provenance; an episode-level validator checks every demonstration, replay reproduces recorded states exactly, and converters export RLDS and LeRobot.

# install (Python 3.10+, MuJoCo 3.5.0)
$ git clone https://github.com/Physical-Super-Intelligence/PLUMB_Bench.git
$ cd PLUMB_Bench && pip install -e .

# make a task and read its instruction
$ python -c "import afford_pro; env = afford_pro.make('industrial_shaft_assembly'); \
    obs, info = env.reset(seed=0); print(env.language_instruction)"

# oracle on one task across the paired suites
$ python scripts/evaluate_policy_suite.py --policy oracle \
    --task industrial_shaft_assembly --episodes 5 \
    --suites nominal,p2_spatial,p3_visual,p1_object,p4_physical,combined_ood \
    --out-dir results/oracle

# 50 validated oracle demonstrations per task
$ python scripts/generate_all_210_tasks.py --domain industrial --num-demos 50
09

citation

Cite PLUMB-Bench.

If you use PLUMB-Bench in your research, please cite it. CITATION.bib ↗

BIBTEX · PLUMB-Bench
@misc{sarowar2026plumbbench,
  title  = {PLUMB-Bench: A Physically Verified Benchmark for Precise, Generalizable Robot Manipulation},
  author = {Sarowar, Selim and Physical Superintelligence Research Team},
  year   = {2026},
  url    = {https://github.com/Physical-Super-Intelligence/PLUMB_Bench}
}