trajectory memorization
open-loop replay · closed-loop oracle
Replaying a successful trajectory, as a perfect memorizer would, reproduces all 629 episodes on the same seed but succeeds in only 8.9% of new layouts from the same nominal distribution.
PLUMB-Bench · A physically verified manipulation benchmark
PLUMB-Bench is a MuJoCo benchmark of 210 instruction-accurate manipulation tasks in seven domains, from shaft seating and electronics assembly to burger building, espresso service, and centrifuge loading. Every instruction compiles to an ordered conjunction of physical goal predicates, and a placement counts only when the body is released, supported, at rest, and within 2–8 mm of a goal pose calibrated by letting the body settle in its target, or wholly inside its target region. Five paired levels shift layout, appearance, friction, and mass, and matched Base/Dynamic rollouts displace objects mid-execution. A privileged closed-loop oracle certifies every condition, and random and null policies solve none.
physical success
Common success checks accept an object whose centre lies within centimetres of its target, whether or not it was released or came to rest. PLUMB-Bench checks every placement physically.
No finger touches the body.
A non-robot body carries it, after it was lifted at least 2 cm.
‖v‖ < 0.02 m/s and ‖ω‖ < 0.2 rad/s.
Tilted at most 5° on precision workcells, 15° elsewhere.
Of the settled goal pose at a bore, slot, or rack hole, or with its whole footprint inside a region.
Ordered goals. Each instruction compiles to goals g1, …, gk with a partial order. A goal counts only when it becomes true after every goal ordered before it, so a button pressed too early or a part already in place does not count. Besides placements (On), goals can require a latching button pressed (Pressed), a drawer or vise jaw closed (Closed), a body slid without lifting (Slid), a body held at a clearance above a target (HeldOver), a hover or touch held for consecutive steps (GripperAbove, Touching), or a body raised (Lifted).
Attainable by construction. Goal poses are not set by hand. When a task is built, every goal body is lowered into its target and simulated to rest; the settled pose becomes the goal pose, and a pose that drifts more than 5 mm while settling rejects the task. There is no attachment, snapping, or teleport: grasping is frictional and force-limited (about 35 N per finger), so a body that slides, tips, or binds on a rim fails exactly as it would physically.
task taxonomy
D1–D4 are single-goal skills; D5–D7 chain 2–6 ordered goals across workcells and precision tiers. Every object, mechanism, and target an instruction names exists in its workcell.
| Domain | Workcells | Example instruction | Goal predicates | Goals | Tol. (mm / °) |
|---|---|---|---|---|---|
| D1 Industrial | assembly jig, electronics | “Seat the transmission shaft in the jig bore.” | On, Pressed, Closed, Lifted, GripperAbove | 1 | 2 / 5 |
| D2 Culinary | burger, espresso | “Crown the burger: put the top bun on the tomato slice.” | On, Pressed, Slid, Touching | 1 | 8 / 15 |
| D3 Domestic | sorting table | “Stack the green cylinder on the red block.” | On, Closed, Slid, Touching | 1–2 | 8 / 15 |
| D4 Laboratory | laboratory bench | “Put the lid on the centrifuge.” | On, HeldOver, Touching, Pressed, Lifted, GripperAbove, Slid | 1 | 2 / 5 |
| D5 Long-horizon | five workcells | “Build the burger on the bottom bun on the cutting board: patty, then cheese, then tomato, then the top bun.” | On, Pressed, HeldOver, Slid, Closed | 2–5 | 2–8 / 5–15 |
| D6 Precision chain | jig, electronics, lab | “Seat the shaft in the jig bore, slide the bearing ring over it, then close the vise jaw.” | On, Pressed, HeldOver, GripperAbove, Closed, Touching | 2–4 | 2 / 5 |
| D7 Service chain | burger, espresso, sorting | “Put the mug on the drip tray, press the brew button, then put the mug on the saucer.” | On, Pressed, Closed, Slid | 2–6 | 8 / 15 |
Tolerances are (εxy, εθ) of placement goals; two loose fits (a bearing ring over a shaft, a centrifuge lid) use 6 mm, and chains take the tolerance of their workcell.
generalization
Suites are paired on seeds, so a change in success measures the shift rather than a new scene. Every shifted parameter is restored before the next sample.
Each moved object starts uniformly in its own region (half-widths 2–40 mm, yaw up to ±10°–±45°), in one of two receptacles, or pre-assembled; the initial TCP is jittered by ±1.5 cm per horizontal axis.
Region starts move by a radius 𝒰(7, 12) cm in a random direction, outside the L1 support, with yaw magnitude 𝒰(45°, 90°).
Light intensities scaled by 𝒰(0.5, 1.5) and light positions jittered by ±0.5 m; a separate Color suite scales the colors of the moved objects by ±25%.
Sliding friction μ ~ 𝒰(0.12, 0.25) on every moved object and both finger pads, and mass and rotational inertia scaled by 𝒰(1.8, 2.5), at unchanged layout and appearance.
L2, L3, L4, and color shifts sampled jointly.
A deployed policy must also absorb changes after it has started acting. Following LIBERO-MAX, an event case pairs a Base rollout with a Dynamic rollout that shares the task, seed, initial state, and executed action prefix and differs by one event. When the hand first arrives above the event body (within 3 cm horizontally and 15 cm above it, not yet grasped), the Dynamic rollout displaces the body by 3 cm along ±x or ±y at rest and hands control back to the policy. A displacement counts only if the body lands on a support within 3 mm of its original height and penetrates nothing by more than 1 mm.
(1, 1)preserved success
(1, 0)regression
(0, 1)recovery
(0, 0)persistent failure
Reported per policy: the paired change ΔSR and the conditional regression rate CRR = n10 / (n11 + n10).
related benchmarks
Nine design dimensions; every cell is taken from the cited papers and code.
feasibility ceilings and floors
The oracle reads poses that policies do not observe, so its success bounds solvability rather than predicting learned performance.
| Condition | Oracle | Random | Null |
|---|---|---|---|
| L1 Nominal | 99.9[99.7, 100.0] | 0 / 630 | 0 / 210 |
| L3 Visual | 99.9[99.7, 100.0] | 0 / 630 | 0 / 210 |
| Color | 99.9[99.7, 100.0] | 0 / 630 | 0 / 210 |
| L4 Physical | 98.9[98.1, 99.5] | 0 / 630 | 0 / 210 |
| L2 Spatial | 97.7[96.4, 98.9] | 0 / 630 | 0 / 210 |
| L5 Compound | 95.6[94.0, 97.1] | 0 / 630 | 0 / 210 |
95% task-bootstrap intervals; 6,300 oracle, 3,780 random, and 1,260 null episodes. Every domain stays at 89.3% or above in every condition.
what PLUMB-Bench measures
Three probes with the privileged oracle, on seeds disjoint from data generation and evaluation.
trajectory memorization
open-loop replay · closed-loop oracle
Replaying a successful trajectory, as a perfect memorizer would, reproduces all 629 episodes on the same seed but succeeds in only 8.9% of new layouts from the same nominal distribution.
millimetre sensitivity
oracle on 2 mm tasks with a 3 mm goal error
A fixed goal-localization error of 3 mm halves oracle success on the 55 tasks whose tightest goal allows 2 mm; both point tiers reach 0% at 12 mm, an error a 3 cm predicate would still accept.
mid-execution events
Dynamic SR · open loop · oracle
Over 921 matched 3 cm displacements on 154 tasks, each Dynamic prefix bit-identical to its Base rollout, the oracle re-plans its grasp (CRR 2.4%), while continuing the Base actions regresses in 907 of 920 solved cases.
evaluation protocol
Policies train on 50 oracle demonstrations per task and are evaluated with 50 episodes per task and condition on disjoint seeds, plus the 921 event pairs.
Metrics. Success rate with 95% task-bootstrap intervals, goal completion (the fraction of goals reached), the physical robustness index PRI = SR(𝒫) / SR(L1), ΔSR and CRR for events, and the peak contact force on the robot as logged telemetry.
Reference policies. Behavior cloning, Diffusion Policy, ACT, DP3, and fine-tuned VLAs (OpenVLA, Octo, π0, SmolVLA), each trained with three seeds; 63,000 rollouts per policy over the static conditions. The benchmark is designed to test four hypotheses:
open benchmark
One privileged oracle solves all 210 tasks through the same controller and action space as a learned policy. It generates robomimic HDF5 demonstrations with both cameras, proprioception, the full simulator state, instructions, and provenance; an episode-level validator checks every demonstration, replay reproduces recorded states exactly, and converters export RLDS and LeRobot.
# install (Python 3.10+, MuJoCo 3.5.0)
$ git clone https://github.com/Physical-Super-Intelligence/PLUMB_Bench.git
$ cd PLUMB_Bench && pip install -e .
# make a task and read its instruction
$ python -c "import afford_pro; env = afford_pro.make('industrial_shaft_assembly'); \
obs, info = env.reset(seed=0); print(env.language_instruction)"
# oracle on one task across the paired suites
$ python scripts/evaluate_policy_suite.py --policy oracle \
--task industrial_shaft_assembly --episodes 5 \
--suites nominal,p2_spatial,p3_visual,p1_object,p4_physical,combined_ood \
--out-dir results/oracle
# 50 validated oracle demonstrations per task
$ python scripts/generate_all_210_tasks.py --domain industrial --num-demos 50
citation
If you use PLUMB-Bench in your research, please cite it. CITATION.bib ↗
@misc{sarowar2026plumbbench,
title = {PLUMB-Bench: A Physically Verified Benchmark for Precise, Generalizable Robot Manipulation},
author = {Sarowar, Selim and Physical Superintelligence Research Team},
year = {2026},
url = {https://github.com/Physical-Super-Intelligence/PLUMB_Bench}
}