AFFORD-BENCH · PHYSICAL CONTACT ROBUSTNESS
AFFORD-Bench measures whether robot policies adapt under mid-trajectory physical contact disturbances without causing dangerous contact force spikes (F > 150 N) or motor torque saturation. Every Dynamic rollout is counterfactually matched to a Base nominal control sharing identical scene setup, seeds, and executed prefix a0:t*.
the physical evaluation gap
Base and Dynamic share the identical executed prefix a0:t*, isolating the outcome effect of exogenous contact disturbances.
STATIC BENCHMARKS (LIBERO, ROBOMIMIC, LIBERO-PLUS)
Static benchmarks test generalization from shifted initial poses. Crucially, they lack actuator motor torques and contact force sensors—leaving violent collisions, jamming, and motor stalls completely invisible.
AFFORD-BENCH PHYSICAL DYNAMIC ROBUSTNESS
Measures Compliant Dynamic Success: Ssafe = Sdyn ∧ (Fmax ≤ 150 N) ∧ (|τj| ≤ τmax,j). Evaluates whether policies absorb disturbances or cause catastrophic contact damage.
the physical taxonomy
Frozen across 960 matched evaluation pairs
Workpiece displaced ±3 cm along ±x, ±y, or ±xy mid-approach, requiring visual-tactile re-acquisition.
The insertion hole or receptacle shifts during transit, testing whether insertion adapts dynamically.
Monitors external contact wrenches against the ISO/TS 15066 safety bound (Fmax ≤ 150 N).
Checks joint motor torque limits [87, 87, 87, 87, 12, 12, 12] N·m to detect actuator stall.
Non-reversible mechanical locking requiring continuous downward threshold press before twist unlock.
Spring-loaded doors and caps exhibiting nonlinear opposing torque during turning and opening.
Thin plates pushed over table overhang edges before fingers can establish a stable pinch grasp.
Sudden clearance snags during peg and pin insertion causing part jamming and high friction.
benchmark construction
From 210 core tasks across D1–D7 to 30 frontier reactive tasks in D8, frozen into 960 deterministic evaluation pairs.
| Benchmark | Tasks | Dynamic Mid-Task Shift | Actuator Torques (τact) | Contact Force (Fcontact) | Force Limit Check (150 N) | Frozen Manifests |
|---|---|---|---|---|---|---|
| AFFORD-Bench (Ours) | 240 | ✓ Paired Prefix (960 pairs) | ✓ 7-DoF Motor Torques | ✓ Real-Time Wrench | ✓ Compliant Ssafe | ✓ Full + Lite |
| LIBERO-MAX (2026) | 130 | ✓ Paired Prefix | ✗ None (pixels only) | ✗ None | ✗ None | ✓ Full + Lite |
| robomimic (2021) | 7 | ✗ Static reset only | ✗ None | ✗ None | ✗ None | ✗ Ad-hoc |
| LIBERO-PRO (2024) | 100 | ✗ Static OOD reset | ✗ None | ✗ None | ✗ None | ✗ None |
experimental findings
Four policy families evaluated on 960 matched pairs using released inference parameters.
of nominal Base successes fail under dynamic contact shift.
Furthermore, 78.4% of surviving kinematic rollouts violate physical safety limits—causing severe part jamming, torque saturation (>87 N·m), or exceeding the 150 N human-robot collaborative force ceiling.
| Policy | Base Rate (%) | Dynamic Rate (%) | Drop (pp) | Compliant Safe Ssafe (%) | Peak Force Fmax (N) | Torque Saturation |
|---|---|---|---|---|---|---|
| Compliant Closed-Loop Oracle | 95.2[93.8, 96.5] | 91.4[89.6, 93.1] | -3.8 | 88.1[86.0, 90.1] | 32.4 ± 6.2 | 2.1% |
| Diffusion Policy (Vision-State) | 74.1[71.3, 76.8] | 38.3[35.2, 41.4] | -35.8 | 15.2[12.9, 17.6] | 248.5 ± 34.1 | 46.8% |
| Diffusion Policy (State-Only) | 79.3[76.7, 81.8] | 42.1[38.9, 45.2] | -37.2 | 18.4[15.9, 21.0] | 214.2 ± 28.5 | 39.2% |
| BC-RNN (Vision-State) | 62.4[59.3, 65.4] | 21.2[18.6, 23.9] | -41.2 | 8.3[6.6, 10.2] | 284.1 ± 42.0 | 58.4% |
| BC-MLP (State-Only) | 68.0[65.0, 70.9] | 24.5[21.8, 27.3] | -43.5 | 9.1[7.3, 11.1] | 276.9 ± 38.2 | 54.2% |
| Open-Loop Replay | 97.4[96.3, 98.4] | 3.1[2.0, 4.3] | -94.3 | 1.2[0.6, 2.0] | 310.8 ± 45.0 | 82.6% |
Italics indicate 95% bootstrap confidence intervals over 960 evaluation pairs. Compliant Safe Success requires Ssafe = Sdyn ∧ (Fmax ≤ 150 N) ∧ (|τj| ≤ τmax,j).
open benchmark
Every benchmark manifest, schema specification, evaluation launcher, and torque channel validator is fully reproducible.
# 1. Clone repository & install
$ git clone https://github.com/Physical-Super-Intelligence/AFFORD_Bench.git
$ cd AFFORD_Bench
$ pip install -e .
# 2. Validate frozen benchmark manifests
$ python scripts/validate_manifests.py
# 3. Run matched paired evaluation (Lite track: 96 pairs)
$ python scripts/evaluate_paired_dynamic.py \
--manifest benchmark/lite/afford_bench_lite.json \
--out results/paired_lite.jsoncitation
If you use AFFORD-Bench or its torque/force channels in your research, please cite our RSS 2027 paper. Download BibTeX.
@article{sarowar2026affordbench,
title = {AFFORD-Bench: A Benchmark for Physical Affordance, Compliance, and Reactive Contact Robustness in Robot Manipulation},
author = {Sarowar, Selim and Physical Superintelligence Research Team},
institution = {Yeungnam University},
journal = {Robotics: Science and Systems (RSS)},
year = {2027},
url = {https://github.com/Physical-Super-Intelligence/AFFORD_Bench}
}