Explore the scenes

Md Selim Sarowar and Sungho Kim
Advanced Visual Intelligence Lab, Yeungnam University

Pose-posterior grasp planning

Robot grasps planned on where an object could be, not on one pose estimate.

A cluttered LM-O test image with the FoundationPose estimate of every object as an orange box and sixteen posterior samples as teal boxes.
Point estimatePosterior

A pose that passes the accuracy test can still drop the object. Planning on a pose posterior lifts it.

Top-1 grasp success 0.791 to 0.859, level with the annotated pose (0.856).

Small errors, lost grasps

One LM-O driller. Top: the FoundationPose estimate, 4 percent of the diameter off, and its grasp, which drops the driller in MuJoCo. Bottom: sixteen posterior samples and the grasp they choose, which lifts it.

0.984

success where the estimate is off by 5 to 10% of the object's diameter, against 0.796 on the point estimate

0.972

of certified grasps succeed, against 0.904 when planning on the point estimate

Unseen clutter

LM-O test frames, never used for tuning. Over all 1,474 objects the posterior grasp lifts 0.750, the point-estimate grasp 0.683.

One pose, many poses

The estimator stays frozen. Its own measured errors become the uncertainty. Open the full figure

Estimate

FoundationPose returns one pose and no uncertainty.

The RGB-D input with the visible mask, and the FoundationPose estimate as an orange box.

Residual bank

Its errors on annotated training frames, kept with fit features.

Training frames with the estimate and the annotation, and the bank of their errors.

Retrieve

Training frames whose fit looks alike lend their errors.

Depth points coloured by their distance to the posed model, and the bank sorted by feature distance with the drawn rows marked.

Sample

Sixteen poses the object could be in.

Sixteen posterior samples as teal boxes on the image and on the depth point cloud.

Roll out

Every grasp at every pose, under four physics draws.

Sixteen candidate grasps and MuJoCo rollouts of two of them at several pose samples.

Decide

Act on the best grasp, certify it, or look again.

Simulated success under every sample, the posterior mean per grasp, and the certificate and execution checks of the chosen grasp.

Every method,one table

LM-Grasp test split, 2,579 scenes. Top-1: success of the chosen grasp. AUC: ranking within a scene. Sel@25%: success on the most confident quarter of scenes. Cert.: certified fraction and precision at α = 0.1. Mean ± s.d. over training seeds.
MethodnTop-1AUCSel@25%Cert.
Learned outcome scorers (no object model)
Dex-Net 2.0 architecture50.637 ± 0.0130.8290.9020.611 / 0.831
RGB-D outcome scorer50.683 ± 0.0060.8760.9290.647 / 0.857
Deep ensemble of 530.705 ± 0.0060.8910.9380.664 / 0.884
Learned fusion with the planner score
POISE (scorer + point score)50.821 ± 0.0020.9580.9630.874 / 0.927
POISE-U (scorer + posterior score)50.859 ± 0.0020.9880.9850.851 / 0.972
Pose-then-plan (mesh, simulator and physics prior at decision time)
Pose regressor + planner50.707 ± 0.0070.8590.817–
FoundationPose + planner10.7910.9350.9020.844 / 0.904
FoundationPose + planner, 64 draws10.8140.9510.919–
Pose posterior (ours)10.8590.9870.9880.849 / 0.972
+ second view (ours)10.8650.9930.997–
Privileged references
Annotated pose + planner10.8560.9900.9840.854 / 0.974
Oracle (best candidate)10.877–––
Grasp success falls as the pose error grows while the ADD(-S) pass rate stays high; binned by FoundationPose error, the posterior planner tracks the annotated pose.
Pose error against grasp success
Top-1 success of every method, success on the most confident scenes, and conformal certificates.
Every method, and when to act
Calibration of the posterior in rotation and translation, and the gain from a second view of the least confident scenes.
Calibration and a second view