SPECTRA

Shades of Success

Evaluating Manipulation Quality from Object-Centric Constraints

Anonymous Authors

Paper (coming soon) arXiv (coming soon) Code (coming soon) BibTeX (coming soon)

Same success, different quality.
SPECTRA scores the quality of robotic manipulation with a Task Quality Index Q ∈ [0, 1].

How it works

Constraints are learned from successful demonstrations; new executions are scored by how well they respect them. Evaluated on seven tasks.

Part 1

Constraint estimation

Demonstrations are aligned by task progress. Relative-pose directions that vary little across demonstrations are identified as constrained, and their spread defines the admissible variation.

Part 2

Policy evaluation

Deviations of an unseen execution along constrained directions are penalized only beyond the admissible variation. The mean cost and a terminal gate are combined into the Task Quality Index Q.

Object-centric representation: reference r, manipulated m, gripper g and support s, with the five relative-pose pairs (r,m), (s,r), (s,m), (g,m), (g,r).

Object-centric, robot-agnostic, no human labels

  • Relative poses. Five pairs: the manipulated object in the reference's frame, and both objects relative to the gripper and the support.
  • Aligned by progress. Demos are split into sub-tasks (reach, grasp, carry, release) and resampled onto a shared progress axis.
  • No human labels. The reference comes from 30 successful demonstrations scripted in simulation.
  • Constrained where demos agree. At each step, any direction whose spread across 30 demos stays under 4 mm is constrained, and that spread becomes its tolerance.

Where demonstrations agree, the task is constrained

Learn constraints in simulation, score policy rollouts on hardware

Stack Cube

Place Can

Agreement with human judgment

15 raters scored 105 rollouts from Diffusion Policy, ACT and π0.5. Spearman ρ with the mean rating:

Show as table

Use SPECTRA to score demonstrations and select training data

Can quality scores pick better training data? An exploratory study on the UR3, for Stack Cube and Place Can.

  1. 1

    Scoring

    180 teleoperated demonstrations per task, each scored with SPECTRA.

  2. 2

    Selection

    The 100 highest-scoring demonstrations, compared with 100 drawn at random from the same pool.

  3. 3

    Training & evaluation

    A Diffusion Policy trained on each set, evaluated over 10 trials on the UR3.

SPECTRA quality Q of 180 teleoperated demonstrations per task, sorted by rank, for Stack Cube and Place Can; the top 100 are shaded.
Steps 1–2. Q of every demonstration, sorted. The shaded part is the top 100.

Step 3. Successful trials out of 10

BibTeX

Coming soon.