starRoboHarness
Reliable Evaluation and Efficiency Analysis for Policy–Reasoner Robot Control.
Ongoing developmentAn auditable boundary between reasoning and robot action
starRoboHarness binds policy proposals to exact observations, checks reasoner decisions before execution, and records actual action acknowledgements and native terminal outcomes. Policy, reasoner, and environment implementations connect through adapters.
Across the same 50 saved RoboDojo scene layouts, historical QwenPI_v3 + GPT-6 Astra records 17/50 successes, QwenPI_v3 alone 9/50, and an independent pure π0.5 campaign 10/50. The two QwenPI cohorts and the later π0.5 run differ in collection conditions, so these are descriptive comparisons. In a still incomplete π0.5 + Astra continuation, nine cases have native terminal receipts; a separate ten-scene policy-only cadence control helps test how much shorter action prefixes matter. We report completion, partial score, validity, and inference cost together.
Keywords: robot policy · multimodal reasoner · hybrid control · native evaluation · execution trace
Use strong reasoning where it matters most
Robot policies produce fast, continuous action sequences, while multimodal reasoners can inspect visible failures, challenge a policy's next intent, and plan short recoveries. The central question is how to combine those strengths without turning uncertainty into uncontrolled intervention.
starRoboHarness treats the learned policy as the default controller. A reasoner observes the same public evidence, reviews the previous execution and the next proposal, and receives authority only through explicit gates. The design keeps benchmark instructions and native outcomes intact while making policy families, reasoners, robots, and simulators replaceable behind adapters.
Protocol first
Protocols stay stable while policies, reasoners, robots, and benchmarks remain replaceable.
Evidence gated
Uncertainty alone never authorizes takeover. Every correction must cite observable evidence.
Bounded control
A student prefix uses at most 15 steps; a local correction uses at most five before observing again.
Trace everything
Proposals, decisions, actual acknowledgements, native outcomes, versions, cost, and latency are recorded.
The policy executes fluently; the reasoner intervenes selectively
Every control cycle starts from the benchmark's original instruction and an exact observation. StarVLA/QwenPI_v3 proposes an action chunk. The reasoner separately assesses the previous execution and the visible intent of the next proposal, then either passes the policy proposal or requests a bounded end-effector correction.
The QwenPI_v3 policy contract uses three ordered RGB views, the raw 14D dual-ARX-X5 joint state, and continuous gripper opening with 0 = closed and 1 = open. QwenPI_v3 predicts 50 absolute joint-position actions; the historical hybrid executes at most 15 policy actions before replanning. Pure-policy baselines execute 16 actions per replan. This cadence difference is part of the comparison.
Replace adapters, not the control system
StarVLA, OpenPI, RoboDojo, Isaac Sim, and provider SDKs remain outside the lightweight core package.
Two five-task panels, 50 exact case pairs
Each panel compares QwenPI_v3 policy execution with GPT-6 Astra supervising QwenPI_v3 proposals. The original and added panels were collected in different periods, so their aggregates stay separate. Native simulator receipts determine success and score.
Hybrid successes · policy 1/25
Hybrid successes · policy 8/25
Hybrid · policy 9.0
Hybrid · policy 38.4
Original five-task panel
Five exact case pairs per task. Score is the mean native partial score on a 0–100 scale.
| Task | Policy success | Hybrid success | Policy score | Hybrid score |
|---|---|---|---|---|
| Organize table | 0/5 | 0/5 | 10 | 65 |
| Classify by language | 0/5 | 0/5 | 0 | 0 |
| Imitate sorting | 0/5 | 1/5 | 1 | 36 |
| Arrange largest number | 0/5 | 1/5 | 2 | 37 |
| Fold clothes | 1/5 | 4/5 | 32 | 80 |
| Panel total | 1/25 | 6/25 | 9 | 43.6 |
Added five-task panel
Five exact case pairs per task. Score is the mean native partial score on a 0–100 scale.
| Task | Policy success | Hybrid success | Policy score | Hybrid score |
|---|---|---|---|---|
| Put bottles in bin | 3/5 | 4/5 | 64 | 82 |
| Classify objects | 0/5 | 0/5 | 6 | 22 |
| Build tower | 4/5 | 4/5 | 82 | 86 |
| Make Kong | 1/5 | 3/5 | 20 | 60 |
| Pack objects in box | 0/5 | 0/5 | 20 | 2 |
| Panel total | 8/25 | 11/25 | 38.4 | 50.4 |
Valid native timeouts remain scored failures. Provider and infrastructure failures are kept in a separate attempt ledger. The 18 excluded pre-control attempts in the added-policy collection do not enter either denominator. Download sanitized case rows · Download computed summary.
Task-level findings
Fold clothes and Make Kong drive gains
Fold clothes rises from 1/5 to 4/5 successes in the original panel. Make Kong rises from 1/5 to 3/5 in the added panel.
More score can still mean zero success
Added-panel classification rises from mean score 6 to 22, while both methods remain at 0/5 completions.
Packing needs a trajectory study
Both methods fail all five packing cases, and hybrid mean score falls from 20 to 2. All five hybrid episodes reach the native 1,300-step limit.
Successful runs can use zero corrections
The archived build-tower success and two later successful development cases record zero correction steps. Approval and intervention must be studied separately.
Quality and usage belong on the same page
The hybrid records 4,627 correction steps, 462.5 million reported input tokens, and 1.56 million output tokens. About 93.95% of input tokens are recorded as cached. These are asynchronous usage snapshots, not final billing.
Mean control-loop wall time was 330.5 s for policy and 4,999.8 s for hybrid in the added panel. Hardware, transport, and collection period were different, so this is operational telemetry rather than a controlled speed ratio. The combined hybrid index is 17/50 successes and score 47.0; it is a descriptive union of two historical blocks.
Policy choice changes which layouts succeed
The π0.5 comparator uses a RoboDojo-fine-tuned OpenPI checkpoint. Its 50 selected scene-layout files match the historical QwenPI panel by SHA-256, but it was run in a separate campaign, without Astra. Each cell gives native successes out of five and mean partial score out of 100.
Mean native score 23.7
Mean native score 47.0
Mean native score 31.8
π0.5-only ↔ QwenPI-only successes
All ten tasks, three completed methods
Each task has five saved layouts. Mean score is descriptive across separately collected methods.
| Task | QwenPI | QwenPI + Astra | π0.5 |
|---|---|---|---|
| Organize table | 0/5 · 10 score | 0/5 · 65 score | 0/5 · 35 score |
| Classify by language | 0/5 · 0 score | 0/5 · 0 score | 0/5 · 0 score |
| Imitate sorting | 0/5 · 1 score | 1/5 · 36 score | 0/5 · 0 score |
| Arrange largest number | 0/5 · 2 score | 1/5 · 37 score | 0/5 · 3 score |
| Fold clothes | 1/5 · 32 score | 4/5 · 80 score | 0/5 · 16 score |
| Put bottles in bin | 3/5 · 64 score | 4/5 · 82 score | 3/5 · 76 score |
| Classify objects | 0/5 · 6 score | 0/5 · 22 score | 1/5 · 26 score |
| Build tower | 4/5 · 82 score | 4/5 · 86 score | 3/5 · 68 score |
| Make Kong | 1/5 · 20 score | 3/5 · 60 score | 3/5 · 60 score |
| Pack objects in box | 0/5 · 20 score | 0/5 · 2 score | 0/5 · 34 score |
| Total / mean | 9/50 · 23.7 score | 17/50 · 47.0 score | 10/50 · 31.8 score |
Four layouts succeed with both pure policies; six succeed only with π0.5 and five only with QwenPI. Their 15-layout union is a retrospective complementarity count, not a deployable selector. All three completed methods fail every language-classification layout. Packing has no completion, yet mean scores differ: QwenPI 20, QwenPI + Astra 2, π0.5 34. Download panel summary.
Interim π0.5 + Astra evidence
Of a planned 50-scene continuation, nine exact layouts currently have valid native terminal receipts. The hybrid completes five and the original-cadence pure π0.5 run completes two on those nine layouts. This is selected interim case evidence, not a 50-scene success rate.
Language classification, layout 0
Pure π0.5 scores 0 under original, fixed-15, and fixed-5 execution. The hybrid reaches native success and score 100 at step 711, with 44 accepted policy prefixes and eleven five-step corrections. This is one trajectory, not proof of a general correction effect.
Sorting, layout 0
The hybrid reaches score 50 with 385 correction steps but does not complete the task. Pure π0.5 under original, fixed-15, and fixed-5 cadences scores 0.
Monitoring can still be expensive
Four successful hybrid trajectories make no correction, yet together use 3,731 s of controller time and 34.10 million reported tokens including cache. Completion alone cannot credit the reasoner.
Unscored attempts still consume resources
The nine valid cases record 1,180 correction steps and 187.27 million reported tokens including cache. Fifteen further attempts lack native termination and are excluded from outcome rates; eight usage snapshots already sum to 110.51 million tokens, a lower bound.
Shorter prefixes can change the outcome
On one saved layout from each task, pure π0.5 executes 15 actions from each new 50-action prediction without a reasoner. Against its original cadence, scores rise on three layouts, fall on three, and tie on four. Fixed-15 completes object classification and folding without corrections, matching two hybrid successes; it loses the original Make Kong completion. The hybrid column is shown only where a valid native receipt exists.
One native rollout per method and layout
✓ = native completion; × = valid native failure; number = partial score; — = no valid hybrid terminal.
| Task · layout 0 | π0.5 original | π0.5 fixed 15 | π0.5 + Astra |
|---|---|---|---|
| Organize table | × · 75 | × · 50 | × · 50 |
| Classify by language | × · 0 | × · 0 | ✓ · 100 |
| Imitate sorting | × · 0 | × · 0 | × · 50 |
| Arrange largest number | × · 15 | × · 15 | × · 15 |
| Fold clothes | × · 20 | ✓ · 100 | ✓ · 100 |
| Put bottles in bin | ✓ · 100 | ✓ · 100 | ✓ · 100 |
| Classify objects | × · 15 | ✓ · 100 | ✓ · 100 |
| Build tower | × · 30 | × · 10 | — |
| Make Kong | ✓ · 100 | × · 0 | — |
| Pack objects in box | × · 10 | × · 25 | — |
On seven scenes with both fixed-15 and hybrid receipts, scores tie in five; language classification and sorting are higher with the corrected hybrid. Fixed-5 pure-policy checks also score zero on those two scenes. One run per method and layout cannot separate cadence, supervision, and stochastic effects. Download cadence comparison.
Selected episode replays
These selected clips come from an earlier five-task website release. They illustrate execution paths but are not the case-level evidence for the 50 pairs summarized above. Choose a policy or hybrid replay for each task.
Report quality, validity, and cost together
A valid comparison aligns task, layout, seed, instruction, action contract, and native termination. Success and score remain environment outcomes rather than model self-reports.
Use the environment's terminal receipt, never the model's own claim of success.
Compare methods case by case with task, layout, seed, and termination aligned.
Measure interventions made without sufficient evidence of failure or misalignment.
Measure whether the system regains progress after an observed execution failure.
Report total calls, input and output tokens, and cached-input usage.
Track end-to-end reasoner decision and student-policy latency separately.
Measure the share of executed control steps that came from reasoner corrections.
Report compute and infrastructure failures separately from model outcomes.
What we can test next
Each completed panel contains only five cases per task. The historical QwenPI cohorts and independent π0.5 campaign differ in collection periods, provider sessions, infrastructure, and policy cadence. The 50-pair pooled counts are descriptive: nine hybrid-only successes, one policy-only success, and 40 ties. They should not be read as a single prospectively frozen trial.
Packing
Trace time after a verified grasp, transfer distance per correction, and the evidence that triggered each takeover.
Classification
Track credited objects and whether repeated corrections revisit the same state before timeout.
Controlled follow-up
Freeze the method identity and rerun exact cases on matched hardware, provider routes, and action cadence.
Full accounting
Report native outcome, paired score, correction ratio, calls, tokens, decision latency, and excluded attempts together.
The nine valid π0.5 + Astra cases and ten fixed-15 controls are exploratory one-run-per-layout evidence, not a finished factorial experiment. The three-case dev validation is separate from these panels: two successes, one native timeout, and no concurrent policy baseline. Astra Direct has only one early off-panel build-tower case in the report and is not part of the paired tables.
From trustworthy baselines to a transferable control layer
Reproduction
Freeze manifests and protocol boundaries for GPT-as-Policy and QwenPI_v3.
Core
Establish contracts, gates, traces, fake runtimes, and conformance tests.
Native QwenPI_v3
Establish native execution receipts and reconstruct historical paired baselines.
Controlled quality and efficiency
Finish the π0.5 + Astra continuation, study packing and classification traces, and freeze matched runs that report quality and full cost together.
Generalization
Add a non-VLA policy and a second environment without adding platform constants to the core package.
The project succeeds when
Every intervention has evidence, and every improvement survives paired evaluation.
A new policy should require only a small adapter, while every result states its validity boundary, latency, and compute cost.