Computer Use Provider Evidence

This layer defines the evidence contract for real-model Computer Use runs. It does not claim that any provider has completed a real run.

Scenario Contract

The scenario library defines:

  • an owned Electron fixture;
  • the exact user prompt and expected state;
  • forbidden effects;
  • allowed actions and per-action budgets;
  • required execution capabilities;
  • deterministic state evaluation.

The fixture helper imports Electron only. It does not import Maka Runtime, provider transports, or execution backends.

Report Contract

Reports separate three evidence classes:

  • real-runtime: a live provider model used the production Maka runtime;
  • fault-injection: a live provider and Runtime exercised a named injected failure, but the run cannot qualify as real host evidence;
  • hermetic-protocol: a fake transport proved protocol behavior;
  • static-contract: source or schema checks only.

Only real-runtime can satisfy a provider matrix cell marked real. Policy-bypassed runs remain visibly labeled and cannot become an unqualified pass.

Real reports also fail closed unless producer, transport, policy mode, model, fixture PID/window ownership, latest-observation lineage, action budgets, and dispatch provenance are explicit. Expected failures must be authorized by the scenario; a report cannot authorize itself.

The sanitizer preserves action types, timing, result codes, aggregate state, and allowlisted trace fields. It removes coordinates, typed text, raw UI content, credentials, full URLs, and provider payloads.

Consolidation Findings

Review found that the AppKit producer emitted traces while qualification read driverTraces, and the Desktop launcher derived its fixture window allowlist from the actions being judged. The first mismatch rejected valid AX evidence; the second allowed circular ownership proof.

Both producers now emit one canonical schema. Desktop fixture identity is collected independently from the launcher-owned PID and cua-driver window inventory before model execution. Qualification waits for matching dispatch traces and requires each target to belong to that independent identity.

Qualification also keeps three fail-closed invariants:

  • restart recovery authorizes only the scenario-declared stale set_value / target_missing result and budgets the required fresh retry;
  • disallowed and over-budget model attempts are recorded as canonical failed action evidence before the harness rejects them;
  • the Desktop launcher and provider matrix call the same real-report validator, so a launcher cannot exit successfully for a matrix-invalid report.

The old direct real-machine qualification runner was removed. The five-round restart runner remains available as e2e:computer-use-process-restart-soak, but is regression-only and cannot satisfy a provider matrix cell. There is one qualification path rather than parallel evidence standards.

Next Layer

A provider launcher must:

  1. pin a scenario from this library;
  2. run against the owned fixture and production Computer Use backend;
  3. enforce the scenario action budget before dispatch;
  4. emit a sanitized real-runtime report;
  5. let the provider matrix validate fixture state and forbidden effects.

First Real Run

The first qualifying run completed with:

  • provider: OpenAI;
  • model: gpt-5.4;
  • evidence class: real-runtime;
  • tool exposure: direct E2E, with only the production maka_computer tool;
  • action: one app-scoped observe;
  • tool latency: 1117 ms;
  • total run latency: 7502 ms;
  • terminal status: complete / end_turn;
  • fixture oracle: verification code matched and interaction count remained zero.

The direct E2E tool exposure is deliberate. The default deferred load_tools path remains a separate product contract; the launcher narrows provider variables while still exercising the production tool implementation, permission engine, Runtime, Desktop host, and cua-driver backend.

During this run, OpenAI Responses tool continuation exposed a product bug: server-side storage generated an item_reference in the second request without a previous_response_id, so custom Responses endpoints rejected the tool result. OpenAI provider options now use store:false, matching the existing Codex subscription boundary and keeping function calls/results inline.

The next run should perform one AX semantic mutation after executor hardening is merged.

That L1 run has now completed:

  • scenario: l1-single-click;
  • provider/model: OpenAI gpt-5.4;
  • actions: two observations and one click_element;
  • no coordinate or compatibility input action was allowed;
  • semantic click latency: 1445 ms;
  • total run latency: 26023 ms;
  • fixture oracle: primary click count 1, danger click count 0, over-click count 0;
  • terminal status: complete / end_turn;
  • result: pass.

The run initially failed closed when the user's foreground ChatGPT window occluded the synthetic target. The fixture host now settles and raises its layer-0 window with showInactive() and moveTop() before declaring readiness, without focusing it or using an always-on-top overlay layer.