Finished Is Not Accepted

Abstract

Coding agents produce useful software, but evaluating their work requires specifying what counts as completing a task. Finishing an agent run, accepting a change, publishing it, deploying it and observing the intended result answer different questions. We ask: what evidence is needed to establish that a coding agent has achieved the outcome required by its task? We reconstruct selected task records and software histories from four organisational sources, using source-specific record definitions and owner accounts of how the records were written. The reconstruction shows why evidence must be interpreted in relation to the task: a read-only review need not produce a code change, and a recorded repository revision need not identify the change accepted for that task. We contribute an evaluation contract that specifies the required outcomes and ties each observation to a task, revision, environment and observer, while distinguishing missing evidence from failure. This provides a concrete basis for documenting achieved work and designing prospective evaluations of task completion. The retrospective records support these methodological distinctions; they do not measure organisational performance or establish that the proposed contract improves it.

1. Introduction

A coding agent's work can result in a reviewed patch, a published revision or a working service. Evaluating that achievement requires knowing which result the task called for and what observation establishes it. Passing tests can support a patch review; observing the deployed revision and checking the intended user path answer further questions when the task requires a working service. An agent's final message can summarize these observations, but the message alone does not establish them.

These are familiar distinctions in software engineering. Their importance for coding agents is that many interfaces compress them into a single notion of completion. That compression obscures where work remains, who has authority to perform it and what observation would settle the outcome. We ask what evidence establishes that an agent achieved the outcome required by its task, and propose retaining separate evidence obligations for that outcome.

The claim is not that benchmark scores are meaningless. METR's maintainer-review study gives direct evidence that automated passing and review acceptance can diverge, while acknowledging limited repositories, reviewer noise and the absence of feedback iteration. Our contribution extends the measurement boundary through the surrounding controller and delivery system rather than claiming to discover the test/review gap.

The strongest objection is that CI, release engineering and end-to-end testing already distinguish these events. We do not propose another mandatory delivery pipeline. The contribution is an evaluation contract: select the obligations required by the task, bind each observation to its revision and environment, and preserve missing or invalidated evidence when scoring accepted completion. This permits prospective forecasts of a specified outcome under a budget without treating a provider's final message as ground truth. Its usefulness must exceed the cost of the additional observation.

W3C PROV-DM already represents entities, activities, agents, derivation and responsibility. in-toto provides cryptographic verification of software supply-chain integrity. Neither is a substitute for specifying which result this particular task requires. Our proposed contract can use such provenance and integrity evidence; its added application-level choice is which native, accepted, published, deployed and user-result predicates to evaluate, for which revision/environment and horizon. CI and release records can discharge some of those obligations. Agent evaluation must still specify the rubric and preserve unsupported or invalidated coordinates. This is an operational synthesis, not a claim to invent provenance or release engineering, or a demonstrated improvement over either framework.

2. A completion record needs more than a Boolean

Represent a task outcome as evidence for N, A, P, D and U:

These are evidence coordinates, not a universal linear workflow. A research task may require only A; publication and deployment are not applicable. A pre-existing deployment may be observed without a new publication. Retries may lead to a new candidate, and a once-successful service can regress. Preserve timestamps and invalidation rather than converting an old observation into permanent status.

A record should include (task_id, attempt_id, candidate_sha, predicate_id, environment_id, observer, observed_at, result, evidence_ref). Record not-applicable separately from unknown and failed. Do not join a test result for one candidate to deployment of another. When source, configuration or the external dependency changes, re-evaluate which observations still apply.

Figure 1. Independent evidence coordinates and invalidation

Conceptual illustration of the measurement contract. Arrows bind evidence to a task or mark affected evidence for review; they do not prescribe an N → A → P → D → U sequence.

For an illustrative local manuscript task, N and A may be satisfied while P, D and U are not applicable under its contract. For an illustrative deployment task, N, A and P may be satisfied while D is unknown and U failed under a valid path check. Neither vector is a measured company outcome. A changed candidate cannot inherit an old candidate's acceptance; a changed target configuration may invalidate D or U without erasing the historical publication record. The observer must identify which predicates depend on the changed input rather than resetting every coordinate indiscriminately.

3. What the four-source reconstruction demonstrates

The result is a set of verified record identities with bounded, source-specific meanings. GG/1puni (including subprojects), nsnodes, Boating and OC are four organisational sources, not established independent operators or a comparable pooled outcome sample. Limits of these records are limits on retrospective inference, not evidence of poor organisational performance. Selected counts and normalization limits appear in the public methods summary. Private source records are not released.

We analyze retained selections from four source groups with different recording rules and observation dates. The October 4 tables retain their original cohorts; the October 16 recovery adds separate Boating operating chains and nsnodes handling receipts. We verify selected bytes and record links locally, while attributing extraction coverage, requirement summaries and recording semantics to their source owners. This establishes specific recoverable edges rather than a representative task-outcome sample. The methods supplement records selection, units, normalization and verification boundaries.

The following table preserves the October 4 selection.

Three diagnostic examples

Required evidence depends on task kind. The nsnodes owner export describes 91 read-only light reviews and six ordinary digest tasks at named accepted revisions. All six digest tasks have exact candidate landing links; the 91 reviews have none in that subset. Six links among 97 records is not yield. Requiring a product landing from a read-only review would impose the wrong acceptance condition. Current kind metadata is not proof of unchanged historical obligations. The installed runtime also recognizes publication/ancestry evidence beyond the exported trailers, so an absent trailer does not prove no landing.

Exact content identity does not settle the edge's meaning. Boating's diary reconstruction links 994 of 1,080 references to ancestors of observed main. The owner-inspected writer records current HEAD at report time, including no-change closures, rather than a task-specific accepted candidate. The 86 nonancestor references are not failures. The legacy done directory can also contain retired work; its mutable attempts field increments on claim and can decrement during recovery. No acceptance rate, attempt total or original admission date follows from relabelling those fields. The inventory was observed after the analytic cutoff; old filename dates do not establish archive state at that cutoff.

Examined control flow is not an observed failure. In the examined OC harness source, the owner-examined code puts a merged-tree gate PASS before staging or shipping, and can mark work done despite a subsequent push failure. This constrains what the label alone can certify. It does not prove which version executed a retained record, whether a push failed, or how often completion diverged from publication. Matching ciphertext and source hashes do not supply missing runtime evidence.

Recovered requirements, implementation and operating evidence

The separate October 16 recovery supplies six native push reports corroborated by retained Boating Git. Two worked chains connect immutable requirements, before/after revisions and changed artifacts to those reports. Unlike a diary's report-time HEAD, these records identify specific reported push operations. They recover evidence of useful implementation work without treating incomplete outcome observations as organisational failure.

The weather-overlay requirement asks for an opt-in GFS wind overlay, backend fetching, decoding and caching, forecast controls and fixture-driven tests without live network access. The selected change spans 18 paths and the native result reports a successful main push. A later immutable brief reports a real fetch rendering no wind and says the fixture E2E never exercised that path. The reported push and later defect report can coexist. The latter was not independently reproduced here; it establishes neither a current failure nor violation of the original fixture-only test requirement. An evaluator asking whether the real user path works must observe that predicate at the relevant revision and environment.

The theme requirement includes night-theme repairs, preserved day appearance, screenshot comparisons and TypeScript/Vitest checks. A six-path change and successful-main-push report are recovered; independent visual and user acceptance remain unknown. In both cases a gate invocation is retained, but the native result is tailed and historical code can skip clean-fast-forward re-gating. Neither an invocation nor a push report establishes a fresh gate pass or semantic acceptance. These corroborated historical reports also do not independently satisfy P as defined in §2: containment at the intended remote was not separately observed. Native episode completion, semantic acceptance, deployment and user success remain unestablished as full coordinates. The later weather report is not a reproduced U-failure at a known deployed revision and environment.

A complementary nsnodes result names a different predicate. After deduplicating 822 copies, all 118 completed-result-handling receipts join accepted decisions and their work, including 105 already in the October 4 cohort. Their task-status keys nevertheless distinguish 109 done, seven blocked and two waiting; 67 replies are empty. Completed handling therefore does not mean 118 successful tasks, sent messages or user acceptances. The recovered link establishes that handling occurred; other required outcomes need their own evidence. The methods supplement records selection and verification limits.

GG/1puni supplies the complementary identity warning: 120 of 367 native blob identities occur across task trees. Artifact inclusion is not a distinct episode and cannot allocate its usage to every containing task. Original-key bindings associate 131 of 141 native identities with current tasks; four alias-supported joins raise this to 135. These are retrospective associations observed after cutoff, not historical attempt ownership. Message stop labels are likewise not provider-episode or task-completion events.

The earlier frozen linkage slice selected ten of 14 eligible admissions regardless of outcome and found 10 exact landing links among 11 retained candidates. That supports an artifact edge, not acceptance yield or semantic author approval. Missing execution keys do not imply absent executions, and unmatched records stay unresolved.

Selected implementation examples distinguish preparing a tested candidate from observing publication, and static pages from the credentials and consent needed for a mailbox connection. Source inspection alone proves neither deployed behavior nor a broken connection.

The selected reconstruction combines artifact identities, owner-attributed semantics and unresolved completeness. It supports preserving evidence boundaries. It cannot estimate divergence prevalence, prove better reliability, or rule out near-perfect agreement between native completion and a well-scoped acceptance condition. Descriptive reconstruction needs no new trial; forecast scoring still needs predictions issued before outcomes.

4. Models propose; external machinery controls consequential acceptance

The preserved architectural principle is a division of authority. A model proposes a change; machinery outside that proposal controls whether its consequential acceptance criteria have been met. This is stronger than asking a model to label its own work successful, but it is not a guarantee of correct evaluation. Weak tests, an incorrectly specified target or an overly permissive reviewer can still accept a bad result.

Monperrus's epistemic-agent prototype makes declared tool-call intent available for inspection. Such reasons can help classify a recovery attempt or locate a mistaken assumption. They remain self-report. The effect of a command and the correctness of a delivery must be checked independently. A plausible reason should not convert a mutating action into a read-only action or make an unauthorized result acceptable.

This distinction also disciplines architectural minimalism. The bootstrap demonstration shows that a compact specification can generate a working agent implementation. It does not establish that the implementation includes every responsibility needed to publish or operate work safely. The sufficient system is the smallest one that preserves required behavior, authority and recoverability at acceptable total effort—not the one with the fewest lines.

Cordis, as reviewed on 2 October from Shi, Zhang and Cui's arXiv:2608.25512v1, offers a concrete related distinction: tracked component inverses and dependency activation address lifecycle recovery within a stated boundary. External emissions still need delayed commitment or compensation; dependency mediation is not an external sandbox. We distinguish component recovery from recovery of an undertaking, which additionally requires identifying current authority, completed work and the next justified action. The reviewed Koishi case concerns v3 while the paper presents v4, and is observational; it supplies no controlled estimate of accepted agent completion or reduced operating effort.

5. Proposed study: accepted completion under a budget

The study's target is Pr(A by time t and within resource budget b | task stratum, model, harness, intervention policy). Replace A with a composite requiring D or U only when the task brief actually requires those states. Human-equivalent task duration, as used in METR time horizons, can help characterize difficulty. It is not the same variable as elapsed agent time.

Enroll comparable tasks before outcomes and freeze their acceptance criteria. Record native completion and each required downstream observation. Pin model, tools and harness. Specify what recovery the agent may attempt and when human intervention is allowed. An autonomous cohort and a supported cohort have different estimands; a successful supported result is not retrospectively autonomous success.

At admission, record a probability of accepted completion at prespecified time/resource horizons, plus a fixed task-stratified baseline forecast. The existing protocol has not selected its historical source, sparse-stratum rule, one primary horizon or a multiplicity policy across horizons; those are unresolved pre-study choices, not hidden decisions made here. A digest and timestamp bind the prediction before the first outcome-bearing attempt. Optional updated forecasts can be issued at declared landmarks, but retain the earlier predictions. Score only outcomes known at the corresponding horizon; unresolved observations require bounds or a prespecified censoring method, not confident failure labels.

The methods summary describes terminal competing outcomes, selection records and cohort comparisons. A private synthetic accounting fixture calculates fixed-horizon Brier scores and conservative yield bounds; it does not fit or validate a learning model. Improvement across cohorts must survive matched task difficulty and changes in intervention, tooling and selection. More favorable later assignments are not evidence that an agent learned.

A proposed historical extension could supply separately labeled sandbox tasks once contemporaneous authority and evaluator inputs are selected. Historical outcomes cannot be retrospectively forecast: a valid prediction must precede its replay's first attempt and remain separate from the known historical record. A later source narrative is not evidence that the agent had that information or permission at the original decision boundary.

6. Warnings, recovery and the operator

A warning is necessary when proceeding requires missing authority or information essential to meet the task contract. It is avoidable when the contract already resolves the issue and the agent could take an authorized, feasible next step. Some warnings cannot be judged from available evidence and should remain indeterminate. Fewer warnings is not inherently better: suppressing a necessary clarification may increase unsafe action or wrong delivery.

A premature exit is an agent termination before the acceptance predicate is satisfied when an authorized, feasible next action was available under the remaining budget. An external permission block is not automatically premature, and a provider crash is not an agent choice. Recovery attempts need evidence of action directed at an observed obstruction; repeatedly checking unchanged state is not automatically productive recovery. Reviewers should code these events blind to the study arm where possible and retain disagreement.

Operator intervention includes substantive correction, missing information, credential/configuration work, review and approval. Record active time directly with declared timers or contemporaneous logs. A long gap between messages may be sleep, other work or queue delay. Neither fast agent execution nor a large number of tool calls establishes reduced human effort. The early METR productivity experiment and its later design update are useful counterweights to both universal speedup and universal slowdown claims.

7. Possible counter-results and limitations

A useful negative finding would be that native completion closely predicts the required acceptance predicate in a tightly bounded workflow, making additional instrumentation unnecessary there. Another would be that richer evidence improves auditability but adds enough latency or operator work to reduce accepted yield. Those results should be reported with the same care as improvements. A forecast that fails to beat a task-stratified base rate should remain a descriptive failure, not be reframed as calibrated judgment.

The source examples are selected, and the additional record-linkage slice uses deterministic frozen selection rather than representative sampling. They establish possible evidence gaps and bounded linkage feasibility rather than the frequency of failures. Missing outcome observations do not establish unsuccessful work, and these records do not rank the organisations by performance. The study design has not been executed, and no claim is made that a particular warning policy or controller architecture causes better outcomes. The evaluator can be wrong, user acceptance can be subjective and some external effects are expensive to observe. Observation costs belong in the accounting.

The framework applies to observable work. It neither diagnoses conversational behavior nor equates stored history with understanding. Storage, responsibility and current-undertaking knowledge are separate candidate constructs; testing that distinction needs controlled interruption experiments, not access to personal narratives. Cross-project replication remains contingent on authorization and comparable outcome definitions.

The contribution is a task-specific contract for documenting achieved work. It states which outcomes matter and records the revision, environment and observer supporting each judgement. The recovered chains show what source-specific translation can establish about implementation and handling, while preserving unknown outcomes. This supplies a defined outcome for future evaluation and forecasting; whether the additional observations improve decisions enough to justify their cost remains an empirical question.

Sources and reproducibility

See the public bibliography for cited primary sources and their publication status, and the methods summary for selected observations and proposed studies. Private records and executable research materials are not part of this release. The summaries do not constitute an independent raw-source audit. Martin Monperrus did not personally review or endorse these papers.