When Does Context Earn Its Keep?

Abstract

Which context preparation is worth paying for when a coding agent must deliver useful accepted work? We propose comparing strategies by the total cost of preparation and all attempts per independently accepted task, together with acceptance yield. We reconstruct selected records from four source groups using source-specific definitions and a lossless Boating adapter. Two Boating cases connect immutable requirements and implementation changes to native push reports corroborated by retained Git; nsnodes receipts connect completed result handling to accepted decisions. These recovered edges show how existing operating records can support evaluation. Repeated usage snapshots and shared session identities also show why those edges alone cannot allocate complete task costs. We specify a task-blocked context comparison that can test whether preparation earns its cost. The present contribution is the accounting method and bounded reconstruction, not an observed context benefit or a comparison of organisational performance.

1. Introduction

A cache hit tells us that input was reused. It does not tell us whether that input helped complete the task. Before comparing context strategies, we need to know which input, preparation and retries belong to which task, and what counted as acceptance.

Selected records from GG/1puni, nsnodes, Boating and OC expose concrete obstacles to that measurement: repeated counters, inherited artifacts and lifecycle labels whose meanings differ across sources. These are four organisational sources, not four established independent operators. GG/1puni includes its subprojects as one organisation. Our demonstrated contribution includes recovered links from requirements to implementation artifacts and historical push reports, together with a method for preserving their source meaning and unresolved resource attribution. Our proposed study asks which context strategies deliver the required accepted work at a cost worth paying.

Repository instructions, file reads, tool results and earlier responses can return as input on later calls. Dark Tokens and the local measurement essay motivate observing that input. Establishing its task join and coverage comes before interpreting its cost or benefit.

Two easy conclusions should be resisted. A large input total does not establish waste: relevant information may prevent expensive search or a failed implementation. A large cached-input share does not establish efficiency either: cached irrelevant history can still consume resources and attention, and the resulting task can still fail. The right comparison is between complete strategies for delivering the same accepted outcome.

This is not a claim that cost-aware agent evaluation is new. Kapoor and colleagues already argue for joint attention to cost and accuracy. Our narrower proposed contribution integrates preparation allocation, provider-specific usage normalization, cross-session attempt identity and independent task acceptance in one context comparison. Sections 2–3 of that work already examine retry baselines and fixed versus variable costs; neither cost-aware evaluation nor charging preparation is new here. The proposed integration addresses work that crosses native sessions and controller retries, where the successful final response alone understates the resources spent.

The strongest objection is that this is ordinary cost-effectiveness analysis with a new label. The ratio alone warrants that objection. The methodological contribution is the join that makes it interpretable: preparation, unsuccessful attempts and successful responses belong to the same enrolled task, whose acceptance is judged independently. Without that join, a cached-token percentage and a successful final answer can coexist with unknown total cost and unknown yield. Prepared relevant context may improve this outcome, but that is a hypothesis to test, not a conclusion from cache accounting.

2. The accounting object is a task, not a response

Define a task before execution: an immutable brief, source revision, authorized scope, evaluator and acceptance rubric. A task may require several attempts. An attempt is a bounded execution episode; retries after an unsuccessful episode remain attached to the original task. Responses are billing observations within attempts, not independent successes. A session can contain multiple tasks, and a task can span multiple sessions, so session counts cannot replace task counts.

For response r, let U be ordinary input, R cache-read input, W cache-write input and O billable output. These are disjoint billing categories after provider-specific normalization. With per-million rates p, response expense is:

Write durations or context bands requiring separate rates get separate categories/rate cards. Add explicit storage, tool and other metered charges. Report human minutes separately unless a prespecified valuation is supplied; do not disguise an assumed hourly wage as observed expenditure. Reasoning counts already included in O are not charged twice. Unknown cost is retained as unknown.

For arm a with N enrolled tasks and A accepted tasks by the declared horizon:

If A is zero, the cost-per-accepted estimate is undefined, not zero. Show the spend and zero yield. If observation is incomplete, show cost to date and unresolved outcomes without promoting this ratio to a final comparison. The denominator rewards useful delivery but can conceal selection: an arm assigned only easy tasks may look better. Randomized common task blocks and reporting yield together with cost address this problem more directly than dividing only successful-run expense by successes.

A strategy could accept fewer tasks cheaply and obtain a favorable ratio. Therefore a recommendation to replace another strategy also needs a prespecified yield margin. In the proposed confirmatory decision rule, a cheaper strategy must meet the acceptance-yield noninferiority criterion as well as the cost criterion; otherwise it is a tradeoff to report, not an unconditional improvement.

Table 1. Task accounting join

Proposed collection contract. This is a schema, not a populated observational dataset; the October 1 usage aggregate cannot fill these joins.

Consider an invented comparison on ten matched tasks per strategy. Minimal context costs 100 units across all attempts and yields eight independently accepted tasks: 12.50 units per accepted task, with 80% yield. Prepared context costs 60 units across all attempts, plus 60 units of preparation allocated to those tasks, and yields nine accepted tasks: 13.33 units per accepted task, with 90% yield. Omitting preparation would make its cost appear to be 6.67 units per accepted task. Including preparation changes the interpretation: the prepared strategy delivers more accepted work at a higher cost per accepted task. That is a tradeoff to judge against the declared task requirements, not a demonstrated efficiency improvement. These numbers illustrate the accounting only.

3. Replay grows quadratically under a simple baseline

Suppose each of n calls contains a fixed base B and another d tokens from each earlier round. Then call k contains B + (k-1)d, so cumulative input is:

With fixed B and d, this is quadratic, not exponential. This derivation counts presented input; it does not price it. If a stable prefix is repeatedly cached, the same token sequence may incur a write charge once and a discounted read charge later. If a timestamp changes near the beginning, that reuse may vanish. Compaction can reduce future input and simultaneously incur summarization cost, lose useful information or change the reusable prefix.

The fixed-growth assumption is deliberately transparent. Real traces may have variable tool output, truncation, branching and context limits. We should calculate observed per-response categories when available and use this formula as a sanity check, not fit every session to it. Nor should quadratic presented volume be confused with a claim about transformer compute complexity or provider profit.

4. Establish identity and coverage before estimating cost

Repeated reports and inherited artifacts can double-count work before a price is ever applied. The retained reconstruction therefore distinguishes serialized appearances, event identities, counter increments, message snapshots and task-owned attempts. A hash resolves one kind of identity; it does not resolve all of them.

The selected observations below have unequal retention and observation endpoints. They are neither a continuous four-source census nor comparable pooled task outcomes. The public methods summary describes normalization and limits; private source records are not released.

We analyze retained selections from four source groups with different recording rules and observation dates. The October 4 tables retain their original cohorts; the October 16 recovery adds separate Boating operating chains and nsnodes handling receipts. We verify selected bytes and record links locally, while attributing extraction coverage, requirement summaries and recording semantics to their source owners. This establishes specific recoverable edges rather than a representative task-outcome sample. The methods supplement records selection, units, normalization and verification boundaries.

The following table preserves the October 4 selection; the recovery below has separate units and denominators.

Three diagnostic examples

Artifact inheritance is not repeated execution. In GG/1puni, 120 of 367 native blob identities occur in more than one task's candidate trees. Charging each inclusion to its containing task would duplicate evidence. The selected usage export also contains 47,857 serialized appearances but only 7,239 event hashes. Event deduplication still leaves repeated cumulative reports: 46 are removed before reconstructing 4,909 Codex counter observations. Four mismatches remain excluded and unresolved; 159 decreasing-counter/equal-last cases use a labelled reset-like fallback, not a proven reset. The 1,136 latest message usage snapshots are a different native unit, not additional verified responses.

A precise link can have the wrong meaning. Boating's 1,080 matched diary records resolve 465 unique commits; 994 references are ancestors of the observed product main. The owner-inspected writer records current HEAD at report time, which no-change closures can reuse. These are report-to-snapshot links, not accepted-task edges. The other 86 references are not failed tasks. Neither that series nor its monthly counts supplies the accepted-task denominator.

Repeated counters survive a different source too. OC normalization removes 61 repeated cumulative reports from 895 events to obtain 834 selected counter observations. Those observations support a partial resource series, without identifying complete task costs. The examined harness can label work done even if a subsequent push fails; this is a source-code possibility, not evidence that such a failure occurred. Pricing the counters would not repair either missing join.

The separate October 1 supplied aggregate remains an illustration of a different limit: 634 unique response-usage records from 112 accessible sessions contain 18,920,180 input tokens, including 15,394,688 cached (81.37%) and 3,525,492 noncached, plus 84,460 output tokens including 4,490 reasoning tokens. Only seven responses exceed 64k input, all in one session; maximum input is 80,210. The supplied arithmetic is checkable, but the source extraction was not repeated here. The noncached ordinary/write split, billing, preparation and outcomes are unknown. Cached share is not savings.

A separate frozen linkage slice selects the first ten of 14 eligible company documentation repository admissions by admission time and full task ID in the half-open interval from 2 October 2026, 00:00 UTC to 3 October 2026, 13:14 UTC, regardless of status. It links 10 of 11 retained candidates to exact landings. 10/11 is record linkage, not acceptance yield. Observation at October 3 13:29:02 UTC was retrospective; surviving refs lack an independent deletion/archive audit. No task, response, cost or outcome join to the October 1 aggregate is established. Neither source supplies complete attempt or acceptance records.

The selected summaries distinguish reconstructed identities from owner-attributed semantics and unresolved completeness. Unknown expense stays unknown; reasoning already included in output is never charged twice. These findings justify a measurement contract, not a conclusion that prepared context helps.

Recovered operating chains clarify what must be costed

The October 16 recovery connects useful implementation work to its requirements. Six distinct Boating native push reports are corroborated by retained Git; two cases connect immutable requirements, changed artifacts and reports of successful main pushes. The weather-overlay case spans backend fetching, decoding and caching, frontend controls and fixture tests. A later immutable brief reports that real fetching rendered no wind and that fixture E2E tests had not exercised that path. This historical report was not independently reproduced and does not establish a violation of the original fixture-only requirement. It identifies an operating outcome whose observation and repair may belong in the cost of a task requiring real fetching. The theme case similarly recovers implementation and push evidence while leaving visual and user acceptance unknown. Local Git corroboration and native push reports do not independently establish historical containment at the intended remote.

The early Boating selection contains 1,007 usage snapshots across 361 message identities and 67 loop starts. These are not task-attempt counts or exclusive costs for either case. Charging every message in a session that mentions a branch to that task would manufacture the required resource link. A current done record and a quarantined duplicate also share one session pointer; preserving that identity avoids inventing two executions. A requested gate command supplies no fresh pass receipt where historical code can skip re-gating clean fast-forwards and the retained result is tailed.

In nsnodes, 822 receipt copies reduce to 118 completed-result-handling receipts joined to accepted decisions and their work. The underlying 74 native blobs and 3,200 serialized usage events are unchanged. Twelve additional task references are procedures. More task and handling records therefore do not establish more native work or accepted-task cost. Source-specific translation recovers useful evidence without completing the economic join. The methods supplement distinguishes the cohorts and verification boundaries.

5. Why a crossed study is needed

Context-file studies report conflicting outcomes: Gloaguen et al. provide negative evidence for broadly generated instructions; Lulla et al. report efficiency gains in selected repositories; Khatri reports a bounded null result. Their task selections, context interventions and outcome definitions differ. We should not average those headlines into a claim about “context” as a single variable.

Our three arms separate mechanisms. Prepared context supplies a bounded, pre-task packet of relevant project evidence and acceptance constraints. Minimal context supplies the same brief and essential authority constraints, with repository search available. Growing history preloads the chronological record of authorized predecessor stages, subject to a declared context cap. The primary contrast is initial context assembly: all arms then accumulate target-stage messages identically. Vary predecessor-history depth in prespecified strata, and make the same source artifacts retrievable in every arm. All three receive identical tools and permissions. Preparation uses only starting materials, never the target-stage gold patch or outcome. Retrieval expense is measured, not hidden.

Each arm is crossed with cold and deliberately warmed conditions where the provider permits a defensible manipulation. Warming cost is counted. A cache hit is verified from returned usage; elapsed waiting or a fresh session name is insufficient proof of coldness. Where cache isolation cannot be controlled, the study reports an observational cache stratum and does not call it randomized. Model, reasoning setting, task difficulty, compaction policy and retry allowance are fixed within blocks or explicitly varied in separate blocks.

Cache-strategy experiments motivate this separation but do not answer our accepted-task question. Their savings against an intentionally cache-broken baseline should not be treated as savings against a normally operating minimal-context agent.

6. Proposed analysis and reportable outcomes

The methods summary describes task blocking, acceptance, censoring and decision criteria. Begin with an instrumentation pilot, then determine confirmatory sample size from task-level variance and a declared smallest useful effect. The proposed pilot has 24 independent task snapshots. Each supplies 12 task-condition trial slots (three context arms × two cache conditions × two repetitions), for 288 slots; each trial can contain an initial attempt and at most two recovery attempts. Repetitions and responses do not increase the independent task sample size. Each condition is reported with its own trial denominator; the task snapshot remains the resampling block. Analyze paired task blocks; bootstrap entire task blocks, not response rows. Hold out task families to test transfer and publish all exclusions.

The primary display is a cost/yield frontier, with preparation and failure costs visible. Report cold and warm estimates separately before any weighted mixture; the mixture needs a stated deployment frequency. Report preparation amortized over actual reused tasks and sensitivity to one-time versus repeated reuse. A reusable packet is not free because an analyst produced it before timing began. The protocol distinguishes allocation across actual task reuse from allocation among repeated condition trials, so one preparation record is charged once. The five-percentage-point yield margin has no demonstrated practical justification here. Its justification, allocation weights and the treatment of undefined bootstrap ratios must be frozen before confirmatory use; this is a proposed design, not a preregistration.

A useful negative result would be that preparation reduces retrieval calls but costs more overall, or that it improves test passing while worsening independent acceptance. A precise equivalence result would also matter. An underpowered null is inconclusive. A context strategy can be worthwhile for one task class and harmful for another; that conditional finding is more actionable than a universal slogan.

A proposed historical extension would compare context strategies in reconstructed sandboxes. Only contemporaneously available evidence and authority may enter each packet; later outcomes remain sealed from preparation and execution. Costs of reconstruction and packet preparation remain visible. Such a replay estimates a sandbox contrast, not what the historical agent would actually have done. The existing candidate snapshot supplies neither an enrolled task nor the missing acceptance/billing joins.

7. Limitations and scope

There is no randomized evidence from this company in this edition. The October 1 usage cohort lacks the joins required by our estimand; the separate October 2–3 record-linkage slice does not repair those missing joins. Provider documentation is mutable and native clients may expose different usage fields; current tariffs must not be projected backward without validation. Subscription accounting may not track marginal API prices. Independent acceptance can itself be noisy, so rubrics and evaluator disagreement must be retained.

This design studies observable behavior and accepted artifacts. It does not measure internal understanding, hidden reasoning quality, operator mental state or provider motives. Large-context QA results such as Lost in the Middle justify position controls, not assumptions about the current models. Cross-project generalization requires comparable tasks and authorized evidence that are not yet established.

The contribution is a way to connect context expenditure to useful accepted work. The reconstruction recovers requirement-to-artifact and historical push-report links, while preserving source meanings and unresolved resource attribution. The proposed comparison links preparation and all attempts to independently evaluated tasks, reporting cost with acceptance yield and separating cache conditions. This makes whether context earns its keep testable; the present records do not yet answer which strategy does so.

Sources and reproducibility

See the public bibliography for cited primary sources and their publication status, and the methods summary for selected observations and proposed studies. Private records and executable research materials are not part of this release. The summaries do not constitute an independent raw-source audit. Martin Monperrus did not personally review or endorse these papers.