Daniel Arcé

Project history is worth supplying to a coding session when it helps the session complete the task. It may prevent a repeated investigation or bring an important constraint back into view. It may also distract the agent or revive a decision that no longer applies.

A task-relevant context packet is useful if it reduces total token use without lowering outcome quality. Fewer tokens alone would not establish a useful result if the agent completed less of the task or introduced more errors.

The audit gives a starting point for that test: the recorded size and contents of context packets. It cannot establish their effect on task outcomes.

The audit snapshot describes cost and contents

My summary of an earlier audit describes three project handoff.db databases. The audit date and reproducible extract are unavailable here. A token is a unit of text processed by a model; task totals can include many requests and retries. Its summary reported small initial context packets alongside much larger recorded task totals. It also found that most findings marked resolved had a written resolution.

These figures are unverified, author-reported aggregates from that audit. The summary does not provide the underlying rows or per-measure sample sizes, so the figures should not be treated as a general performance benchmark.

Dividing those two summary figures gives 300 / 300,000 = 0.001, or 0.1%. This compares a mean packet size with a median task total across different observations; it is neither a matched-task overhead estimate nor a savings measurement. Retrieval, later context use, and decisions made from stale material can add costs beyond that initial packet. These token aggregates do not establish a monetary return.

The audit cannot establish an effect

The audit’s main limitation is an instrumentation mismatch. Interactive sessions received context packets but lacked the audited task-level token measurement used in the analysis. Orchestrated worker lanes supplied token measurements but did not receive those packets.

That mismatch prevents a meaningful treated-versus-untreated comparison. The two groups differ in their work and measurement as well as their use of project history. A controlled comparison must record both the intervention and its cost for the same assigned tasks.

The resolution figures have a similar boundary. A written resolution establishes that material exists to retrieve. It does not establish that retrieval selects the right resolution, or that an agent uses it well. Those effects belong in the comparison.

The comparison tests whether the packet changes the work

The proposed comparison runs each selected task twice from the same base commit: once with a frozen, task-relevant packet and once without that packet. Both runs receive the same task specification, model configuration, and resource limits. Their order is randomised, and neither updates shared memory until the pair is complete.

Calibration uses A/A pairs: two cold runs with the same starting conditions. The variation between those attempts estimates how much results can differ without the memory intervention. That variation informs the sample-size procedure for the main comparison.

The content probe compares a relevant packet with an equally sized unrelated packet. This tests whether relevant content makes a difference beyond merely adding text. A relevant-versus-unrelated result would not, by itself, establish a benefit over receiving no packet.

Matched task pairs provide the no-packet comparison. Each pair contains one run with the relevant packet and one without it. A reviewer who does not know which condition produced the work scores both results against a fixed rubric. The system records total tokens per assigned task, including unsuccessful attempts. The protocol must count packet preparation, embedding and retrieval, repeated context input, output, retries, and review effort consistently. For money rather than tokens, each category needs its applicable rate; cached input and output may cost different amounts. This article supplies no provider-price forecast.

Keeping unsuccessful attempts in the analysis matters because a quick failure can use fewer tokens than a completed task. The quality assessment and failure rate must therefore accompany the token result.

The decision criterion includes quality

The proposal uses a quality margin: the largest acceptable deterioration on a predefined review rubric. It sets a target of at least a 20% reduction in token use, with blinded quality kept within an acceptable margin declared before the trial. This is a proposed decision criterion; no controlled savings result has been measured.

The readout should report the paired effect with an uncertainty interval, alongside the quality comparison and failure rate. A reduction in tokens would qualify only if the quality condition also holds. For each task, compare the memory-run total with the cold-run total before aggregating; report absolute totals as well as ratios, since a small-task percentage can hide a large-task cost. An inconclusive interval would remain inconclusive rather than being presented as a savings result.

The initial comparison would concern one repository’s workload. A result there would not establish the same effect for other projects, models, or teams without further evaluation.

Specify the measurement before running it

The protocol still needs its operational definitions completed. The interactive task meter and packet-selection rule must be recorded and frozen. The quality rubric, acceptable quality margin, and sample-size procedure (how many task pairs the uncertainty target requires) must be specified before the main comparison.

Those definitions determine what the result will mean. Changing them after inspecting the results could change which runs count as successful or which apparent saving meets the target. Fixing them first makes the eventual decision checkable.

This is a proposed evaluation, not a completed or externally registered study. Canon v0.25.6 supplies review prompts, not empirical validation: UXR-01 Match claim to validation level (Visualization Analysis and Design (Munzner); Spatial Semiotics and Spatial Mental Models, ch. 3) keeps the method aligned with the claim. EXP-01 Base-rate idea failure (Trustworthy Online Controlled Experiments, ch. 1) asks for a controlled comparison or an explicitly lower-trust method.

The decision in the meantime

I use WorkBay to retain inspectable project history and resume work across coding sessions. Until the comparison is run, I can assess the recorded history and retrieval behaviour, but cannot claim measured effort savings.

Project Memory Outside the Conversation describes that implementation. Five Questions I Ask the Project Record describes how I use it in practice.