Skip to content
Skip to content
Weekly synthesis2026-W42

The Acceptance Pipeline Needs Evidence the Agent Cannot Rewrite

10 min read

What changed our view

This week strengthens last week's acceptance-capacity thesis and adds a harder requirement: the evidence used to accept agent work must be independently produced, typed by failure class, and outside the worker's writable surface. Last week, the central problem was throughput—agents could create work faster than a team could understand, verify, and own it. This week showed why simply adding reviewer models does not repair that queue. Deployment-Integrity Blindness reports that stronger output judges still accepted tampered runtime stacks because the missing fact was identity, not reasoning quality; a manifest-hash channel supplied evidence the output could not.

For one recognizable TypeScript change, the practical distinction is simple. A passing test says the observed behavior met one oracle. It does not say which model, instructions, tools, dependencies, policy, credential issuer, files, or evaluator ran; whether important content was skipped; whether the judge completed; or whether an interrupted external effect committed. Acceptance is therefore not one score. It is a join across independently owned receipts for outcome, coverage, identity, authority, effect, and terminal state.

  1. 01Give every green state a typed receipt.
  2. 02Keep one protected channel outside the worker and judge.
  3. 03Measure controls by reviewer reconstruction.
Jump to section

Scout Weekly — October 12–18, 2026

Evidence

Monday placed deterministic policy at the final effect boundary. One Word Opens the Gate reported that typed model decisions used as guardrails can change sharply with caller-defined option wording, while the MCP OAuth advisory made issuer binding a concrete SDK and credential-migration requirement. The common lesson was not “remove models from safety work.” It was to let models surface risk while application code owns exact caller, issuer, audience, arguments, resource, approval, and effect.

Tuesday moved the same boundary into generated interfaces. OpenAI's Intelligent UI release and CopilotKit/OpenIntelligentUI made model-selected presentation credible, but a React application still has to own authoritative state, permitted components, action binding, retries, and terminal outcomes. Presentation can be generative; business state cannot become true because a generated view says it is.

Wednesday exposed the coverage channel. Alibaba's Open Code Review issue #1650 described prompts skipped for token limits while the scan still appeared successful, and CRBugBench reported bounded recall on verified bugs rather than universal reviewer competence. A useful green result must name the exact revision, eligible and inspected scope, skipped or truncated content, tool failures, model and budget, and whether the terminal state is pass, fail, or unknown.

Thursday applied the same structure to memory. Memory Echo studies repeated exposure from one historical source without independent corroboration, while direct demand asks whether local memory remains correctable and deletable. A memory record needs source lineage, validity, supersession, retrieval reason, proof of use, and deletion. Repeated copies do not become independent evidence.

Friday turned orchestration into a costed control surface. LoopArena holds the coding worker fixed so the controller's allocation, verification, budget, and stopping decisions can be evaluated separately. GitHub's agentic autofix supplies the practical shape: an agent proposes a repair, CodeQL reruns as an independent oracle, and the result remains a draft for review. A planning stage, critic, retry, or artifact earns its control budget only by catching a named failure or reducing reconstruction.

Saturday made the human limit explicit. Orca, ECC, and official plugin catalogs kept attracting attention while an ExperiencedDevs discussion reported review load, context switching, and fatigue under several simultaneous workers. SpecAgent points toward selective allocation: let a cheaper model execute grounded read-only steps, but hold the first state-changing action for stronger verification. The discussion is not representative productivity measurement, and the paper is not a reproduced production result. Together they support measuring accepted work per reviewer-hour instead of launched workers.

Sunday completed the evidence map. The deployment-integrity paper isolates runtime identity; AgentPerturb injects controlled interaction changes and reports that agents often propagate altered observations without recognizing them; INFUSE spreads indirect prompt injection across realistic long-horizon workflow surfaces; and MemScope localizes memory-architecture repairs from traces, runtime state, and code. The direct MCP Test Kitchen record turns one part of that research direction into a small operator fixture for transport, protocol, tool-result, delay, elicitation, and multi-round-trip failures.

Distribution accelerated at the same time. android/skills made official platform guidance a versioned skill dependency at a first clean 355-star/day baseline; cursor/plugins held 388/day; and anthropics/claude-plugins-official held 1,397/day. The counter-signal is concrete: Anthropic issue #6355 reports that a refusal or unparseable security-review response can be recorded as zero vulnerabilities. Publisher identity and curation help admission; they do not prove execution or a valid verdict.

Counter-evidence

Independent evidence channels can become bureaucracy. A manifest proves recorded component identity, not that the component is secure or appropriate. A coverage receipt can be complete while the policy is wrong. A deterministic test can encode a stale requirement. A human reviewer can approve bad work. Stronger judge models can still improve classification, explanation, and fault discovery when they receive relevant evidence. The argument is therefore not that every task needs a large provenance platform or that model review has no value.

The new research is also early. DIB-Bench, AgentPerturb, INFUSE, SpecAgent, MemScope, and the other papers report their authors' evaluations; Scout has not reproduced them. Vendor-authored material such as AWS's production evaluation framework is useful architecture guidance, not independent proof that the proposed stages improve every workload. The appropriate response is a small, local acceptance contract with a few distinct receipts—not an immediate internal platform build.

Package and traffic signals show that these systems are no longer hypothetical. NuGet reported about 42.1 million downloads for ModelContextProtocol and about 10.4 million for Microsoft.Agents.AI when observed; PostHog publishes measured calls from its own MCP server. Those numbers establish distribution and traffic, not unique production users, secure configurations, or accepted outcomes. They make narrow failure testing more urgent without making a universal evaluation stack proportionate.

What Matters to Us

  1. Give every green state a typed receipt. A code review, memory lookup, tool call, generated UI action, and deployment evaluation should expose the evidence particular to its failure class. Missing identity is not a behavioral failure; skipped scope is not zero findings; unknown external state is not a retryable error by default.

  2. Keep one protected channel outside the worker and judge. For a normal app task, that can be a lockfile-and-policy manifest, a protected integration test, an issuer-and-audience check, a Postgres invariant, or a human approval token bound to exact arguments. It need not be elaborate, but the authoring agent must not be able to rewrite both the work and the evidence that accepts it.

  3. Measure controls by reviewer reconstruction. Record whether a receipt lets a reviewer answer what ran, what changed, what was inspected, what failed, what external state exists, and why the terminal decision followed. Add a planner, critic, or worker only if it catches a named failure or lowers reviewer minutes without weakening the protected result.

  4. Treat skills and plugins like dependency code. Pin publisher and immutable revision, inspect permissions and external endpoints, run a task-specific canary after install and update, distinguish failure from clean completion, and rehearse removal. First-party provenance narrows the admission problem; it does not eliminate runtime verification.

One Experiment Worth Running

Build one three-receipt acceptance fixture. Start with a read-only MCP-backed TypeScript task whose correct result can be asserted. Produce three records outside the agent workspace: (1) a runtime manifest naming model, instructions, tools, dependencies, and policy; (2) a tool-attempt record naming exact input, observed output, retry decision, and terminal state; and (3) an evaluator record naming revision, inspected scope, skipped content, protected result, and pass, fail, or unknown.

Run the normal case, then change one runtime component while preserving plausible output, return one malformed or delayed tool result, and make the evaluator refuse or emit invalid JSON. The fixture succeeds only if those cases become identity mismatch, tool uncertainty, and evaluation unknown. This can establish that the channels stay distinct on one task; it cannot establish general agent safety. Consider a larger suite only if the records reduce reconstruction time and remain outside the worker's writable surface.

One Thing to Ignore

A complete catalog, methodology, or agent fleet before one control proves its value. Android Skills, Cursor plugins, Anthropic plugins, ECC, AI-DLC, and Orca contain useful pieces, but artifact count, stage count, and worker count are supply metrics. Ignore whole-catalog installation and broad process adoption. Revisit one pinned behavior when a current task names the failure it should catch, the authority it needs, the protected behavior it must preserve, and the evidence that will prove completion and rollback.

Also keep large off-lens movers—VoiceStudio, creative tools, Android packages, and media clients—outside the active roadmap without a corresponding workload. Attention preserves a signal; it does not create demand.

Supply vs. Demand

Supply now covers nearly every visible layer: model guardrails, code reviewers, memory stores, generated UI, orchestrators, fleet supervisors, skill and plugin catalogs, MCP SDKs, security benchmarks, fault servers, durable sessions, and mobile device agents. The newer supply is increasingly versioned and instrumented, which is real progress.

Observed demand is more specific. Practitioners ask how to trust generated production code, keep several workers from overwhelming review, prove that a security check actually ran, recover from a bad tool interaction, preserve source lineage, and know which runtime handled credentials or effects. The match is improving at the primitive level—manifest identity, protected oracles, fault injection, issuer binding, and coverage receipts are all concrete—but the application team still has to join them into one small acceptance record. The reviewed discussions are attributed reports, not representative market demand or search-volume evidence.

People to Watch

Category Shifts

Category This Week Last Week Direction
LLM eval / testing Runtime identity, coverage, perturbation, indirect injection, and evaluator completion split into independent checks Acceptance pipeline and review capacity became the central delivery constraint ↑ Acceptance becomes a join across typed evidence channels
Code dev tools Plugin ecosystems, control budgets, generated UI state, and worker concurrency Review pipelines, repository understanding, and convergence limits ↑ Packaged workflow supply grows; proof of execution remains scarce
MCP tooling Issuer migration, deliberate bad servers, package distribution, and real tool-call traffic Exact pre-effect policy, audience checks, and lifecycle controls ↑ Protocol adoption is now large enough to justify adversarial integration tests
Skills ecosystem Android joins first-party versioned distribution; fail-open plugin behavior becomes visible Add-ons became executable dependencies with lifecycle canaries ↑ Provenance improves, but completion and rollback need independent evidence
Memory / RAG Source lineage, correction, retrieval integrity, and localized repair Evidence transformation and retention-versus-retrieval ↑ Memory quality becomes provenance plus bounded change
Web / UI agents Generated UI state contracts and Android device automation Browser and UI effects inside the acceptance pipeline ↑ Presentation and device control gain explicit state and teardown boundaries
Agent infrastructure Control budgets, selective verification, and acceptance-queue limits Durable recovery and effect uncertainty ↑ Runtime cost and human capacity join identity and effect safety

Open Questions

  • What is the smallest independent runtime manifest that proves useful identity without turning every local agent task into an attestation project?
  • Can unknown remain visible across SDK, plugin, review, CI, and deployment layers, or will one compatibility adapter inevitably translate it into success?