The Agent Stack Became the Evaluation Unit¶
What changed our view¶
This week, the useful unit of comparison expanded from a model or tool to the complete agent stack: model, harness, runtime contract, retained state, client admission, authority, and acceptance evidence. Finding the Right Fit made the interaction measurable by reporting ranking reversals across 66 model-harness configurations, while MCP Events, Figma's hosted MCP policy, and OpenShell 0.1.2 made lifecycle, admission, and runtime policy concrete product surfaces. This strengthens last week's view that memory belongs inside the control plane, but changes the comparison rule: persistent state cannot be evaluated independently of the harness that writes, selects, authorizes, and retires it.
- 01Name the whole tested configuration.
- 02Separate selection from authority.
- 03Put lifecycle state in ordinary durable data.
Scout Weekly — September 28–October 4, 2026
Evidence¶
Monday and Tuesday moved multi-agent work away from worker count and toward evidence custody, semantic coordination, and review. OpenRig accelerated while RankEvolve composed complete Claude Code and Codex products as review-and-repair nodes. TUIOS made session persistence and a shared completion inbox visible. The practical question was not whether several workers could run, but whether disagreement, process ownership, accepted output, and cleanup could be reconstructed by a reviewer.
Wednesday and Thursday made runtime assumptions explicit. Evaluating Agents Across Runtime Contracts treated persistent versus stateless execution as part of learned behavior, while NVIDIA/OpenShell exposed filesystem, process, credential, and network policy as runtime configuration. Finding the Right Fit then showed the consequence: model rankings can reverse when the harness changes. A benchmark, internal bakeoff, or procurement claim that names only the model is therefore incomplete.
Memory research tightened the same boundary. What Should an Agent Remember? separates retention from retrieval and shows how a mixed comparison can attribute an access difference to selection quality. Just-In-Time Agent Memory preserves raw history behind a navigable store and constructs evidence for each request at runtime. These papers extend last week's memory thesis: “has memory” is not a stable system description unless the write, eviction, access, selection, freshness, and authorization paths are fixed.
The protocol layer supplied a concrete lifecycle. MCP Events specifies signed webhook delivery, retries, replay, expiry, refresh, unsubscribe, and access-loss behavior. Pi 0.99.2 makes broad tool discovery lazy, while Figma's catalog requirement demonstrates that protocol implementation does not guarantee hosted-service admission. The reviewed MySQL MCP advisory provides the hard limit: a valid protocol surface with weak transport checks and database authority can expose unauthenticated SQL execution.
Direct demand remains operational. A hydrated EngineeringManagers report describes review queues, incidents, and unclear returns; a LocalLLaMA sandbox thread asks for one-command, one-folder isolation. These are attributed practitioner reports rather than representative market measurements, but they point at the same missing product: simple boundaries with recoverable human understanding.
Counter-evidence¶
The conclusion should not become an excuse to build an internal agent platform. Official SDKs, .NET package distribution, managed hosted servers, OpenShell, coding-agent clients, and ordinary job infrastructure can remove substantial work. A Coder customer story attributes a governed multi-tenant deployment to more than 2,500 developers; it is vendor-authored and describes agent-specific use as future exploration, but it demonstrates that centralized development-environment policy is not limited to lab prototypes. A small team should buy or reuse commodity transport, sandboxing, OAuth, queues, and tracing when those pieces meet the workload.
The evidence also does not rank the available stacks. GitHub stars measure attention, package downloads measure distribution, vendor documentation describes supplied behavior, Reddit posts report individual experience, and this run did not reproduce the arXiv results. The defensible conclusion is about evaluation scope, not a winning harness, memory system, protocol client, or runtime.
What Matters to Us¶
-
Name the whole tested configuration. Every model comparison should record the harness revision, tool schemas, state mode, permissions, credentials, retries, timeout, acceptance set, and reviewer budget. If any of those change, the result belongs to a different system.
-
Separate selection from authority. Deferred tool search, memory retrieval, and model routing answer “what looks relevant?” Application policy answers “may this caller perform this effect on this target now?” Keep the two decisions independently testable.
-
Put lifecycle state in ordinary durable data. Subscriptions, task identity, replay cursors, retries, grants, expiry, cancellation, supersession, and terminal acceptance should be queryable and revocable through application-owned records, not inferred from a chat transcript.
-
Optimize accepted work per reviewer-hour. Worker count, generated code, tool calls, context reduction, and benchmark score are intermediate measures. Include reviewer reconstruction, protected-test failures, retries, recovery, cleanup, and total spend before calling a stack faster.
One Experiment Worth Running¶
Compare one model across two pinned harness contracts. Choose one reversible TypeScript bug with a protected test and short reviewer rubric. Run the same model once in the current coding harness and once in an inspectable alternative, holding repository state, prompt, allowed tools, persistence, credentials, retries, timeout, and budget fixed. Record accepted result, changed lines, tool calls, reviewer corrections, time, cost, and cleanup. This can reveal a material harness interaction for one workload; only a meaningful difference should justify a 2×2 model/harness test.
One Thing to Ignore¶
Protocol, skill-pack, or worker-count popularity as a proxy for workload value. MCP compatibility, a trending instruction bundle, and a multi-agent dashboard can all be useful, but none proves safe authority, maintainable output, lower review cost, or recoverable state. Revisit one candidate when it can be pinned and one representative task produces a better accepted outcome under the same contract.
Supply vs. Demand¶
Supply is converging on complete stack surfaces: persistent multi-agent harnesses, deferred tool search, hosted MCP catalogs, event subscriptions, policy runtimes, shared memory, design-language skills, and telemetry consoles. Observed demand is narrower: confine one agent to one folder, prevent a review queue, know which tool and account acted, preserve current project intent, revoke stale access, and reconstruct an accepted result.
The match remains partial. Vendors and maintainers are shipping more lifecycle primitives, but the operator still has to bind them into one workload contract and measure human acceptance. The demand evidence is direct but limited to the reviewed discussions; it should guide experiments, not be generalized as market prevalence.
People to Watch¶
- Michael Schwarz / OpenRig — heterogeneous coding-agent coordination is now a useful harness fixture. Watch accepted work, conflicting advice, trust mutations, and reviewer time under a pinned contract.
- Juli Huang — the retention-versus-retrieval distinction gives memory evaluations a cleaner failure taxonomy.
- Yixuan Li and collaborators — the released model-harness trajectories make interaction effects inspectable rather than anecdotal.
- NVIDIA/OpenShell — current releases turn runtime filesystem, process, network, and credential policy into an installable surface. Watch alpha stability and workload-level evidence.
- Pi maintainers — deferred server connection and tool search make context and discovery policy explicit parts of the harness.
Category Shifts¶
| Category | This Week | Last Week | Direction |
|---|---|---|---|
| Code dev tools | Model-harness interaction, owner/checker teams, design-language and process skills, reviewer reconstruction | Typed workflows, declared intent, and per-session controls | ↑ Complete configuration replaces model-only comparison |
| MCP tooling | Deferred discovery, hosted-client admission, signed event lifecycle, schema drift, and a database advisory | Broad reach, scoped keys, mobile effects, and weak-authority warnings | ↑ Connectivity becomes governed lifecycle |
| Agent infrastructure | Installable runtime policy and explicit execution contracts | Persistent shells, managed workspaces, teardown, and deletion | ↑ Runtime assumptions become test inputs |
| Memory / RAG | Retention separated from retrieval; query-time construction becomes a research path | Shared banks, freshness, scope, and deletion joined the control plane | ↑ Memory is evaluated as a read/write policy |
| LLM eval / testing | Model, harness, task, state, and acceptance treated as one unit | Replay and frozen-system evaluation widened beyond answer scores | ↑ Interaction effects become measurable |
| UI generation | Installable design-language behavior held attention while HTML-to-video cooled | Artifact generation remained secondary to governance | → Useful abstractions, limited outcome evidence |
Open Questions¶
- What is the smallest portable receipt that names the model, harness, tool schemas, selected evidence, caller, credentials, effects, retries, reviewer decision, and terminal state without copying sensitive content?
- Can deferred tool and memory selection reduce context cost without hiding a dependency, stale fact, or authorization-relevant signal that changes the accepted result?