Agent Work Is Splitting Into an Execution Plane and an Evidence Plane¶
What changed our view¶
This week strengthens last week's independent-evidence thesis and adds the missing counterpart: accepted agent work needs both an execution plane that constrains authority and an evidence plane that records enough distinct facts to justify the terminal decision. Last week argued that outcome, coverage, identity, authority, effect, and evaluator completion cannot collapse into one score. This week showed where those facts can live. OpenShell v0.1.5 makes isolation a versioned backend contract; Cloudflare's security-audit skill makes coverage and independent finding verification explicit artifacts; Meta's thin orchestrator keeps routing and reliability checks deterministic while specialist agents reason.
For one ordinary repository task, the execution plane should decide which files, hosts, credentials, processes, budget, and effects are possible. The evidence plane should record the exact runtime and policy, eligible and inspected scope, attempted and observed effects, protected result, evaluator completion, and final owner. Neither plane replaces the other. A sandbox can contain an incorrect task, while a strong test can approve output produced through excessive authority.
- 01Put hard authority below the worker.
- 02Keep acceptance evidence outside the worker and diagnostic judge.
- 03Make orchestration boring on purpose.
Scout Weekly — October 19–25, 2026
Evidence¶
Monday made installed behavior look like ordinary dependency code. Official plugin and skill catalogs were gaining distribution, but Anthropic issue #6355 supplied a concrete fail-open shape: refusal or unparseable output could be stored as zero vulnerabilities. Publisher identity and successful installation narrow admission; they do not prove invocation, coverage, a valid verdict, or rollback.
Tuesday moved the control plane from worker management into evidence. AuditKIT separates verified outcomes from trajectory diagnostics and leaves missing evidence unscored, while Orca's orchestration roadmap distinguishes idle, exited, unreachable, settled, and successful workers. A trace judge can propose a failure location; protected application state or an accountable human still owns acceptance.
Wednesday applied the same separation to memory. Local and inspectable storage is useful, but a practitioner incident showed why delegated principal identity and observed effects cannot collapse into the user's identity, and a DigitalOcean tutorial recorded a session marked succeeded without a branch or pull request. Memory, audit, and terminal state all need abstention when their decisive evidence is missing.
Thursday and Friday made proof part of the workflow. Cloudflare's skill separates reconnaissance, coverage-led hunting, candidate validation, structured findings, and independent record checking. CoSec evaluates privacy and authorization across changing communities, memories, files, and tools; OPRD studies whether agents act on constraints and feedback already present in long trajectories. Source coverage, authorization, process adherence, and protected outcome are different failure classes.
Saturday changed the release unit. Can LLMs Improve Their Own Harness? makes harness construction and revision an evaluated task, while Compositional Safety Failures in Harness Evolution reports unsafe combinations among memory, prompt, skill, and tool updates that looked acceptable individually. Once the controller can rewrite the context and tests that judge it, the complete manifest and protected oracle have to live outside that evolving surface.
Sunday supplied the lower boundary. OpenShell's current release publishes a Sandbox Protocol and documents DNS, Kubernetes metadata, persistence, and observability fixes. anthropics/sandbox-runtime provides a lighter OS-level boundary, while its open issues expose signals, proxies, IPC, and path behavior as active compatibility and security work. Isolation is becoming a testable contract, not a checkbox.
Current research makes the evidence split visible. StateTrack-Bench reports that a model can recall a constraint while no longer applying it. TauIndianBankBench evaluates required procedure steps that may leave the final database state unchanged. MemPABench tests whether personal agents update when to clarify, confirm, or act as preferences change. TabAgent explores moving repeated closed-set routing and gating decisions out of generative calls. Together, these papers argue against treating remembered instruction, final state, and one model verdict as interchangeable evidence.
Production accounts give the architecture a bounded operator shape. Meta reports a deterministic router for more than 20 workflows and 14 specialist agents, with configuration work reduced from days to under an hour. AWS reports that Rescale's test-triage agent improved agreement with human triage and reduced deployment-blocking analysis time while engineers retained validation and release responsibility. These are attributed case studies, not independent causal proofs, but both begin with narrow workflows, explicit gates, and human ownership rather than unconstrained general autonomy.
Counter-evidence¶
Two planes can become twice the bureaucracy. A runtime policy can be complete and still authorize the wrong business action. A coverage ledger can truthfully list every file while the auditor misses the vulnerability. An external manifest can prove component identity without proving that the component is safe. A protected test can encode a stale requirement, and a human can approve a bad result. Small application teams should not build a general attestation platform for every read-only task.
The evidence is also uneven. OpenShell, sandbox-runtime, Cloudflare's skill, Meta's system, and the Rescale account are described by their maintainers, vendors, or operators. The arXiv results are author-reported and not reproduced here. Direct DevSecOps and agent-security discussions establish concrete questions, not representative market demand. The proportional response is one task-specific runtime contract joined to one small acceptance record.
Package evidence confirms that the primitives are becoming ordinary dependencies. PyPI verifies two releases of sic-agent-sandbox, while NuGet reports 11.0 million total downloads for Microsoft.Agents.AI. Those metrics show packaging and distribution, not secure configuration, unique production use, or accepted-task lift. The npm record could not be deterministically hydrated and carries no unique conclusion.
What Matters to Us¶
-
Put hard authority below the worker. Bind a task to explicit files, hosts, credentials, processes, budget, and effect classes. Prompt instructions and tool descriptions can explain policy, but the runtime must enforce the deny.
-
Keep acceptance evidence outside the worker and diagnostic judge. Record exact runtime identity, eligible and inspected scope, attempts, observed effects, protected results, missing evidence, evaluator completion, cleanup, and final ownership. Preserve
unknownwhen any decisive field is missing. -
Make orchestration boring on purpose. Routing, budgets, retries, cancellation, and promotion are better as deterministic state transitions with specialist reasoning behind narrow interfaces. A model can help classify ambiguity without owning the application invariant.
-
Adopt one control only after a named failure earns it. Compare OpenShell or sandbox-runtime with the smallest existing boundary on one task. Compare an audit workflow with one seeded defect and one non-defect. Measure denied effects, recall, false positives, reviewer reconstruction, latency, and cleanup before adding a platform.
One Experiment Worth Running¶
Build one joined execution-and-evidence canary. Choose a read-only TypeScript task with a deterministic expected result. Run it under one candidate runtime with one allowed directory and host, one denied sibling file and host, no ambient credentials, and logs outside the worker. Seed one finding, one out-of-scope decoy, a delayed tool result, and an evaluator refusal. The final record should distinguish policy deny, tool uncertainty, scope excluded, evaluation unknown, and accepted result, then prove no descendant or credential survived teardown.
This can establish whether the two planes remain distinct on one machine and workload. It cannot establish general sandbox security, universal audit recall, or production value. Expand to a state-changing case only if the first fixture catches the seeded failures and reduces reviewer reconstruction at acceptable cost.
One Thing to Ignore¶
A full agent-security stack assembled from today's top repositories. Do not install OpenShell, sandbox-runtime, Herdr, Cloudflare's audit skill, Claude-Red, and multiple gateways because each covers a visible control. Start with the current failure and the smallest enforceable boundary. In particular, keep offensive skill packs out of ordinary development environments; methodology does not create authorization or containment.
Also ignore a runtime comparison based on stars or feature count. OpenShell's standing measurement warning remains, Hindsight crossed the attention-fading threshold, and first observations for sandbox-runtime and Herdr say nothing about direction or task quality. Use the same allowed, denied, interruption, outcome, and cleanup fixture across candidates.
Supply vs. Demand¶
Supply now spans runtime protocols, OS-level sandboxes, managed workspaces, credential and policy layers, coverage-led audits, trace diagnostics, memory systems, evaluator benchmarks, MCP gateways, package SDKs, and fleet supervisors. The useful shift is that several components now expose versioned contracts and machine-readable artifacts rather than only launch prose.
Observed demand is narrower: a folder-scoped local sandbox, protection for files and secrets after an enterprise rollout, hard agent spending caps, proof that a verifier ran, a way to test tool permissions before deployment, and a final state that distinguishes completion from a plausible message. The match is improving at the primitive level. The missing product is not necessarily another platform; it is the small application-owned join between enforced authority and accepted outcome. The discussions reviewed this week are attributed needs, not search-volume or market-share evidence.
People to Watch¶
- NVIDIA OpenShell maintainers — watch whether the Sandbox Protocol gains independent backends, conformance tests, and stable policy semantics without hiding effective authority.
- Anthropic sandbox-runtime maintainers — watch cross-platform signals, proxies, IPC, temporary grants, child processes, and whether the lighter boundary stays usable without failing open.
- Cloudflare security-audit-skill maintainers — watch seeded recall, false positives, coverage gain, correct unknowns, and total verification cost rather than repository attention.
- Meta's data-pipeline agent team — watch whether the thin-orchestrator pattern transfers to smaller workflows and how failures, retries, and promotion are evidenced.
- Rescale and AWS AI-DLC authors — watch the golden-dataset maintenance cost, classification agreement, escalation quality, and whether the reported deployment benefit persists.
Category Shifts¶
| Category | This Week | Last Week | Direction |
|---|---|---|---|
| Agent infrastructure | Versioned sandbox protocol, lightweight OS enforcement, effective-policy and teardown tests | Identity, authority, recovery, and terminal state converged in one receipt | ↑ Execution policy becomes a testable platform contract |
| LLM eval / testing | Coverage, process adherence, state integrity, required procedure, and evaluator completion split apart | Independent typed evidence channels became the acceptance model | ↑ Evidence specializes by failure class rather than collapsing into one score |
| Code dev tools | Skills, plugins, remote sessions, and control planes became dependency and evidence surfaces | Review capacity and installed behavior dominated | → Distribution grows; accepted work and bounded authority remain scarce |
| MCP tooling | Fresh argument, naming, URI, read-only, and OAuth advisories | Issuer binding and deliberate bad-server tests entered the workflow | ⚠️ Protocol reach grows faster than adversarial application fixtures |
| Memory / RAG | Preference change and state-integrity collapse become central; Hindsight attention fades | Source lineage, correction, retrieval integrity, and localized repair | ↑ Memory is judged by current compliant action, not recall alone |
| Agent orchestration | Thin deterministic routers and narrow production triage show concrete value | Fleet terminal states and trace diagnosis became inspectable | ↑ Boring control flow gains ground over all-knowing coordinator agents |
Open Questions¶
- What is the smallest runtime-and-acceptance record that materially reduces reviewer reconstruction for a normal Node/React/Postgres task without creating an internal platform project?
- Can a common sandbox protocol preserve effective policy, credential scope, interruption, evidence custody, and teardown across local OS controls, containers, and remote backends—or will those semantics remain implementation-specific?