The Harness Is Now the Product—and the Attack Surface¶
What changed our view¶
Agent execution is commoditizing, while the harness that grants authority, selects tools, records evidence, and accepts effects is becoming the durable product boundary. Early in the week, managed runtimes, TypeScript evaluation, consent, model routing, and review agents made more execution available. By the weekend, Plugin4Shell supplied the failure case: a reviewed plugin SHA did not necessarily identify the code a client fetched and ran, according to AIR's disclosure. The week's evidence strengthens last week's acceptance-plane thesis, but makes it more specific—receipts must bind not only instructions and outcomes, but the exact controller and dependency artifacts that executed.
- 01Buy execution; own acceptance.
- 02Promote behavior bundles like executable dependencies.
- 03Turn failures into replayable protected cases.
Scout Weekly — September 14–20, 2026
Evidence¶
OpenAI's Agents API and Amazon's AgentCore September notes show execution moving into managed platforms: long-running work, tools, state, sandboxes, TypeScript evaluation, identity, and end-user consent. GitHub added activity and model-budget controls, while Copilot code review and alibaba/open-code-review made model-assisted review more executable. These are meaningful reductions in platform work, but none defines whether one application effect is correct.
The open-source board concentrated around the missing acceptance layer. cloudflare/security-audit-skill ended at a verified 3,155/day peak with coverage ledgers, independent candidate validation, structured verdicts, and final-record verification. alibaba/open-code-review reached a 3,231/day verified peak earlier in the week before cooling to 985/day, while an open issue showed why configured budgets need runtime receipts. Tencent/BrowserSkill made authenticated browser reuse concrete, and trycua/cua entered at 859/day with cross-OS execution and evaluation; both widen the authority and lifecycle surface that acceptance must cover.
Research turned that operational pattern into measurable objects. An Empirical Study of Harness Design for Coding Agents treats controller choices as a source of performance variation; the corresponding HN discussion attracted 217 points and 59 comments and repeatedly separated the engine from the rest of the car. Chronicle records nondeterministic boundaries and replays selected cut points against changed code. Quantifying Overclaiming Propensity tests whether an agent's final report matches task completion. Together they shift evaluation from “did the model answer well?” to “did this harness, artifact set, and terminal state earn acceptance?”
The demand evidence points to the same bottleneck without establishing representative market demand. Practitioners described rapid multi-agent demos that omit production requirements and operating time, asked whether coding agents should inherit a workstation or remain strictly sandboxed, and discussed ordered planning as a portable layer in the Ordewell Show HN thread. The observed need is not another worker in isolation; it is a trustworthy path from task to accepted effect.
Counter-evidence¶
The major platforms may absorb much of this control plane. AgentCore already combines runtime, evaluation, identity, consent, and observability; GitHub is joining model routing, budgets, sandbox policy, review, and enterprise administration. Application teams should not rebuild commodity scheduling, tracing, or OAuth surfaces merely to preserve architectural purity.
The limit is that an infrastructure feature can only enforce the semantics it knows. A consent portal records a granted scope but does not know whether a particular transfer, deployment, or database mutation is correct. A schema proves shape but not source truth. A trace records execution but may omit external state. A plugin pin can display the intended commit while a resolver fetches something else. Managed controls are valuable inputs to an application-owned acceptance decision, not replacements for it.
What Matters to Us¶
-
Buy execution; own acceptance. Use managed runtimes, sandboxes, identity, tracing, and package ecosystems where they reduce operating work. Keep protected tasks, exact-effect policy, deterministic vetoes, reviewer ownership, recovery, and terminal-state acceptance under application control.
-
Promote behavior bundles like executable dependencies. A useful receipt includes marketplace and publisher identity, repository, reviewed and resolved object, fetched-content digest, transitive dependencies, permissions, model and controller versions, inputs, observed effects, tests, cost, and final verdict. A mismatch blocks execution or acceptance.
-
Turn failures into replayable protected cases. Start with one known integration failure and inspect whether the current trace can reconstruct the smallest safe suffix. If it can, replay in a disposable workspace and add the case to CI; if it cannot, fix capture before buying another profiler or fleet.
-
Measure accepted throughput, not activity. Sessions, messages, comments, workers, stars, and generated diffs are leading indicators. The denominator is accepted outcomes after reviewer minutes, CI work, corrections, escaped defects, recovery, rollback, and total cost.
One Experiment Worth Running¶
Build one executable-identity receipt for one read-only plugin task. In a disposable repository, pin a non-sensitive skill or plugin and record its publisher, marketplace entry, repository, reviewed object, resolved object, content digest, dependencies, permissions, client and harness versions, tool calls, and terminal state. Restrict credentials and network access, run one representative task, and have an independent script or reviewer compare execution with the receipt. A mismatch rejects the artifact; a match only justifies a broader protected-task suite.
One Thing to Ignore¶
A bigger fleet before one worker is replayable and acceptable. CUA, Octop, Orca, and managed subagent features make scale easy to see. More workers multiply partial effects, identities, traces, and reviewer load. Revisit concurrency only when the same workload improves accepted outcomes per reviewer-hour and dollar after planning, recovery, and cleanup are counted.
Supply vs. Demand¶
| Supply shipped or surfaced this week | Demand observed in the same window | Fit |
|---|---|---|
| Managed runtimes, sandboxes, identity, consent, and TypeScript evaluation | Recoverable execution with exact authority and portable evidence | Partial — infrastructure is strong; workload correctness remains local |
| Skills, plugins, package releases, and background updates | Reviewed behavior that cannot silently change underneath a pin | Weak — distribution leads executable identity and revocation |
| Deterministic-plus-agent review and coverage-led security audits | More accepted defects without review floods or hidden gaps | Promising — inspectable workflows exist; recall and total cost need workload tests |
| Semantic profiling, traces, and cut-point replay | Diagnose and regression-test long failures without full reruns | Promising research — capture completeness and side-effect safety remain open |
| Cross-OS computer-use fleets and authenticated browser reuse | Real-state automation with bounded credentials, ownership, and teardown | Early — capability is visible; boundary evidence lags |
| Activity dashboards, routing tiers, and budget requests | Proof of higher accepted delivery per reviewer-hour and dollar | Weak — spend and activity are visible; acceptance economics remain fragmented |
| Team memory, retrieval platforms, and shared scopes | Correct, attributable, reversible state that improves decisions | Partial — governance surfaces improve; decision lift and correction remain local |
People to Watch¶
- Cloudflare — the audit skill's velocity is less important than its explicit separation of coverage, discovery, validation, rejection, and final records. Watch planted-finding recall, false positives, and verification cost.
- Alibaba — Open Code Review makes deterministic selection plus bounded model review concrete. Watch whether runtime budgets, exact reviewed commits, and accepted findings remain joined in evidence.
- Cua — computer-use environments and evaluation are converging in one substrate. Watch one-task replay, isolation, teardown, and cost before fleet breadth.
- Tencent — BrowserSkill and WeKnora expose two hard boundaries: authenticated user state and durable knowledge state. Watch ownership, correction, revocation, and deletion rather than feature breadth.
- Vercel Labs — json-render's component catalog is a useful constrained-output primitive. Watch effect authorization, accessibility, and schema evolution across render targets.
Category Shifts¶
| Category | This Week | Last Week | Direction |
|---|---|---|---|
| Agent infrastructure | Managed runtimes, consent, governed workspaces, and cross-OS computer-use fleets | Execution planes converged around sandbox, state, identity, and tools | ↑ Runtime choice gives way to replay and acceptance evidence |
| Code dev tools | Review workflows, model budgets, and plugin identity became execution surfaces | Specifications, skills, and permissions became versioned process | ↑ Behavior bundles now need executable identity |
| LLM eval / testing | Harness comparison, profiling, cut-point replay, overclaiming, and independent audit records | Full traces, incidents, SLOs, and monitor recall expanded evaluation | ↑ Evaluation reaches controller and terminal state |
| Web / UI agents | Authenticated browser reuse and schema-constrained generative UI | Browser and interface controls were secondary to runtime policy | ↑ User state and rendered effects become explicit boundaries |
| Memory / RAG | Team scopes, authorship, revisions, and state-matched troubleshooting retrieval | Truth, correction, and permission lifecycle dominated | ↔ Governance improves; decision lift remains unproved |
| MCP tooling | npm, PyPI, NuGet, and language SDK distribution stayed broad | Protocol identity, Tasks, events, and enterprise authorization matured | ↔ Connectivity is commodity; effect policy stays application-owned |
Open Questions¶
- Can one portable receipt bind plugin resolution, controller decisions, tool authority, observed effects, and acceptance across managed and self-hosted runtimes without erasing workload-specific semantics?
- Which minimum set of captured boundaries lets cut-point replay reproduce a failure without duplicating an external effect, leaking credentials, or treating an incomplete trace as truth?