Agent Skills Are Becoming Testable; Their Evaluators Need Tests Too¶
π§ Today's Thesis¶
Installable expertise is moving from plausible instruction files to measurable behavior, but the evaluation path must be treated as another fallible dependency. Modern Web Guidance 1.0 ships reviewed, task-scoped browser knowledge with public vendor evaluations; NVIDIA SkillEvaluator can compare the same task with and without a skill and keeps judge-infrastructure failure distinct from a zero score. That is a meaningful step beyond βinstall it and look at the demo.β
The operational boundary appears in the counter-evidence. MetaMask's advisory review case reports 43 accepted fixes across roughly 480 pull requests, but also documents declined findings, false positives, unmeasured cost, and the need to keep humans reviewing. SkillEvaluator's fixed issue #153 credited reading a script as executing it. The useful architecture is therefore paired evaluation plus protected cases, clean controls, explicit missing verdicts, and a reviewer outside the component being promoted.
π― What Matters to Us This Week¶
-
Use domain skills as advisory dependencies before making them gates. Modern Web Guidance has credible release, evaluation, and production-case evidence, while its active issue tracker still contains accessibility, cross-browser, content, and real-project evaluation corrections. Pin one revision to one existing React task and let repository-owned browser and accessibility checks decide the result.
-
Test the evaluator with planted failures and clean cases. Auditing AI peer reviewers injects localized errors by type and severity and also measures false positives on clean papers. The exact benchmark was not reproduced here, but the pattern transfers: seed one known defect, include one correct change, preserve inspected coverage, and keep the protected verdict outside both worker and judge.
-
Measure review as accepted work per reviewer-hour. A direct ExperiencedDevs discussion describes generated code as cheap while understanding and review remain expensive. Plannotator, Open Code Review, and advisory skills should compete on reconstruction time, accepted findings, clean-change noise, corrections, and protected outcomesβnot comment volume.
-
Prefer narrow executable contracts over broad behavior bundles.
kajisho5/ffmpeg-skillexposes 42 named local media operations and an MCP server. That makes one media workflow easier to inspect than an unrestricted shell, provided exact inputs, arguments, outputs, temporary files, cancellation, and visual or audio acceptance remain external.
π₯ Top Movers¶
- GoogleChrome/modern-web-guidance-src (1,219 β today, 7,135 total) β maintained cross-browser expertise held near its verified 1,925/day peak as release and case-study evidence deepened.
- dream-num/univer (988 β today, 25,593 total) β the AI-office SDK rebounded near its 1,142/day clean peak; renewed attention is not a reason to replace an existing document stack.
- backnotprop/plannotator (866 β today, 11,820 total) β set a new clean peak around structured human review and feedback handoff.
- ChromeDevTools/chrome-devtools-mcp (717 β today, 56,900 total) β set a new clean peak as production browser-verification examples became concrete.
- alibaba/open-code-review (691 β today, 52,048 total) β remained outside the strict fading threshold; its deterministic-plus-model split matters more than the rebound.
- NVIDIA/SkillEvaluator (122 β today, 1,653 total) β held close to its 133/day peak while exposing unusually explicit evaluator and infrastructure-failure semantics.
π What Changed the Frontier¶
-
A skill can now arrive with an inspectable promotion path, not only an installation path. Modern Web Guidance publishes guided-versus-unguided evidence, SkillEvaluator packages paired runs, PyPI exposes the current 0.7.2 artifacts and missing provenance attestations, and the MetaMask case reports accepted and declined advisory findings. The newly practical move is to promote one pinned skill to one workload through a control arm, negative case, and protected result.
-
Browser evidence can close a real frontend verification loop. Policybazaar and Pensionbazaar report a code-to-browser audit workflow, while WeTV reports trace-driven performance debugging in CI. These are vendor-attributed outcomes rather than independent studies, but they make a small React canary concrete: current code, live browser, named checks, evidence, and human disposition.
π First Appearances¶
- kajisho5/ffmpeg-skill β first clean Scout baseline at 331/day, 5,959 total. A narrow local media-tool contract worth one bounded experiment.
- ModernRelay/omnigraph β first clean baseline at 174/day, 2,637 total. Interesting branchable graph-storage architecture; ignore for adoption until a measured Postgres constraint exists.
- assistant-ui/assistant-ui β first clean Scout baseline at 75/day, 13,236 total. This is an old registry gap, not a new launch; reusable React UI still sits below product-owned state and approval.
- HKUDS/AutoSwarm β first clean baseline at 64/day, 2,538 total. Coordination supply without accepted-task evidence over one worker.
π± Rising Stars¶
(Only repositories with repeated, window-labelled daily observations receive rising claims.)
- backnotprop/plannotator β rose from 408 to a new 866/day clean peak. Test reviewer reconstruction rather than generated-feedback volume.
- ChromeDevTools/chrome-devtools-mcp β rose from 664 to a new 717/day peak. Browser visibility is useful; session authority and application acceptance remain external.
- Tencent/LoopForge β its second clean observation rose from 184 to 423/day, establishing a rising attention signal around durable coding-agent delivery workflows.
- ag-ui-protocol/ag-ui β rebounded from 9 to a new 355/day clean peak. The protocol regained attention; server-side state and authorization still own effects.
- tradermonty/claude-trading-skills β set a new 706/day peak. Consequence, provenance, and transaction boundaries keep it out of the active stack.
π Fading¶
(Attention labels compare only repeated stars_today observations; they do not grade product quality.)
- SnailSploit/Claude-Red β peaked at 699/day, now 131 (81.3% lower). The strict fading signal does not make offensive procedures safer; explicit authorization and independent enforcement remain mandatory.
- google/ax β peaked at 2,305/day, now 31 (98.7% lower). Its task, workspace, gateway, and model contracts remain useful study material, while Kubernetes-scale adoption stays off-lens.
βοΈ Battles (same category, competing)¶
- Modern Web Guidance vs repository-native guidance β the maintained skill brings current cross-browser expertise, public vendor evaluations, and an active correction loop; a small reviewed repository guide is easier to pin, understand, and remove. One existing React interaction under the same protected checks should decide incremental value.
- SkillEvaluator vs a small paired CI script β SkillEvaluator supplies structured tiers, graders, and failure semantics; an ordinary script may be easier to trust for the first with-skill/without-skill task. Adopt the platform only when repeated promotions make evidence assembly the measured bottleneck.
- Plannotator vs Open Code Review β one structures human plan and diff feedback, while the other joins deterministic rules with model review. Compare reviewer reconstruction, accepted defects, clean-change false positives, exact inspected scope, cost, and final protected result.
π¬ From Research¶
- Auditing AI peer reviewers: a benchmark on real scientific papers β injects localized errors and measures detection plus false positives on clean papers, offering a strong evaluator-test pattern.
- Skill Adapter: Personalized Modification and Online Routing of General-Purpose Skills β makes local adapters and dynamic routing part of the skill runtime, expanding what must be pinned and evaluated during a long task.
- Actor, Judge, or Teacher? β separates acting, judging trajectories, and teaching policies and reports that the roles do not share one reliable ranking.
- CapLite β remains the week's clearest authority complement: derive selected write policy from typed tools and the user request before untrusted results arrive.
All four are current arXiv records inside the November 10β16 research window. Their reported results were not reproduced in this run.
π What's Changing¶
The ecosystem is beginning to treat skills as promotable software dependencies. A useful skill now has a publisher and revision, an activation path, cases, a control arm, evaluator behavior, and production or practitioner evidence. That is substantially better than copying instructions into a repository and trusting the next output.
The weak point moves one layer outward. Skill text can influence the worker; the evaluator can misread the trace; a model judge can fail; a case can omit the dangerous path; a live browser can inspect the wrong session; an advisory bot can sound confident about product behavior it does not know. The durable record therefore joins dependency revision, task source, authority, planted failure, clean control, eligible and inspected scope, missing verdicts, protected application result, reviewer decision, cost, rollback, and removal.
Ordinary dependency distribution continues outside the headline skill ecosystem. The verified Microsoft.Extensions.AI 11.0.0 NuGet record showed 22,898 downloads for that release and 44.5 million total package downloads when discovered. Those figures establish reach across versions, not unique production users, safe configuration, or accepted outcomes.
All 20 general web-discovered URLs hydrated successfully across Chrome documentation, GitHub, PyPI, NuGet, Reddit, GitHub's changelog, and arXiv. The direct discussions indicate reported needs, not representative prevalence; vendor case studies establish attributed outcomes, not independent replication. The optional YouTube lane contained only one transcript and added no decisive evidence, so no fallback discovery delayed the run.
π§ͺ One Experiment Worth Running¶
Run one advisory skill-review canary with a planted defect and a clean control. Proposed first step: pin one Modern Web Guidance revision and select two small, existing React changesβone containing a known accessibility or browser-compatibility defect covered by a guide, the other already correct. Run the same agent, checkout, budget, browser checks, and reviewer with and without the skill. Preserve retrieved guide, comments, inspected scope, accepted and declined findings, false-positive minutes, protected test result, runtime, tokens, and removal. One pair can justify advisory use on that workload; it cannot establish a universal merge gate or reproduce the vendor studies.
β οΈ One Risk to Track¶
The worker, skill, and evaluator can converge on the same plausible mistake. Watch for self-authored cases, a grader that credits intention instead of execution, a judge failure converted to zero or pass, skipped scope hidden by a final status, or browser evidence gathered from the wrong session. The downside is a coherent green report with no independent proof of the intended application state. Keep at least one planted failure, clean negative control, protected postcondition, and accountable reviewer outside the promoted dependency.
π One Thing to Ignore¶
A multi-agent coordination layer before one-worker acceptance is stable β AutoSwarm's first 64/day baseline shows supply, not accepted-task lift. Revisit only when a bounded task with genuinely separable work demonstrates lower accepted latency without more conflicts, context copies, retries, reviewer minutes, spend, unresolved terminal states, or cleanup residue.
π‘ Surprise Pick¶
kajisho5/ffmpeg-skill β the unexpected value is not autonomous video editing. It is the dependency shape: a bounded local tool contract with outputs that can be checked through ordinary media metadata and small visual or audio samples. That makes it a good laboratory for skill admission, exact effect receipts, cancellation, and independent acceptance.
π Supply vs. Demand¶
| What's being built (supply) | What people want (demand) | Match? |
|---|---|---|
| Paired skill-evaluation frameworks and public vendor evals | Proof that a skill improves one owned task without hidden regressions | Improving β control arms exist; operator-owned cases and protected outcomes remain scarce |
| Maintained web expertise plus live browser tools | Current, accessible frontend code with less manual verification | Promising β production cases are concrete but vendor-attributed and workload-specific |
| Advisory and agentic review surfaces | More generated work without losing system understanding or reviewer capacity | Partial β review remains the reported bottleneck and false-positive cost is still local |
| Dynamic skill adapters and routing research | Personalized behavior that remains inspectable as task state changes | Early β adaptation expands the manifest and regression surface |
| Multi-agent coordination products | Lower accepted-task latency | Unproven here β worker count and shared context do not establish accepted throughput |
π Category Pulse¶
| Category | New Today | Trending Count | Signal |
|---|---|---|---|
| Code dev tools | 0 | 9 registry-matched daily repos | β Maintained expertise and structured review converge on promotion evidence |
| MCP tooling | 1 | 3 | β Narrow executable contracts join broad browser control |
| Backend for AI | 1 | 3 | β Branchable graph storage is interesting but off the immediate roadmap |
| Agent frameworks | 1 | 3 | β οΈ Coordination supply still lacks one-worker accepted-task evidence |
| UI generation | 1 | 2 | β Reusable React agent UI enters as a registry gap |
| Agent infrastructure | 0 | 2 | β Attention fades while the contract ideas remain useful |
| LLM eval / testing | 0 | 1 + research | β Evaluator regressions, clean controls, and missing verdicts become first-class |