WTF are evals? How to test agents and the software they use
Vercel’s guide to agent evaluations argues that teams should test whether agents complete user tasks reliably, not merely whether individual calls look plausible.
- claude
- claude-code
- codex
Live desk · agent-shaped software
An atlas of agent-shapedsoftwareinterfacesworkflowsteamsruntimessoftware
What changes when agents do the work: systems, signals, shifts — from primary sources, daily.
Daily notes and analyses from /posts.json and /analysis.json
Vercel’s guide to agent evaluations argues that teams should test whether agents complete user tasks reliably, not merely whether individual calls look plausible.
Epoch AI estimates that VNET’s operational Bayin Ulanqab campus has 123,000 H100-equivalents of AI compute, 221 MW of IT power and an $8.4 billion capital cost.
A federal magistrate judge in California ordered former OpenAI engineer Abdulrahman Alfozan detained pending trial, finding that no release conditions would reasonably assure his appearance.
GitHub is rolling out shared Copilot Automations, letting teams create and maintain repository-owned cloud workflows rather than keeping each automation personal to its creator.
Google announced Gemini 4 Hydrogen, a model positioned for fast execution across everyday coding and agent workflows, alongside a model-switching feature in Google Antigravity.
Researchers report that language models can learn some multi-step tasks while emitting only filler tokens between a prompt and an answer.
Microsoft Security Research and the Microsoft AI Red Team describe two ways post-training artifacts can carry dormant backdoors into coding agents: poisoned fine-tuning data and attacker-trained model checkpoints.
Exhibit 1 Where the evidence sits
Share of 9,751 topic tags by area (column width) and by topic within the area (height), %
Exhibit 2 Blind spots
Top 30 topics: records on the topic (log scale) against the share already written up as Field Notes or atlas records, %
11 of 30 topics sit below what is typical for their size.
Exhibit 3 From signal to thesis
Top: records surviving each step, count. Below: which kind of signal feeds which thesis, band width = signals
Releases and tools24 links · 41% of all signals → 30% of cited
Field reports and operating models48 links · 43% of all signals → 57% of cited
Research and analysis10 links · 15% of all signals → 13% of cited
Agent security moves to runtime boundaries7Agent-ready software exposes capabilities6Goal-scoped loops replace manual continuation6Harness architecture outlives model choice6Agent economics moves to completed work5Company memory needs write and correction loops5Verification bandwidth is the scarce engineering resource5Scientific agents need artifact evidence4Generative media becomes application infrastructure8Agent protocols become the interoperability layer7AI-native organizations move toward review and orchestration6Design systems become executable agent context6Model answers become task interfaces4Skills become a portable capability layer4Machine readers need operator identity3Exhibit 4 What is moving
Share of dated records that touch each area, %. Left: before and during the 90 days to 12 Nov 2026. Right: month by month, and the change in points
Before 15 Aug 2026884 records
Last 90 days3,867 records
AreaMonthly share, Oct 25–Nov 26Change, pts
Exhibit 5 Thesis scorecard
Working theses by confidence, supporting signals and the window in which those signals were seen. Click a column to sort
01 / 05
Arrow keys page throughGoverned shared context layer that multiple AI clients can read with explicit provenance and access rules.
challenged · 4 in the graveyardThe human spends less time reading logs for the agent and explaining what exactly broke.
source-inspected · 4 deltasTurn public repository observations and ecosystem references into a ranked repository radar and a navigable map of the wider production AI stack.
B2 · One local content pass plus a GitHub metric refreshTurn public repository observations and ecosystem references into a ranked repository radar and a navigable map of the wider production AI stack.
NR>
Loading the privacy-safe route aggregate…
Open the JSON contract