What 1,500 Agent Runs Taught Us About Reading Documentation

Coding agents now read documentation on their own. They look up auth headers, rate limits, setup commands and deprecations, then act on what they find. We wanted to know how reliably they do it, and what a score like "92% solved" actually measures.

We gave three command-line agent harnesses 222 real developer-documentation questions and recorded every step they took on 24 live documentation sites, across 1,509 valid runs.

The short version: agents mostly get the answer, but more than half the questions don't need the web at all, the harness you pick matters more than the model for cost, and recovering from errors is still the weakest skill.

What We Did

• Agents: Claude Code 2.1.197, OpenCode 1.19.3, and the Codex CLI 0.163.0 harness. All three ran Claude models. The Codex CLI harness reached Claude through a LiteLLM gateway, so its results describe that harness with Claude, not Codex with OpenAI models.
• Models: Claude Haiku 5.5 on every task, twice. Claude Sonnet 5.5 once on a stratified 59-task subset.
• Tasks: 222 questions developers actually ask: auth headers, rate limits, setup commands, deprecations, cross-page questions, broken start links, and questions the docs don't answer. 197 core tasks plus a 25-task harder tier.
• Setup: A fresh sandbox per run, all traffic recorded, each agent's normal user agent plus a research header. No logins, and no bypassing of bot protection or rate limits.
• Scoring: Answer keys copied from the documentation and checked against the live pages, with a second opinion from an LLM grader. Runs broken by infrastructure were excluded from every rate.

The runs took place on 26 and 27 October 2026.

Most Documentation Questions Don't Need the Web

Before trusting any agent score, we asked each model every question with no tools and no web access. Haiku answered 116 of 197 core questions correctly from memory. Sonnet answered 132. Popular APIs are well represented in training data, so "find the auth header for this API" is often a recall test.

A benchmark that reports "92% solved" without this baseline overstates what the agent contributed. We now report web lift, the questions solved with the web that the model couldn't answer alone. We also move held-out questions that both models answer from memory out of the test set.

Hard to Memorise Is Not Hard to Research

We then built a harder tier: three-page questions, version-specific details, current-versus-deprecated conflicts and changelog dates. We kept only questions that neither model could answer from memory. Closed-book, Haiku got 1 of 25 and Sonnet 4 of 25.

With the web, the same agents solved 86% of runs (129 of 150). Once a question points at the right product, today's agents are good at targeted lookup. Obscurity alone doesn't make a documentation task hard for them.

The Harness Matters as Much as the Model

With the same model (Haiku 5.5) on the same 197 tasks, the three harnesses landed within a few points of each other on accuracy. Their cost per run differed by about five times.

The accuracy intervals overlap, so those differences are small. The cost difference is not.

Model size bought less than we expected on these tasks. On the 59 questions both models ran, Sonnet 5.5 and Haiku 5.5 were within each other's confidence intervals for every harness, and Sonnet cost roughly 5 to 14 times more per run. For documentation lookup, picking the harness is the bigger lever.

One Successful Run Is Not Reliability

Every core task ran twice per harness. Solving it at least once and solving it both times are different numbers. The gap between them is flakiness that a single-run benchmark never shows.

On best-of-two, the harnesses look nearly identical. On "does it get this right every time", OpenCode is clearly steadier, with a gap of 1.5 points against 10 to 12 for the other two. If you build on an agent, the second number is the one you live with.

Where Agents Still Struggle

• Recovering from errors. When a run hit a broken link, a 404 or a failed tool call, it went on to solve the task 74% of the time (336 runs with an error). Recovery is the weakest skill we measured.
• Exact values. Questions about a specific limit, default or size were solved 86% of the time, the lowest of the larger categories. Close-but-wrong numbers were common.
• Using the page in front of them. When the answer was on the page the agent started on, 17% of runs still didn't get it right (24 runs, a small sample).
• What went well. Agents caught false premises almost every time (98.6%), such as a parameter that doesn't exist. They said "this isn't documented" when it wasn't (96.7%), and almost never gave up on a question they could answer (0.3%).

Measuring Agents Means Measuring Carefully

Five practices changed our numbers more than any single agent did. We'd recommend them to anyone evaluating agents on the open web.

1. Run a closed-book baseline. Without it you can't tell research from recall.
2. Separate your own infrastructure from the site. A politeness cap in our recorder produced most of the rate-limit responses we saw. Counting those as "the site blocked the agent" would have told a false story. Real bot challenges were rare on documentation sites, and none ended a run.
3. Test each harness and model pairing before scoring it. A harness release can send request options that a newer model rejects. Those runs are infrastructure failures, not agent failures, and need to be detected and excluded.
4. Audit answer keys against the live site. Keys written from a single page miss answers that live elsewhere in the docs. A second grader that disagrees with the key is the cheapest way to find them.
5. Calibrate difficulty from data. Our rule-based difficulty labels predicted nothing: "easy" tasks were solved 83% of the time and "hard" ones 92%. Labels set from observed solve rates on one attempt predicted the next attempt cleanly: 97%, 74% and 33%.

What This Means for Documentation Teams

Agents are already reading your docs, and they mostly succeed. Of the 30 sites we surveyed, 21 publish an llms.txt index complete enough to find documentation pages from, and those indexes made pages easy to find.

The failures we did see cluster in a few places:

• exact values spread across pages;
• version-specific details;
• recovering when a link has moved.

Clear canonical pages for limits and defaults, stable URLs with redirects, and explicit "current vs deprecated" statements help agents as much as they help people.

One more point for anyone building datasets from agent traces: many documentation sites' terms restrict automated access or reuse, and only some license their docs openly. Check the terms for each source before you collect.

Limitations

• One model family. All agents and the grader are Claude models. Results may differ with other model families.
• A focused benchmark. 222 tasks on 24 sites is not a census, and confidence intervals are wide for small categories.
• Documentation changes. Answer keys were checked against live pages on 27 October 2026.
• Provider-side search. The Codex CLI harness used web search run by the model provider, which our recorder can't observe, so its evidence is less verifiable than the other two.

About This Study

This post summarises internal Glintbase experiment exp_2026_10_26_001: 1,568 recorded runs, of which 1,509 were valid after excluding runs broken by infrastructure. The run data isn't published. It contains third-party documentation excerpts, and rights to those are under review for each site. The post names no documentation sites in its findings.

Disclosure: Glintbase sells agent-readiness scoring (ARS) and audits of how AI agents experience websites and documentation. All three harnesses ran Claude models from Anthropic, and the LLM grader was also a Claude model. The study was planned in writing before data collection, and its deviations are logged.