On 12 and 13 October 2026 we rebuilt how the Agent Readiness Score is calculated.
The short version: a check now passes only if the scanner actually observed the evidence for it. That sounds like it should always have been true. It was not.
This post explains what was wrong with ARS 3.1, what ARS 3.2 does instead, why every Glintbase surface (the website, the CLI and the MCP server) now runs the same engine, and what happened to the leaderboard when we rescanned all 101 companies on it. It also covers what the new score still does not prove.
A readiness score that passes checks it never verified is not a conservative score or a generous score. It is a guess with a number attached. ARS 3.2 is built on one rule: no evidence, no points.
What Was Wrong With ARS 3.1
We found the problems by doing something we should have done earlier: reading every check in the registry and asking, for each one, "what did the scanner have to observe for this to pass?"
For too many checks, the honest answer was "very little".
MCP quality checks passed on detection alone. Once the scanner found any MCP endpoint, about 13 MCP quality checks passed automatically. That included schema strictness, readOnlyHint and destructiveHint annotations, prompts and resources support, in-channel error reporting, protocol version and an SSRF guard. The endpoint did not have to answer. A server that returned HTTP 401 to every request was enough.
API checks passed because a file existed. Every api-* check passed as soon as an OpenAPI file was found. Nobody read the spec. A two-line placeholder scored the same as a carefully maintained OpenAPI 3.1 document.
Links were never followed. The llms-txt-links-resolve check did not fetch a single link. It passed if the llms.txt file was present.
auth.md stages were assumed. The auth.md stage checks passed if the file existed, regardless of what it said.
The CLI scored fixed inputs. The audit and check commands in the CLI were fed hardcoded data that said "MCP live" and "auth found". On those code paths, the CLI's score did not depend on what it scanned.
Two engines, two answers. The stdio MCP package scored sites with an older model (ARS 1.0) than the website. The same URL could get different scores depending on which Glintbase tool you used.
Whatever each shortcut was meant to be when it was written, the cumulative effect was that the score systematically flattered any site that had the right files in the right places, whether or not those files worked.
The New Rule: Evidence or Nothing
In ARS 3.2 (score version ars-3.2.0, shipped in product version 3.3.0), every check resolves to one of five statuses:
| Status | Meaning | Counted in the score? |
|---|---|---|
pass | The scanner observed evidence that the requirement is met | Yes |
warn | Evidence observed, requirement partially met or degraded | Yes |
fail | Evidence observed, requirement not met | Yes |
na | The surface does not exist on this site | No (excluded) |
skip | The surface exists but the scanner could not verify it | No (excluded) |
There is no partial credit. A check is not "probably fine".
Two details matter here.
First, skip is new and deliberate. If a site has an MCP endpoint but it requires authentication the scanner does not have, we cannot verify its tool schemas. In 3.1 those checks passed. In 3.2 they are excluded from the score and reported as unverified, so they neither reward nor punish the site for something we could not see.
Second, absent surfaces are penalised exactly once. If a site has no MCP server, its MCP existence check fails and the dependent quality checks become na. You lose the points for not having the surface; you do not lose them again thirteen times over.
The check registry is now the single source of truth for points and bonus status. It holds 129 checks across four layers: Discovery 17, Access 45, Usability 61, Payments 6. Scoring code reads weights from the registry rather than hardcoding them, so the methodology page, the scanner and the CLI cannot drift apart on what a check is worth.
What the Scanner Now Actually Does
Removing the shortcuts meant building the probes they were standing in for.
Live MCP probing
When the scanner finds an MCP endpoint, it now talks to it:
- It sends
initializeusing the 2026-07-28 protocol version and falls back to legacy wire forms. Support for 2026-07-28 only counts if the server actually negotiates it, not if it merely accepts the request. - It calls
tools/list, carrying the session id the server issued, and inspects the real tool schemas and annotations. - It sends a call for a tool that does not exist (a canary) to check whether the server reports errors in-channel rather than failing silently or dropping the connection.
- It will only ever invoke a tool that the server itself annotates
readOnlyHint: trueand that has no required arguments. It never calls unannotated tools on third-party servers.
That last rule is a safety boundary, not an optimisation. A readiness scanner that triggers side effects on someone else's production MCP server is a liability, and we would rather leave a check unverified than take that risk.
Real OpenAPI inspection
The scanner now parses the spec and looks for the things an agent needs: operationIds, request and error schemas, pagination patterns, rate-limit headers, application/problem+json error responses and OpenAPI 3.1. An api-* check passes because the spec contains the evidence, not because the file exists.
llms.txt links that resolve
The scanner fetches up to 8 links from llms.txt. At least 80% of the sampled links must resolve for the check to pass.
Rubric Changes
A few scoring decisions changed alongside the evidence model:
- Legacy SSE transport is bonus-only. It no longer counts toward the base score.
- The 8 auth.md stage checks are bonus-only. They can add to a score but no longer count toward the base.
- A duplicate in-channel error check was removed. Two checks were measuring the same behaviour.
- Grades and colour bands are unified. Every surface now uses the same scale.
| Grade | Score |
|---|---|
| A+ | 95–100 |
| A | 86–94 |
| B | 70–85 |
| C | 48–69 |
| D | 28–47 |
| F | 0–27 |
Evidence Coverage: Showing What We Did Not Measure
Every scan now reports its own evidence coverage:
- how many checks were measured versus how many are registered,
- what percentage of the scored points were actually verified,
- and which layers had less than half of their points verified. Those layers are flagged "low evidence".
We want to be direct about one layer in particular. On URL scans today, Discovery is always flagged low evidence. Most of its base checks are not probed yet. Until those probes ship, the Discovery portion of a URL scan tells you less than the other layers do, and the scan says so on its face.
A score without a coverage figure invites people to read more into it than it contains. Publishing what fraction of the score was verified is less flattering than a single clean number, but it is the only way a reader can tell a confident 80 from a hopeful one.
Unreachable Is Not a Score
ARS 3.2 refuses to score a site that never responds. The scan returns TARGET_UNREACHABLE instead of a number.
We learned this one the embarrassing way. During a rescan, a network failure that looked like an outage caused glintbase.dev itself to be saved with a score of 2 out of 100. The site was up. One of its Vercel edge IPs was unreachable from one network route, and the scanner treated "could not connect" as "has nothing".
Two fixes came out of that:
- A site that never responds is not scored, and nothing is written to the leaderboard.
- The homepage timeout went from 5 seconds to 15 seconds, with one retry.
One Engine Everywhere
The scoring engine is now published as a standalone package, @glintbase/scanner-core 3.3.0, with zero runtime dependencies. @glintbase/cli 3.3.0 and @glintbase/mcp both score with it, and so does the website.
We checked parity directly: ora.ai scored 89 (A) on the website and 89 (A) in the CLI.
A duplicate copy of the CLI that lived inside the scanner repository was deleted, which removes one more place for scoring logic to quietly diverge.
# The same engine the website uses
npx @glintbase/cli@3.3.0 audit https://docs.example.com
What Happened to the Leaderboard
We rescanned all 101 companies on the leaderboard with ARS 3.2.
The average score moved from 54 to 48. That is the expected direction: checks that used to pass by default now have to earn their points.
The grade distribution after the final rescan is approximately:
| Grade | Companies |
|---|---|
| A | 1 |
| B | 23 |
| C | 30 |
| D | 31 |
| F | 16 |
The top five are Ora 89, Boat 85, Unkey 81, Glintbase 81 and OpenRouter 80.
Yes, Glintbase is near the top of its own leaderboard. That is an obvious conflict of interest and we are not going to pretend otherwise. We build to our own rubric, so we would expect to score well on it, and you should weigh our score accordingly. What we can offer instead of an assurance is inspectability: the methodology is public, the full specification lives in the glintscanner repository under docs/methodology/ars-3.2.md, and the engine itself is an open package anyone can run against our site and theirs.
ARS 3.2 scores are versioned (ars-3.2.0) and are not comparable with ARS 3.1 or earlier scores. A company that "dropped 10 points" between versions may not have changed anything; the ruler changed. Compare scores only within the same score version.
A Spot-Check: deepmind.google
Large score moves were the first thing we investigated after the rescan, because they are where scanner bugs hide.
deepmind.google fell by about 37 points. Most of that drop was real: the scanner found no MCP server and no OpenAPI spec, and 3 of the 8 sampled llms.txt links were broken. Under 3.2, each of those costs points.
But the investigation also exposed two genuine scanner bugs:
- Minified HTML with unquoted attributes. The page used
name=descriptionandtype=application/ld+jsonwithout quotes, which is valid HTML. Our parser expected quotes and read the meta description and JSON-LD as missing. - Large sitemaps. Sitemaps over 20 KB were treated as missing.
Both are fixed. A separate spot-check found that the surface inventory showed 1 of 13 surfaces for Ora because discovery was being cut off at 7 seconds. That is fixed too.
The lesson we took from this is that a stricter score is also a more sensitive one. When checks can actually fail, parser bugs and timeouts show up as lost points instead of hiding behind automatic passes. Every large move deserves a human look before anyone draws conclusions from it.
What ARS 3.2 Does Not Prove
This is the part we most want readers to take away.
ARS is not yet validated as a predictor of agent success. It measures whether readiness signals are present on a site and whether the scanner could verify them. It does not measure whether an AI agent will complete a task on that site faster or more reliably. We believe the signals matter, which is why we built the score, but belief is not validation. Correlating ARS with real agent outcomes is future work, and until we publish that, treat the score as a structured audit, not a forecast.
It also does not measure prose quality, the correctness of every documented example, or anything behind authentication we were not given access to. Those are either skip or out of scope, and the evidence coverage figure is there to make that visible.
Where to Go Next
- Read the full ARS 3.2 methodology.
- Read the specification at
docs/methodology/ars-3.2.mdin the glintscanner repository. - See what else changed on 12–13 October in what auditing our own stack taught us, and the research corrections in we audited our own research.
Run a Free AI Readiness Audit
Identify broken parameter signatures, missing Model Context Protocol schemas, and sitemap issues in your developer documentation.
