Glintbase sells measurement. If our own measurements do not hold up, nothing else we publish should be trusted either.

So during the overhaul on 12 and 13 October 2026 we did to our two published studies what we ask other companies to let our scanner do to their documentation: we checked every claim against the evidence underneath it.

The studies were the Glintbase Integration Index: Sandboxes Edition 1 and the State of Agent Readiness 2026 report. Both had errors. Some were small. Several were not.

On 12 October we published two errata documents:

This post summarises what they contain, how we found the problems, and why we chose dated errata over quietly fixing the pages.

Insight

Every error below was discoverable from files we had already published. None of it required new data. That is the uncomfortable part: the evidence was public, and our headline numbers still disagreed with it.

How We Audited

The method was deliberately boring. For every number in a report, README, blog post or web page, we asked: which file does this come from, and does that file say the same thing?

For Sandboxes Edition 1 that meant going back to the 30 agent transcripts, the per-trial run logs and the verification scripts. For State of Agent Readiness 2026 it meant comparing the PDF against the published dataset (full-scores.csv and research-dataset.json), which agree with each other.

When sources disagreed, we picked the most primary record and made it authoritative. For the sandbox study, that is the transcripts recorded by the agent runtime. Hand-edited summary tables come last.

We did not re-run either benchmark. The errata correct what was published; they do not add new results.

Sandboxes Edition 1: What We Got Wrong

The pass rate was 28/30, not 30/30

We published a perfect pass rate. Two trials did not actually pass.

In Trial 6 for Modal and for boat, the agent printed a "finished" marker without running any tests, because the upstream test file it was meant to run no longer existed. Our verifier accepted the strings "ok" or "passed" anywhere in the output, so it counted both as successes.

The corrected pass rate is 28/30 (93.3%).

Human stops were zero, not 1/1/1

We published one human stop each for Modal, boat and Vercel. Every one of the 30 transcripts contains exactly one user turn, and every prompt ends by telling the agent not to ask questions. No transcript records a human intervention for any provider.

If setup work happened outside the transcripts, such as creating credentials, it was not recorded, so it cannot be counted.

Several headline traps are not in the transcripts

Some of the most quotable findings in the report were specific error stories: a boat HTTP 402 payment gate, a Modal malformed-token error, a Vercel human stop, a Daytona HTTP 401 on the preview URL. We searched all 30 transcripts for each one. None of them appears.

boat's real recorded stumble was a pip PEP 668 externally-managed-environment error, which the agent fixed itself.

"0 leaked sandboxes" could not be verified

We claimed no sandboxes were leaked. The verifier that supported that claim listed each provider's sandboxes after a run, but it also silently deleted any it found and still reported success. So it could never have detected a leak. Daytona Trial 4 did in fact leave one sandbox behind, which the verifier removed.

Leak status is now reported as not verifiable, and the verifiers have been fixed to fail when sandboxes remain.

The ranking is indicative, not controlled

The report said every sandbox received an identical prompt. It did not. Some prompts named the exact API calls needed to succeed; others were mostly plain language. And the tokens, steps and wall-time figures that feed the Integration Tax were self-reported by the agent at the end of each trial, not measured by the harness.

Under the published penalty rule, with corrected pass rates, the Integration Tax order is E2B, Vercel, Modal, Daytona, boat. We still label that order indicative. It is not a controlled comparison, and it should not be read as one.

We have added a script that measures turns and tool calls directly from the transcripts, so future editions do not depend on the agent's own accounting.

Research Note

The transcripts we published originally contained credentials that were not fully redacted. They have been redacted, and the affected keys were revoked. We are stating this plainly because a research repository that leaks keys is a security problem, not a footnote.

State of Agent Readiness 2026: What We Got Wrong

The core problem here was simple to state: the PDF disagreed with its own data.

Item	PDF said	Dataset says
OpenAI ARS	85	35
Anthropic ARS	78	57
Highest score	85 (OpenAI)	79 (Runway)
Platforms scoring 80 or above	6	0
Mean summed traversal hops	27.4	17.6

Beyond the table:

llms.txt adoption cannot be determined. The dataset's llms.txt flag is true for all 100 platforms, so no adoption rate can be computed from it. Any figure we quoted was unsupported.
The $433,200 "token tax" was mislabelled. It is a modelled figure per 100,000 agent sessions per month, not per 1,000 developers.
The dataset is a simulated traversal, not live agent runs. It comes from a deterministic scanner that does not call an LLM, and its token figures are a function of hop counts.
The "450 live runs" have no published data. We cannot point anyone to transcripts or outputs for them, so they should not be relied on.

Two things held up. The mean of 50.7 and median of 52 are correct.

An open issue we have not resolved

There is one problem the errata log but do not fix: the published scores cannot be reproduced from the per-benchmark pass/fail data. The correlation between a platform's benchmark passes and its ARS is weak (Pearson r of about 0.43). Chroma passed 2 of 10 benchmarks and scored 68. Voyage AI passed 8 of 10 and scored 42.

We do not yet have an explanation we can document. It is logged as an open issue rather than patched over, and until it is resolved, treat individual platform scores in that study with caution.

Strategic Observation

A ranking that cannot be recomputed from its own inputs is a claim, not a measurement. That standard applies to us as much as anyone, which is why the reproduction gap is listed as an open issue rather than left out.

Smaller Corrections

Edition 2 is now clearly marked as planned. It has not been published.
External studies are attributed as external. Work such as Ora's AX dataset and WebPageBench is now clearly labelled as someone else's research, not ours.
Our own blog posts carry dated correction notes. Posts that quoted the wrong numbers, including the Integration Tax explainer, the State of Agent Readiness summary and the CI/CD gate tutorial, now open with a "Correction (2026-10-12)" block that says what changed.

Why Errata Instead of Quiet Edits

It would have been easier to update the numbers and move on. The pages would look better today. We chose not to, for three reasons.

The old numbers were public. Some of the original figures may already have been quoted or built on. If we silently changed them, anyone comparing an old citation with the current page would find a mismatch with no explanation, and they would be right to distrust both versions.

Corrections are themselves evidence. A dated erratum tells a reader exactly what was wrong, when it was found and what the evidence is. A silent edit tells them nothing, and teaches them that our pages can change underneath them.

We ask the same of others. Glintbase exists to tell companies that what their documentation claims and what their systems actually do have drifted apart. Applying a lower standard to our own research would undercut the whole premise.

So the originals stay in place, corrected passages carry inline notes, and the errata list each change with its evidence. For the sandbox study, the raw data rows were left unchanged so the original publication can still be audited, with corrected values in a separate file.

What We Are Changing in How We Do Research

The errata fix the past. These changes are meant to fix the process:

Primary records are authoritative. Transcripts beat run logs; run logs beat hand-edited tables.
Verifiers fail closed. A check that cannot confirm a result reports it as unverified, never as a pass. A cleanup step never runs before the thing it is supposed to check.
Metrics are measured, not self-reported. Turns and tool calls now come from transcripts.
Every published number links to its source file. If we cannot point to the file, the number does not ship.
Controlled means controlled. If prompts differ by provider, the result is labelled indicative.

None of this makes our earlier work less wrong. It makes the next edition easier to check, and easier for you to catch us if we slip.

Read the Full Errata

Sandboxes Edition 1 errata
State of Agent Readiness 2026 errata
How the scanner itself changed in the same overhaul: why we rebuilt the Agent Readiness Score around evidence

If you find something we missed, tell us. We would rather publish another correction than defend a wrong number.

We Audited Our Own Research. Here's What We Got Wrong