--- title: "What Auditing Our Own Stack Taught Us About Building Agent Tooling" description: "An engineering postmortem from the Glintbase overhaul: fabricated fallback scores, permissive auth, rate-limit bypasses, scorecard takeovers and flaky tests across our scanner, site and CLI, and the principles we are keeping." author: "Victor Okolie" publishedAt: "2026-10-13" category: "Engineering" tags: ["Engineering Postmortem", "Security", "Agent Tooling", "MCP Server", "Testing", "Glintbase CLI"] canonical: https://glintbase.dev/blog/what-auditing-our-own-stack-taught-us --- # What Auditing Our Own Stack Taught Us About Building Agent Tooling > An engineering postmortem from the Glintbase overhaul: fabricated fallback scores, permissive auth, rate-limit bypasses, scorecard takeovers and flaky tests across our scanner, site and CLI, and the principles we are keeping. Published 2026-10-13 by Victor Okolie · Engineering · HTML version: https://glintbase.dev/blog/what-auditing-our-own-stack-taught-us Most of what Glintbase does is point a scanner at other people's software and report what we find. On 12 and 13 October 2026 we pointed the same scrutiny at ourselves: the scanner, the glintbase.dev site and API, and the CLI. We reviewed code, spot-checked scans by hand, and read our own endpoints the way an agent would. The scoring changes from that work are covered in [why we rebuilt the Agent Readiness Score around evidence](https://glintbase.dev/blog/ars-3-2-evidence-only-scoring), and the research corrections in [we audited our own research](https://glintbase.dev/blog/we-audited-our-own-research). This post is the engineering side: what was broken, how we fixed it, and the principles we are keeping. Everything described below is fixed. We describe the issues at the level that is useful for other builders, not as instructions for exploiting anyone. > **Insight:** > > The most common failure we found was not a crash. It was software that, when something went wrong, produced a plausible answer instead of an honest error. For tools that agents consume, a plausible wrong answer is the worst possible output. ## Part 1: Our Endpoints Were Lying When They Failed ### Fabricated fallback scores glintbase.dev exposes an MCP endpoint at `/api/mcp` and a sandbox endpoint at `/api/v1/sandbox`. When the real scanner behind them was unreachable, both returned scores anyway: a fabricated 88 (A-) from one, 94 from the other. An agent calling those endpoints had no way to know the result was not a real scan. It would have reported a number we never measured. Our docs made it worse: they advertised MCP tools that did not exist on the hosted endpoint. **Fix:** `/api/mcp` is now an honest proxy to the real scanner, and the sandbox endpoint runs a real scan. If the scanner is unavailable, the caller gets an error. ### A waitlist that always said yes The waitlist API returned "success" even when the database insert failed. Emails were not normalised, so the same person with different capitalisation became duplicate rows. And the confirmation copy promised an email that was never sent. **Fix:** the API reports failure when the write fails, emails are normalised before insert, and the copy no longer promises an email we do not send. ## Part 2: Security Issues We are listing these because the patterns are common in agent-facing APIs, and because a company that audits other people's agent surfaces should be open about its own. - **The OAuth token endpoint issued long-lived tokens to anyone who asked.** It now validates the request and issues short-lived, anonymous tokens that grant no extra access, and authorization redirects only go to addresses a client has registered. - **Rate limits could be bypassed by sending any bearer token.** In effect, the presence of a token was being trusted in place of a valid one. The bypass is closed. - **Costly endpoints were not rate-limited at all.** Simulation, the most expensive operation we run, had no limit. It does now. - **An untrusted URL was interpolated into code that ran in a sandbox.** User input must never become code. That path no longer interpolates untrusted input. - **A public scan could replace a company's official score.** By choosing a different rubric, a public scan could overwrite the score shown for a company. Official scores can no longer be replaced that way. - **Lookalike subdomains could take over a scorecard.** Scanning something like `stripe.attacker.com` could claim another company's public scorecard slug. A lookalike host can no longer claim another company's scorecard. - **A page live-probed arbitrary URLs with no rate limit.** That made our infrastructure usable as a free probe against third parties. The page no longer probes anything; it only shows scans that already exist. > **Engineering note:** > > Several of these share a root cause: a value supplied by the caller (a header, a URL, a hostname, a rubric choice) was trusted to mean more than it did. Agent-facing APIs receive a lot of machine-generated input, and none of it should be treated as authenticated or well-intentioned by default. ## Part 3: Scanner Bugs From Real-World Inputs The stricter ARS 3.2 scoring made scanner bugs visible, because checks that used to pass automatically could now fail. Spot-checks surfaced a class of bugs that only appear on real, messy websites: - **Minified HTML with unquoted attributes.** `name=description` and `type=application/ld+json` without quotes are valid HTML. Our parser expected quotes and read the meta description and JSON-LD as missing. - **Large sitemaps.** Sitemaps over 20 KB were treated as missing. - **Slow discovery.** Surface discovery was cut off at 7 seconds. For one site it found 1 of 13 surfaces. - **Multi-IP DNS.** One of glintbase.dev's Vercel edge IPs was unreachable from one network route. The scanner read "could not connect" as "has nothing" and saved our own site as 2 out of 100. Sites that never respond are now reported as `TARGET_UNREACHABLE` and not scored, and the homepage timeout went from 5 seconds to 15 seconds with one retry. In total, about 40 scanner bugs were fixed from reviews and spot-checks, and the scanner's test suite grew from 106 tests to about 169. ## Part 4: Process Problems Not every issue was in the code. Some were in how the code was built and shipped. - **Two packages shared the name `@glintbase/cli`.** For a while we documented the wrong one. The docs now point to the right package, and a duplicate CLI inside the scanner repository was deleted. - **A stray temp file was committed to a repository.** It has been removed. - **Tests depended on live websites.** Results changed with other people's uptime. Network access is now blocked in tests. - **CI never built or tested the web app.** The site could break without any check failing. CI now builds and tests it. - **The CLI scored hardcoded inputs** on some code paths, and the stdio MCP package used an older scoring model than the website. All three surfaces now score with one published engine, `@glintbase/scanner-core` 3.3.0. ## The Principles We Are Keeping These are written for anyone building agent tooling, but they are first of all notes to ourselves. ### 1. Fail closed. Never fake data. If the real answer is unavailable, return an error. A fabricated score, a "success" on a failed write, or a placeholder dressed as a result will be consumed by an agent that cannot tell the difference, and passed on to a human who trusts the agent. ### 2. "Unknown" is a state. Model it. "Unknown" is not "pass", and it is not "fail" either. Our scanner now has `skip` for surfaces it could not verify and `TARGET_UNREACHABLE` for sites that never answered. Collapsing unknown into pass flatters; collapsing it into fail punishes. Both produce wrong numbers. ### 3. One source of truth Points and bonus status live in one check registry. The version lives in one constant. The engine lives in one package. Every duplicate we found (two CLIs, two scoring models, hardcoded weights) had drifted. ### 4. Measure your own claims before you publish them Our research errata and our fake fallback scores are the same mistake at different scales: a number was published without being checked against the evidence. When we found errors, we published dated corrections rather than editing silently. ### 5. Test against messy inputs Minified HTML, oversized sitemaps, slow sites and multi-IP DNS are normal on the real web. Fixtures written by the same people who wrote the parser tend to share the parser's assumptions. Spot-checking real sites, and then turning each failure into a test, found bugs our original suite never would have. ### 6. Separate "the site is down" from "our route is down" A connection failure tells you about the path between you and the target, not only about the target. Retries, sensible timeouts and refusing to score on no response keep a network blip from becoming a public verdict. ### 7. Dogfood your own tools Scanning glintbase.dev with our own scanner surfaced the unreachable-site bug, and spot-checks of other sites surfaced the parser bugs. Using the product on yourself is the cheapest test environment you will ever get. ### 8. Version your scores ARS 3.2 scores carry a score version (`ars-3.2.0`). A 3.1 score and a 3.2 score are different measurements. Without a version on every number, someone will eventually compare them and draw a false conclusion. ### 9. Say what a metric does not prove ARS measures whether readiness signals are present and verified. It has not yet been validated as a predictor of agent success. Writing that down next to the score is less impressive than leaving it out, and more useful. > **Strategic observation:** > > None of these principles is new. What changes with agent tooling is the cost of breaking them. A human reading a dashboard might notice that a score looks odd. An agent calling an API will take the number at face value and act on it, often without a human checking in between. ## What Is Next The work is not finished. The most visible gap is that the Discovery layer is still flagged low evidence on URL scans, because most of its base checks are not probed yet. The research reproduction issue in the State of Agent Readiness 2026 data is still open. And ARS validation against real agent outcomes is still future work. We will keep fixing these in the open: the [methodology](https://glintbase.dev/methodology), the [changelog](https://glintbase.dev/changelog) and the [research errata](https://glintbase.dev/research/data/ARS_2026_ERRATA.md) are where changes land first. > Check your own site's Agent Readiness Score for free at https://scan.glintbase.dev