---
title: "Your robots.txt Lets AI Agents In. Your Edge Might Not"
description: "We fetched the same public pages on 150 popular websites as a declared AI agent, an unlabelled script and a headless browser. Saying we were an agent cost nothing, yet on 16.8% of sites whose robots.txt allowed us the edge still stopped us. Here is what we found, and what site owners should check."
author: "Victor Okolie"
publishedAt: "2026-10-23"
category: "Research"
tags: ["The Silent Wall", "robots.txt", "Bot Management", "AI Agents", "Web Access", "Glintbase Research"]
canonical: https://glintbase.dev/blog/robots-txt-said-yes-the-edge-said-no
---

# Your robots.txt Lets AI Agents In. Your Edge Might Not

> We fetched the same public pages on 150 popular websites as a declared AI agent, an unlabelled script and a headless browser. Saying we were an agent cost nothing, yet on 16.8% of sites whose robots.txt allowed us the edge still stopped us. Here is what we found, and what site owners should check.

Published 2026-10-23 by Victor Okolie · Research · HTML version: https://glintbase.dev/blog/robots-txt-said-yes-the-edge-said-no

Most agent-readiness advice starts and ends with `robots.txt`. If the file allows the agent, the site is open to agents. Or so the checklist says.

We wanted to know whether that holds at the network edge, where CDNs and bot-management products actually decide who gets a page. So we ran a pre-registered study, **The Silent Wall**, on 150 popular websites. We fetched the same public pages as three honest clients and compared what each one got back with what each site's `robots.txt` says.

The short version: **declaring ourselves an AI agent didn't cost us access, but `robots.txt` was a poor guide to what the edge would do.** On 16.8% of the sites whose `robots.txt` allowed our agent, the edge still challenged or blocked it.

> **Insight:**
>
> `robots.txt` tells you what a site says it allows. The edge decides what actually happens. On roughly one in six sites that said yes, the answer at the edge was no, and in nearly every case a headless browser was stopped too.

## What We Did

We took 150 domains from the Tranco top-sites list, 30 in each of five sectors: SaaS, e-commerce, finance, media and developer tools. A written rule fixed the selection before any data was collected.

For each site we did two things:

1. **Read the policy.** We fetched `robots.txt` once and parsed it with an RFC 9309 parser. For 14 vendor-documented crawler tokens (`GPTBot`, `ClaudeBot`, `ChatGPT-User`, `Claude-User` and others, plus `Googlebot` as a control) we recorded whether the file names the token and whether it allows `/`.
2. **Observed the behaviour.** We fetched the homepage and up to three public pages from the sitemap as three clients, in random order, from the same connection:

| Client | What it is | User-Agent |
|---|---|---|
| **agent** | HTTP client that obeys `robots.txt` | `GlintbaseResearch/1.0 (+https://glintbase.dev/research/bot)` |
| **script** | unlabelled HTTP client | library default (`python-httpx`) |
| **browser** | headless Chromium | default (`HeadlessChrome`) |

We never pretended to be OpenAI's, Anthropic's or anyone else's crawler. Edges verify those by IP range, reverse DNS or signatures. Imitating them would measure how sites treat impostors, and it isn't a defensible thing to do in a published study. Our agent declared itself honestly and linked to a [public bot page](https://glintbase.dev/research/bot) with an opt-out.

Each response was sorted into pre-registered classes: `ok`, `thin`, `challenge`, `block`, `rate_limited`, `error`, `other` or `self_excluded`. We ran two full rounds. We checked 20 results by hand against the stored evidence, and all 20 matched the rules.

## Finding 1: Saying "I'm an Agent" Didn't Cost Access

![Bar chart: homepages challenged or blocked for the declared agent 16.0%, the unlabelled script 18.7% and the headless browser 18.7%, with overlapping 95% intervals.](https://glintbase.dev/research/the-silent-wall/reports/figures/sw_fig1_identities.png)

Homepages were challenged or blocked for:

- the declared agent on **16.0%** of sites (95% CI 11.0–22.7);
- the unlabelled script on **18.7%** (13.2–25.7);
- the headless browser on **18.7%** (13.2–25.7).

None of the paired differences is statistically significant (McNemar p = 0.45 for agent vs. browser, 0.29 for agent vs. script). The second round repeated these results almost exactly: outcomes agreed for 96.7–98.0% of domains, depending on the client.

So, for a new, honestly labelled agent, the label itself wasn't what got it stopped. Edges that stopped our agent mostly stopped the other two clients as well. On 19 of the 33 sites that challenged or blocked anyone, all three were stopped.

## Finding 2: The Policy Said Yes, the Edge Said No

Among the 137 reachable sites whose `robots.txt` allowed our agent at `/`, the edge still **challenged or blocked it on 16.8%** (11.5–23.9).

That gap is real, but it is mostly not aimed at AI. In nearly all of those cases the headless browser was stopped too. It got the page on only **2 sites (1.5%)**. These edges are stopping non-human clients in general, whatever `robots.txt` says.

A few responses stood out. 12 sites refused our agent even its `robots.txt` (HTTP 403), so the file meant to say what's allowed couldn't be read by the client it was written for. One site answered **402 Payment Required**, and one returned a vendor-specific **498** block code.

> **Research note:**
>
> A `robots.txt` audit tells you what a site intends. It can't tell you what an agent will receive. If a readiness check reads only the policy, a site can pass while its edge turns agents away.

## Finding 3: Where You Host Matters More Than Who You Are

![Grouped bar chart of homepages challenged or blocked by sector and by inferred edge vendor, with three bars per group for agent, script and browser.](https://glintbase.dev/research/the-silent-wall/reports/figures/sw_fig3_breakdown.png)

Blocking rates differed far more by sector and edge vendor than by client identity. For the declared agent:

| Sector | Challenged or blocked |
|---|---|
| e-commerce | 30.0% |
| media | 23.3% |
| SaaS | 13.3% |
| finance | 6.7% |
| developer tools | 6.7% |

By edge vendor, inferred from response headers, the rate ranged from 33.3% on Akamai-fronted sites (27 sites) to 0% on Vercel (7 sites). Within each group, all three clients were treated much the same.

These cells are small, between 3 and 46 sites, so read them as descriptive. They also describe how those vendors' customers configure their defences, not the vendors' products as such.

## Finding 4: Policies Single Out Training, Not Agents

![Bar chart of robots.txt disallow rates at / for 14 crawler tokens, coloured by purpose.](https://glintbase.dev/research/the-silent-wall/reports/figures/sw_fig2_robots.png)

Of the 140 sites with a fetchable `robots.txt`:

- **23.6%** disallow at least one AI training crawler at `/`. The most common were `Bytespider` (19.3%), `CCBot` (18.6%) and `ClaudeBot` (17.1%).
- **13.6%** disallow at least one AI search crawler.
- **12.1%** disallow at least one user-initiated agent such as `ChatGPT-User` or `Claude-User`.
- **0%** disallow `Googlebot`.

The written intent is narrower than "block AI". About twice as many sites disallow a training crawler as disallow an agent that fetches a page because a person asked it to.

## How This Fits With Other Measurements

Another study, published by PageLens on 12 October, ran a similar test on the same Tranco list. It took a different approach: it sent requests *named after* well-known AI crawlers such as `GPTBot` and `ChatGPT-User`. It found that 9.3% of the 740 sites it could reach turned those requests away while letting a browser and a made-up crawler name through.

The two results fit together. Using a famous crawler's name from the wrong network gets you treated as an impostor, and some edges block impostors while letting the real, verified crawler through. PageLens notes that from the outside this looks the same as blocking AI. A new agent that declares its own name doesn't carry that baggage: in our data it was treated like any other non-browser client.

For agent builders, the practical upshot is to identify yourself honestly, and to use a verifiable identity (published IP ranges or Web Bot Auth signatures) if you want to be treated better than an anonymous script.

## What Site Owners Should Check

If you want agents to be able to read your public pages:

1. **Test the edge, not just the file.** Fetch a few public pages with a plain HTTP client and a headless browser, not only a desktop browser. If they get challenges or 403s, your `robots.txt` isn't the policy that counts.
2. **Make sure `robots.txt` itself is reachable.** If your edge blocks non-browser clients from reading `robots.txt`, well-behaved agents can't learn your rules.
3. **Decide what you're blocking.** If your aim is to stop training crawlers, say so for those tokens. Bot rules that block every non-browser client also block the user-initiated agents your customers send.
4. **Allow verified agents explicitly.** Most bot-management products can allow-list verified bots. That's the reliable way to let a known agent in while still stopping impostors.

We're adding a policy-versus-enforcement check to the Agent Readiness Score. It will flag sites whose `robots.txt` allows agents while the edge challenges or blocks them. It's planned for the next ARS release, and we'll document it on the [methodology page](https://glintbase.dev/methodology) when it ships.

## Limitations

- **One vantage point.** All requests came from one residential connection in Nigeria. Edge decisions depend on IP reputation and geography. Comparisons *between* the three clients are controlled because they shared the IP, but the absolute rates may differ elsewhere. Nine domains were unreachable for every client from this location.
- **150 domains.** Sector and vendor cells are small.
- **Headless means headless.** Our browser was identifiable as headless Chrome, so its results aren't a human visitor's.
- **Verified crawlers aren't covered.** We measured how sites treat a declared but unverified agent, which is what most independent agents look like today. We didn't measure how they treat OpenAI's or Anthropic's verified crawlers.
- **Policy as written.** We recorded `robots.txt` as written. OpenAI and Anthropic say their user-initiated agents may not follow it.
- **One reviewer.** The 20-result hand check was done by the person who wrote the harness.

## Read the Full Study

- **Report:** [The Silent Wall (2026)](https://glintbase.dev/research/the-silent-wall)
- **Pre-registration, data and harness:** [github.com/Glintbase/research](https://github.com/Glintbase/research/tree/main/benchmarks/the-silent-wall)
- **Our crawler:** [GlintbaseResearch bot page](https://glintbase.dev/research/bot)

*Disclosure: Glintbase sells agent-readiness scoring (ARS). This study was pre-registered before data collection, and deviations from the plan are logged in the pre-registration.*

> Check your own site's Agent Readiness Score for free at https://scan.glintbase.dev