The Silent Wall (2026)

1. Why this study

Agents that act for people — research assistants, shopping agents, coding agents reading documentation — fetch ordinary web pages. Two layers decide whether they get them. The first is policy: robots.txt, where site owners say which crawlers may visit which paths. The second is enforcement at the network edge: CDNs and bot-management products such as Cloudflare, Akamai, DataDome or AWS WAF that challenge or block traffic they judge to be automated.

There is a widespread assumption that being honest is costly — that an agent which announces itself as an AI and respects robots.txt will be turned away more often than one that hides. We tested that directly, and recorded what popular sites' policies actually say about the different kinds of AI crawler.

Research questions (descriptive)

2. The sample: 150 popular websites

Source. The Tranco top-sites ranking, list `4672X` (retrieved 2026-10-12) — a research-grade ranking that averages several popularity lists and resists manipulation.

Rule. Walk Tranco from rank 1 and assign each registrable domain to one of five sectors by the primary purpose of its public homepage, until each sector has 30 domains. Only the first-ranked domain of a brand is kept (e.g. amazon.com, not amazon.de).

Excluded from all sectors: CDNs, ad-tech and analytics, DNS / registrar / hosting endpoints, app-instance hosts (such as *.force.com), government, education, non-profits and open-source foundations, adult, gambling, gaming, search engines, portals and social networks, hardware and telecom, conglomerates' generic portals, classifieds, and AI chat products.

3. Part 1 — reading the policies

Each site's robots.txt was fetched once with the research identity (≤ 5 redirects, concurrency 2, one retry after 10 s on a network failure). An HTML response or any non-200 status counts as "no robots.txt", which under RFC 9309 means everything is allowed. Files were parsed with Protego 0.7.0, an RFC 9309-compliant parser.

For each of 14 tokens documented by their vendors we recorded whether the file names the token explicitly and whether `/` is allowed for it:

Page list for Part 2. The same census also fixed which pages would be fetched: the homepage plus up to 3 same-host URLs taken in order from the site's first sitemap (the Sitemap: line in robots.txt, else /sitemap.xml, following one level of sitemap index), skipping login, cart, checkout, account and auth URLs. 83 sites had 4 pages, 2 had 3, 4 had 2, and 61 had no usable sitemap and were tested on the homepage only. Our agent's own robots.txt rules excluded 4 of those pages before any request was made.

4. Part 2 — measuring the edge

Every page was requested by three clients, from the same residential connection, so any difference between them comes from how the site treats the client, not from the network path:

• Order: the order of the three clients is randomised per site and round (seeded by SHA-256 of domain|round).
• Pacing: at most 1 request per second per client, at least 30 seconds between clients on the same site, fresh cookies for every client, up to 4 sites in parallel, and never alongside another study's traffic.
• Timeouts: 20 s for the HTTP clients, 45 s for full browser page loads.
• Stored: status code, selected headers, cookie *names*, visible-text length, page title and challenge-marker hits. Page bodies were never stored.
• Rounds: two complete rounds over all 150 sites.
• Vantage point: one residential connection (AS36873, Airtel Networks, Nigeria).

5. How responses were classified

Each request receives one outcome, by the first matching rule:

Per site, the primary outcome for each client is its homepage outcome; a secondary measure takes the worst outcome across all fetched pages (block > challenge > rate-limited > error > thin > other > ok).

Statistics. Wilson 95% intervals for proportions; McNemar's exact test for the paired comparison between clients (the same site seen by two clients); Cohen's κ for agreement between rounds.

6. Results — are honest agents treated differently? (RQ2)

Full size

• Across all fetched pages (not just the homepage), at least one page was challenged or blocked for the agent on 19.3% of sites, for the script on 22.7% and for the browser on 23.3%.
• The browser sees more content where the HTTP clients see a JavaScript shell: thin falls from 4–5% to 0.7%.

7. Results — what robots.txt says about AI agents (RQ1)

Of the 150 sites, 125 served a robots.txt, 12 refused our agent even that file with HTTP 403, 10 could not be reached from our location, and 3 returned other codes (498, 429, 404). Policies could therefore be read for 140 sites, 121 of which had a non-empty file.

Full size

8. Results — does the edge honour what the policy allows? (RQ3)

137 reachable sites have a robots.txt that allows our agent at / (or no robots.txt at all, which allows everything). On 16.8% (11.5–23.9) of them, the edge still challenged or blocked the agent's homepage request.

That looks like a contradiction between policy and enforcement — until the other clients are checked. In almost all of those cases the headless browser was stopped too; it got the page on only 2 sites (1.5%) where the agent was turned away.

9. Results — where blocking happens (RQ4)

Full size

Retail and news sites, and sites fronted by Akamai or DataDome, stop automated visitors most; developer tools, finance sites and Vercel-hosted sites least. Within every group, the honest agent fares about the same as — or better than — the anonymous clients.

10. Notable edge cases

• robots.txt refused. 12 sites answered HTTP 403 when our labelled agent asked for robots.txt itself — the one file meant to tell crawlers the rules.
• Pay to crawl. One major publisher answered our declared agent's homepage request with 402 Payment Required — consistent with a paid-crawling gate — while the unlabelled script and the headless browser were simply blocked with 403. Declaring itself got the agent an offer rather than a wall.
• Vendor-specific block codes. One large retailer's own anti-bot system answered all three clients with a non-standard 498; under the frozen rules this counts as other, although it is effectively a block.
• Login walls. One marketplace showed the headless browser a login page with enough text to count as ok under the rules.
• Unreachable from our location. Nine sites — mostly Russian banks and crypto exchanges — could not be reached by any client, including the browser, consistent with geographic blocking of our vantage point.

11. How the study actually ran

The complete record, including every problem. Times are UTC, 12–13 October 2026.

1. Oct 12 · ≈16:45
   Sample built

   Tranco list 4672X walked from rank 1 under the written rule; 150 domains fixed in data/sample/domains.csv.

2. ≈16:50
   Bot page updated

   The public research-bot page was updated to describe this study, including the plain script and headless browser, before any request.

3. ≈16:52–17:00
   First robots.txt census — too aggressive

   Run at concurrency 8, it produced 83 fetch failures out of 150; 58 of those succeeded when fetched one at a time. The failures were self-inflicted load on the home connection. Deviation 1.

4. ≈17:00–17:22
   Census re-run

   Concurrency 2 with one retry: 140 readable, 10 unreachable (consistent with geo-blocking).

5. ≈17:26
   Pre-registration frozen

   Commit 7113eda — after the census, before any Part 2 request.

6. 18:16–19:42
   Edge round 1 (first attempt)

   Probing never overlapped the other study's traffic.

7. 19:09 → after 20:10
   First connection outage

   From 19:09 every request from every client failed, including sites that had loaded moments earlier. All of round 2 (19:42–20:10) and 18 round-1 sites were discarded. Deviation 2.

8. Oct 13 · 00:51–00:57
   Connection restored; 18 sites re-probed

   Unchanged harness; a per-minute connectivity log was started.

9. ≈00:58
   Browser-measurement defects found

   A first look at round 1 showed a 22% error rate for the browser against 9–10% for the others. Cause: page loads timing out when 6 sites loaded in parallel, and Chromium hiding empty-body error responses. Fixed and tested. Deviation 3.

10. 01:00–02:01
    Round 1 re-run in full

    All three clients, all 150 sites, 4 sites in parallel. Browser error rate fell to 6.7%.

11. 02:01–04:04
    Round 2 — second outage

    The connectivity log shows the line down from 02:44 (44 of 54 checks failed). 71 affected sites discarded. Deviation 4.

12. 07:12–07:44
    71 sites re-probed

    No connection drops; round 2 reproduced the headline exactly.

13. after 07:44
    Analysis, validation, report

    Statistics computed by harness/analyze.py; 20 results validated by hand; report and figures written.

12. Validation

The pre-registration required ≥ 90% agreement on 20 stratified site × client results, checked by hand against the stored evidence: status, headers, cookie names, challenge markers, page title and text length.

Typical evidence: an Akamai "Access Denied" page behind a 403, Cloudflare's cf-mitigated: challenge header, DataDome markers behind a 401 or 403, 429 responses. Three results were correct under the frozen rules but understate what happened — the 498 vendor block and the 402 payment gate (both other) and a login wall (ok). They are reported as rule limitations, not reclassified.

13. Pre-registration deviations

The protocol was frozen in commit 7113eda after the policy census and before any edge request. Four deviations followed, each logged before the affected data was analysed.

Deviation 1 — census re-run at lower concurrency

• What happened: the first census (concurrency 8, no retry) produced 83/150 fetch failures; 58 succeeded when fetched sequentially, so the failures were self-inflicted.
• Action: re-run in full with concurrency 2 and one retry; that run is the one reported. Before freezing, only fetch-status counts and one token count were looked at.
• Result: 10 domains remained unreachable for every client, including a headless browser.

Deviation 2 — first connection outage

• What happened: the connection failed from about 19:09 UTC on 12 October until at least 20:10.
• Discarded: all of round 2 and the 18 round-1 sites where any client started at or after 19:09.
• Action: after connectivity was restored (00:51 UTC), those sites were re-probed with the unchanged harness.

Deviation 3 — browser-measurement defects fixed, both rounds re-run

• How it was found: round 1 showed a 22% homepage error rate for the browser against 9–10% for the HTTP clients.
• Defects: full page loads exceeded the 20 s limit with 6 sites in parallel (the same pages load in 5–8 s alone), and Chromium raises net::ERR_HTTP_RESPONSE_CODE_FAILURE instead of returning empty-body error responses, so blocks were recorded as errors.
• Fix: capture the main-document response even when navigation raises; 45 s browser timeout; 4 sites in parallel; store the first line of each error.
• Action: both rounds re-run in full for all three clients so conditions are identical.
• Disclosure: decided after round-1 outcome tables had been seen; classification rules and analysis unchanged.

Deviation 4 — second connection outage

• What happened: the per-minute connectivity log shows the line down from 02:44 UTC on 13 October until the round ended at 04:04.
• Discarded: the 71 round-2 sites where any client started at or after 02:44.
• Action: re-probed at 07:12–07:44 UTC with the unchanged harness; round 1 had finished before the outage.

14. Limitations

• One vantage point. A residential connection in Nigeria. Edge decisions depend on IP reputation and geography; because all three clients share the IP, comparisons *between clients* are controlled, but absolute rates may differ from elsewhere.
• Declared, not verified. The study does not measure how sites treat vendors' verified crawlers.
• 150 sites, one labeller, and a keyword screen for finance and developer-tool candidates beyond rank 1,500.
• Headless means headless. The browser announces itself as headless Chrome, so its results are not those of a human visitor.
• Policy as written. robots.txt is recorded as written; user-initiated agents may not follow it.
• Rule limitations. Vendor-specific block codes and payment gates count as other, and a login wall can count as ok.
• Interrupted runs. Two outages and a browser-measurement fix meant parts of each round ran hours apart; every affected result was re-probed and the two rounds agree on 97–98% of results.

15. Ethics

• Honest identities only; no client impersonated another company's crawler or a human visitor.
• No CAPTCHA was solved or bypassed — a challenge was recorded, never evaded.
• Public pages only; nothing behind a login.
• About 12 requests per site per round, at most 1 per second, with 30-second pauses between clients.
• A public bot page describing the study, with an opt-out address honoured within 24 hours.
• No page bodies stored. Sites are named only for public robots.txt facts, never for edge behaviour.

16. Reproducing the study

The harness, sample, page list, raw robots.txt files and per-site results are in the public repository:

python -m venv .venv && .venv/Scripts/pip install protego httpx playwright tranco
python census.py                       # Part 1: robots.txt census + page list
python edge_probe.py --round 1          # Part 2: three clients, randomised order
python edge_probe.py --round 2
python analyze.py --round 1 --retest 2  # classification + every number on this page
python figures.py                       # the three figures on this page

Edge behaviour changes over time and by location, so a re-run from another network is expected to give different absolute rates; the comparison between clients is the robust part of the design.