All research
Final report · pre-registered

AI Use Permissions on the Web (2026)

What 754 websites tell AI systems about using their content, in robots.txt, in headers and in Markdown

Glintbase Research ·

Abstract

Content Signals let a website say, separately, whether its content may be used to build a search index, as live AI input (for example retrieval for a chatbot answer), or for AI training. A site can state them in robots.txt or send them in a Content-Signal HTTP response header.

We checked 1,121 sites: the Glintbase agent-readiness leaderboard (103 companies building for AI agents), The Silent Wall's 150 popular domains, and the Tranco top 1,000. For each we compared what robots.txt says with what the homepage's headers say, both as HTML and when an AI agent asks for Markdown. 754 sites were reachable.

93.5% state no Content Signal anywhere. Companies building for AI agents state them almost six times as often (22.5% of the leaderboard against 3.9% of the Tranco top 1,000). Every one of the 22 header signals we saw is the same value, ai-input=yes, ai-train=yes, search=yes; 21 come from one CDN and 21 appear only on Markdown responses, matching the default Cloudflare documents for Markdown for Agents. On 40% of the sites that serve Markdown, an AI agent and a person are told different things about the same page.

93.5%
state no signal
of 754 reachable sites · CI 91.5–95.0
22.5%
of AI-agent companies do
leaderboard, vs 3.9% of the Tranco top 1,000
22 of 22
headers are one value
ai-input, ai-train and search all yes
40.0%
Markdown says something else
of 55 sites serving Markdown · CI 28.1–53.2
Contents
  1. 1. What Content Signals are
  2. 2. Method
  3. 3. Almost nobody states them
  4. 4. The header is one value, from one place
  5. 5. Two statements about one page
  6. 6. What sites write themselves
  7. 7. The leaderboard companies that state signals
  8. 8. What this means
  9. 9. What this can't claim
  10. 10. Validation and reproduction
  11. → Data, code & links

1. What Content Signals are

Content Signals were introduced by Cloudflare in 2025 and are documented at contentsignals.org. They extend robots.txt with a Content-Signal line that grants (yes) or withholds (no) three uses. A missing signal neither grants nor restricts.

SignalMeaning (contentsignals.org)
searchBuilding a search index and providing search results (links and short excerpts). Does not include AI-generated search summaries.
ai-inputInputting content into AI models, e.g. retrieval augmented generation, grounding, or real-time use for generative AI search answers.
ai-trainTraining or fine-tuning AI models.

The directive can apply to all crawlers (User-Agent: *), to named crawlers, or to paths (Content-Signal: /blog/ ai-train=no, search=yes, ai-input=no). Like the rest of robots.txt, it states a preference; not every automated system honours it.

The response header

Cloudflare also sends signals as an HTTP header. Its Markdown for Agents documentation (last updated 2026-11-09) says that when a client asks for Accept: text/markdown, Cloudflare can convert the page to Markdown, and that when the origin response has no content-signal header, it adds a default Content-Signal: ai-train=yes, search=yes, ai-input=yes. An origin-set header is kept instead. The documentation describes no other way to change or remove the default.

Why we measured it

A vendor study by AI Ranked (2026-10-28, 285 high-traffic sites) found 28 homepages serving Markdown, 15 with the default ai-train=yes header, including one site whose robots.txt refuses training. We asked the same question on a defined, larger population with a pre-registered, open method: how often do sites state Content Signals at all, and how often do the two places disagree?

2. Method

  • Sample, 1,121 sites: the Glintbase leaderboard (103 hosts, read 2026-11-04), The Silent Wall's 150 domains, and the Tranco top 1,000 (list 4672X), de-duplicated. 754 were reachable. The other 367 are mostly Tranco infrastructure hosts (CDN, DNS and API domains) that serve no website, plus 37 sites whose robots.txt disallows our crawler from the homepage.
  • At most 8 requests per site: robots.txt; the homepage as HTML; the homepage with Accept: text/markdown; a Markdown twin if one is advertised; and up to 2 linked docs pages, each as HTML and as Markdown.
  • Conduct: every request identified itself as GlintbaseResearch/1.0 (+https://glintbase.dev/research/bot), obeyed robots.txt for that name, and sent at most one request per second to a site. Public pages only.
  • What counts: the robots.txt policy is the User-Agent: * group's signals; signals for named crawlers are reported separately. A header signal is any Content-Signal header on the homepage HTML, its Markdown response, or a docs response.
  • Classes: none, robots.txt only, header only, both agreeing, both conflicting.
  • When: from one residential connection in Nigeria. Network interruptions meant affected sites were re-crawled under a rule fixed in advance, so each site's data is from 2026-11-09 or 2026-11-13/14 (UTC). The pre-registration's Deviations section records exactly which.

3. Almost nobody states them

Where 754 reachable sites state Content Signals
ClassSitesShare
No signal anywhere70593.5% [91.5–95.0]
robots.txt only273.6% [2.5–5.2]
Response header only172.3% [1.4–3.6]
Both, agreeing40.5% [0.2–1.4]
Both, conflicting10.1% [0.0–0.7]

Overall, 6.5% [5.0–8.5] state any signal: 4.2% in robots.txt, 2.9% in a header. Brackets are Wilson 95% intervals.

By sample
SampleReachableAny signalrobots.txtHeaderServes Markdown
Glintbase leaderboard10222.5% [15.5–31.6]17.6%8.8%31.4%
The Silent Wall's 1501378.0% [4.5–13.8]5.1%3.6%6.6%
Tranco top 1,0006393.9% [2.7–5.7]2.2%2.0%3.4%

The samples overlap, so rows don't add up to the total.

Companies building for AI agents state signals almost six times as often as the wider web, and serve Markdown nine times as often. Size doesn't explain it: within the Tranco top 1,000, the top 100 (4.8%), ranks 101–500 (4.1%) and ranks 501–1,000 (3.6%) are indistinguishable.

5. Two statements about one page

55 sites (7.3% [5.6–9.4]) return Markdown when an agent asks for it. On 21 of them the Markdown response carries a Content-Signal header; only 1 sends one on the HTML page.

On 40.0% [28.1–53.2] of these sites, the Markdown response and the HTML page state different signals, almost always because only the Markdown response states any. An agent asking for Markdown is told "training allowed". A person, a search crawler, or an agent fetching HTML is told nothing.

Only 5 sites state signals in both `robots.txt` and a header. 4 agree. 1 conflicts: its robots.txt says ai-train=no, and its Markdown response says ai-train=yes. We don't name it; the evidence points to a CDN default the site likely never chose. With 5 cases, the conflict rate among them can't be estimated usefully.

Markdown for agents, more broadly

  • 55 sites serve Markdown from the homepage and 58 on at least one linked docs page.
  • 96.4% of sites serving Markdown send Vary: Accept, so caches keep HTML and Markdown apart.
  • 30 sites (4.0%) advertise a Markdown version with <link rel="alternate" type="text/markdown"> or a Link header.
  • Markdown comes mostly from Cloudflare (30 sites) and Vercel (15). 33 of the 55 serve it with no signal at all.

6. What sites write themselves

81.3% of reachable sites serve a robots.txt file. 32 of them contain Content-Signal lines.

General policies (User-Agent: *) among the 32
PolicySites
Allow all three: ai-input=yes, ai-train=yes, search=yes15
Search and AI input, no training: ai-input=yes, ai-train=no, search=yes6
ai-train=no, search=yes, use=reference4
Search only: ai-input=no, ai-train=no, search=yes2
ai-input=yes, search=yes (training unstated)1
ai-input=yes, ai-train=no, search=yes, use=reference1
Path-by-path signals plus allow all three1
Named crawlers only, no general policy2
  • Training is the dividing line. Of the 29 general policies that mention training, 16 allow it and 13 refuse it. All 30 that mention search allow it, and 7 of the 13 that refuse training still allow AI input.
  • `use=reference` isn't a defined signal. 5 sites add a use key that contentsignals.org doesn't define, so spec-following systems will ignore it.
  • Different rules for different crawlers. 8 sites set signals for named crawlers; 5 refuse at least one crawler something and 3 set different policies per crawler. Assistant and search crawlers (ChatGPT-User, OAI-SearchBot, Claude-User, PerplexityBot) get search and ai-input; Bytespider is refused all three on 3 of the 4 sites that name it.
Not pre-registered

This chapter, the first table under "The header is one value", the popularity comparison and the leaderboard list are descriptive breakdowns added after the pre-registered analysis. They count the same per-site fields and re-classify nothing (data/descriptives.json).

7. The leaderboard companies that state signals

Public facts: what each site's robots.txt and response headers said when we fetched them. "All three" means ai-input=yes, ai-train=yes, search=yes. The one leaderboard site whose signals conflict is not listed.

Siterobots.txt policyHeaderMarkdownCDN
arrivl.ainamed crawler only (CCBot: all three)—yesother
blackforestlabs.aiai-input=yes, search=yes—noVercel
cloudflare.comnamed crawler only (cohere-ai: all three)all threeyesCloudflare
comfy.orgall three—noCloudflare
coreweave.com—all threeyesCloudflare
cresta.com—all threeyesCloudflare
dodopayments.com—all threeyesCloudflare
elevenlabs.ioall three—yesCloudflare
exa.aiall three, plus path-by-path search=yes—yesVercel
firecrawl.devall three—yesVercel
glean.com—all threeyesCloudflare
llamaindex.aiall three—noVercel
mailtrap.ioall threeall threeyesCloudflare
nvidia.comall three—noCloudFront
ora.aiall three—yesVercel
pxxl.appai-train=no, search=yes, use=reference—noCloudflare
tastelabs.comai-train=no, search=yes, use=reference—noCloudflare
telnyx.comall threeall threeyesCloudflare
together.ai—all threeyesCloudflare
wandb.aiall three—noCloudflare
x.aiall three—noCloudflare
zernio.comai-input=yes, ai-train=no, search=yes—yesVercel

8. What this means

For site owners. If your site is on Cloudflare and Markdown for Agents is on, your Markdown responses probably tell AI systems that training is allowed, whatever your robots.txt says, unless your origin sends its own Content-Signal header. Check in one command each:

Check your own site
curl -s https://your-site.com/robots.txt | grep -i content-signal
curl -sI -H "Accept: text/markdown" https://your-site.com/ | grep -i content-signal

If the two disagree, set a Content-Signal header at your origin; Cloudflare keeps an origin value rather than adding its default. And decide on training explicitly: it's the one use sites disagree about, and leaving it unstated while a CDN states it for you is the gap this study found.

For people building agents and crawlers. A signal on a Markdown response isn't necessarily the site's own statement. Where both exist, robots.txt is where a site writes its policy deliberately.

9. What this can't claim

  • Whether any AI company honours `Content-Signal`, in either place.
  • Intent. A header may be a CDN default the site never chose.
  • The rest of a site. Only the homepage and up to 2 docs pages were fetched; a site can set different signals elsewhere.
  • Small numbers. Only 5 sites state signals in both places, so the conflict rate among them isn't estimable.
  • One vantage point. Responses can vary by location and by how a site treats bots.

10. Validation and reproduction

30 sites (seed 20261104, stratified by class, the conflict site included) were re-fetched one at a time by an independent checker and classified by hand. Agreement: 30 of 30. Founder spot-check: Victor Okolie re-checked 5 of the 30 sites (replicate.com, cloudflare.com, together.ai, twilio.com, intel.com) on 2026-11-16; all 5 matched their class (scope: class assignment).

benchmarks/content-signal-census
python ../web-census-crawler/harness/crawl.py   # robots.txt, HTML, Markdown, docs pages
python harness/analyze.py                        # per-site classes -> data/sites.jsonl
python harness/compute_stats.py                  # every pre-registered number -> data/summary.json
python harness/descriptives.py                   # the descriptive breakdowns -> data/descriptives.json