Contents
- 1. What Content Signals are
- 2. Method
- 3. Almost nobody states them
- 4. The header is one value, from one place
- 5. Two statements about one page
- 6. What sites write themselves
- 7. The leaderboard companies that state signals
- 8. What this means
- 9. What this can't claim
- 10. Validation and reproduction
- → Data, code & links
1. What Content Signals are
Content Signals were introduced by Cloudflare in 2025 and are documented at contentsignals.org. They extend robots.txt with a Content-Signal line that grants (yes) or withholds (no) three uses. A missing signal neither grants nor restricts.
| Signal | Meaning (contentsignals.org) |
|---|---|
search | Building a search index and providing search results (links and short excerpts). Does not include AI-generated search summaries. |
ai-input | Inputting content into AI models, e.g. retrieval augmented generation, grounding, or real-time use for generative AI search answers. |
ai-train | Training or fine-tuning AI models. |
The directive can apply to all crawlers (User-Agent: *), to named crawlers, or to paths (Content-Signal: /blog/ ai-train=no, search=yes, ai-input=no). Like the rest of robots.txt, it states a preference; not every automated system honours it.
The response header
Cloudflare also sends signals as an HTTP header. Its Markdown for Agents documentation (last updated 2026-11-09) says that when a client asks for Accept: text/markdown, Cloudflare can convert the page to Markdown, and that when the origin response has no content-signal header, it adds a default Content-Signal: ai-train=yes, search=yes, ai-input=yes. An origin-set header is kept instead. The documentation describes no other way to change or remove the default.
Why we measured it
A vendor study by AI Ranked (2026-10-28, 285 high-traffic sites) found 28 homepages serving Markdown, 15 with the default ai-train=yes header, including one site whose robots.txt refuses training. We asked the same question on a defined, larger population with a pre-registered, open method: how often do sites state Content Signals at all, and how often do the two places disagree?
2. Method
- Sample, 1,121 sites: the Glintbase leaderboard (103 hosts, read 2026-11-04), The Silent Wall's 150 domains, and the Tranco top 1,000 (list
4672X), de-duplicated. 754 were reachable. The other 367 are mostly Tranco infrastructure hosts (CDN, DNS and API domains) that serve no website, plus 37 sites whoserobots.txtdisallows our crawler from the homepage. - At most 8 requests per site:
robots.txt; the homepage as HTML; the homepage withAccept: text/markdown; a Markdown twin if one is advertised; and up to 2 linked docs pages, each as HTML and as Markdown. - Conduct: every request identified itself as
GlintbaseResearch/1.0 (+https://glintbase.dev/research/bot), obeyedrobots.txtfor that name, and sent at most one request per second to a site. Public pages only. - What counts: the
robots.txtpolicy is theUser-Agent: *group's signals; signals for named crawlers are reported separately. A header signal is anyContent-Signalheader on the homepage HTML, its Markdown response, or a docs response. - Classes: none,
robots.txtonly, header only, both agreeing, both conflicting. - When: from one residential connection in Nigeria. Network interruptions meant affected sites were re-crawled under a rule fixed in advance, so each site's data is from 2026-11-09 or 2026-11-13/14 (UTC). The pre-registration's Deviations section records exactly which.
3. Almost nobody states them
| Class | Sites | Share |
|---|---|---|
| No signal anywhere | 705 | 93.5% [91.5–95.0] |
robots.txt only | 27 | 3.6% [2.5–5.2] |
| Response header only | 17 | 2.3% [1.4–3.6] |
| Both, agreeing | 4 | 0.5% [0.2–1.4] |
| Both, conflicting | 1 | 0.1% [0.0–0.7] |
Overall, 6.5% [5.0–8.5] state any signal: 4.2% in robots.txt, 2.9% in a header. Brackets are Wilson 95% intervals.
| Sample | Reachable | Any signal | robots.txt | Header | Serves Markdown |
|---|---|---|---|---|---|
| Glintbase leaderboard | 102 | 22.5% [15.5–31.6] | 17.6% | 8.8% | 31.4% |
| The Silent Wall's 150 | 137 | 8.0% [4.5–13.8] | 5.1% | 3.6% | 6.6% |
| Tranco top 1,000 | 639 | 3.9% [2.7–5.7] | 2.2% | 2.0% | 3.4% |
The samples overlap, so rows don't add up to the total.
Companies building for AI agents state signals almost six times as often as the wider web, and serve Markdown nine times as often. Size doesn't explain it: within the Tranco top 1,000, the top 100 (4.8%), ranks 101–500 (4.1%) and ranks 501–1,000 (3.6%) are indistinguishable.
4. The header is one value, from one place
| Count | |
|---|---|
Value is ai-input=yes, ai-train=yes, search=yes | 22 of 22 |
| On Cloudflare | 21 of 22 |
| Sent only on Markdown responses | 21 of 22 · 95.5% [78.2–99.2] |
Site also states a policy in its own robots.txt | 5 of 22 |
| CDN | Sites | Header signal | Serves Markdown | Markdown with a header |
|---|---|---|---|---|
| Cloudflare | 220 | 9.5% [6.3–14.2] | 13.6% | 20 of 30 |
| Other / unidentified | 259 | 0.4% [0.1–2.2] | 2.3% | 1 of 6 |
| CloudFront | 100 | 0 | 4.0% | 0 of 4 |
| Akamai | 66 | 0 | 0 | — |
| Fastly | 60 | 0 | 0 | — |
| Vercel | 43 | 0 | 34.9% | 0 of 15 |
CDN identified from response headers.
Every header signal we saw matches the default Cloudflare documents for Markdown for Agents, and appears only where that feature applies. We can't see from outside whether a site chose it. But for 17 of the 22 sites, that header is the only place any Content Signal appears, so their only stated position on AI training is one they may never have set.
5. Two statements about one page
55 sites (7.3% [5.6–9.4]) return Markdown when an agent asks for it. On 21 of them the Markdown response carries a Content-Signal header; only 1 sends one on the HTML page.
On 40.0% [28.1–53.2] of these sites, the Markdown response and the HTML page state different signals, almost always because only the Markdown response states any. An agent asking for Markdown is told "training allowed". A person, a search crawler, or an agent fetching HTML is told nothing.
Only 5 sites state signals in both `robots.txt` and a header. 4 agree. 1 conflicts: its robots.txt says ai-train=no, and its Markdown response says ai-train=yes. We don't name it; the evidence points to a CDN default the site likely never chose. With 5 cases, the conflict rate among them can't be estimated usefully.
Markdown for agents, more broadly
- 55 sites serve Markdown from the homepage and 58 on at least one linked docs page.
- 96.4% of sites serving Markdown send
Vary: Accept, so caches keep HTML and Markdown apart. - 30 sites (4.0%) advertise a Markdown version with
<link rel="alternate" type="text/markdown">or aLinkheader. - Markdown comes mostly from Cloudflare (30 sites) and Vercel (15). 33 of the 55 serve it with no signal at all.
6. What sites write themselves
81.3% of reachable sites serve a robots.txt file. 32 of them contain Content-Signal lines.
| Policy | Sites |
|---|---|
Allow all three: ai-input=yes, ai-train=yes, search=yes | 15 |
Search and AI input, no training: ai-input=yes, ai-train=no, search=yes | 6 |
ai-train=no, search=yes, use=reference | 4 |
Search only: ai-input=no, ai-train=no, search=yes | 2 |
ai-input=yes, search=yes (training unstated) | 1 |
ai-input=yes, ai-train=no, search=yes, use=reference | 1 |
| Path-by-path signals plus allow all three | 1 |
| Named crawlers only, no general policy | 2 |
- Training is the dividing line. Of the 29 general policies that mention training, 16 allow it and 13 refuse it. All 30 that mention
searchallow it, and 7 of the 13 that refuse training still allow AI input. - `use=reference` isn't a defined signal. 5 sites add a
usekey that contentsignals.org doesn't define, so spec-following systems will ignore it. - Different rules for different crawlers. 8 sites set signals for named crawlers; 5 refuse at least one crawler something and 3 set different policies per crawler. Assistant and search crawlers (
ChatGPT-User,OAI-SearchBot,Claude-User,PerplexityBot) getsearchandai-input;Bytespideris refused all three on 3 of the 4 sites that name it.
This chapter, the first table under "The header is one value", the popularity comparison and the leaderboard list are descriptive breakdowns added after the pre-registered analysis. They count the same per-site fields and re-classify nothing (data/descriptives.json).
7. The leaderboard companies that state signals
Public facts: what each site's robots.txt and response headers said when we fetched them. "All three" means ai-input=yes, ai-train=yes, search=yes. The one leaderboard site whose signals conflict is not listed.
| Site | robots.txt policy | Header | Markdown | CDN |
|---|---|---|---|---|
| arrivl.ai | named crawler only (CCBot: all three) | — | yes | other |
| blackforestlabs.ai | ai-input=yes, search=yes | — | no | Vercel |
| cloudflare.com | named crawler only (cohere-ai: all three) | all three | yes | Cloudflare |
| comfy.org | all three | — | no | Cloudflare |
| coreweave.com | — | all three | yes | Cloudflare |
| cresta.com | — | all three | yes | Cloudflare |
| dodopayments.com | — | all three | yes | Cloudflare |
| elevenlabs.io | all three | — | yes | Cloudflare |
| exa.ai | all three, plus path-by-path search=yes | — | yes | Vercel |
| firecrawl.dev | all three | — | yes | Vercel |
| glean.com | — | all three | yes | Cloudflare |
| llamaindex.ai | all three | — | no | Vercel |
| mailtrap.io | all three | all three | yes | Cloudflare |
| nvidia.com | all three | — | no | CloudFront |
| ora.ai | all three | — | yes | Vercel |
| pxxl.app | ai-train=no, search=yes, use=reference | — | no | Cloudflare |
| tastelabs.com | ai-train=no, search=yes, use=reference | — | no | Cloudflare |
| telnyx.com | all three | all three | yes | Cloudflare |
| together.ai | — | all three | yes | Cloudflare |
| wandb.ai | all three | — | no | Cloudflare |
| x.ai | all three | — | no | Cloudflare |
| zernio.com | ai-input=yes, ai-train=no, search=yes | — | yes | Vercel |
8. What this means
For site owners. If your site is on Cloudflare and Markdown for Agents is on, your Markdown responses probably tell AI systems that training is allowed, whatever your robots.txt says, unless your origin sends its own Content-Signal header. Check in one command each:
curl -s https://your-site.com/robots.txt | grep -i content-signal curl -sI -H "Accept: text/markdown" https://your-site.com/ | grep -i content-signal
If the two disagree, set a Content-Signal header at your origin; Cloudflare keeps an origin value rather than adding its default. And decide on training explicitly: it's the one use sites disagree about, and leaving it unstated while a CDN states it for you is the gap this study found.
For people building agents and crawlers. A signal on a Markdown response isn't necessarily the site's own statement. Where both exist, robots.txt is where a site writes its policy deliberately.
9. What this can't claim
- Whether any AI company honours `Content-Signal`, in either place.
- Intent. A header may be a CDN default the site never chose.
- The rest of a site. Only the homepage and up to 2 docs pages were fetched; a site can set different signals elsewhere.
- Small numbers. Only 5 sites state signals in both places, so the conflict rate among them isn't estimable.
- One vantage point. Responses can vary by location and by how a site treats bots.
10. Validation and reproduction
30 sites (seed 20261104, stratified by class, the conflict site included) were re-fetched one at a time by an independent checker and classified by hand. Agreement: 30 of 30. Founder spot-check: Victor Okolie re-checked 5 of the 30 sites (replicate.com, cloudflare.com, together.ai, twilio.com, intel.com) on 2026-11-16; all 5 matched their class (scope: class assignment).
python ../web-census-crawler/harness/crawl.py # robots.txt, HTML, Markdown, docs pages python harness/analyze.py # per-site classes -> data/sites.jsonl python harness/compute_stats.py # every pre-registered number -> data/summary.json python harness/descriptives.py # the descriptive breakdowns -> data/descriptives.json
Data, code & links
Everything above is in this page. For readers who want the source material, the files below are the frozen pre-registration, the full report as plain text, the computed statistics and the open repository with the harness and per-endpoint data.
- Pre-registration and deviations The frozen protocol and every logged deviation
- Full report (Markdown) The report this page is built from
- Summary statistics Every pre-registered number with its interval
- Per-site results One row per site: class, CDN, policies, Markdown flags
- Validation results The 30 independently re-checked sites
- Crawler, harness and data (GitHub) Everything needed to re-run the census
- About the research bot What GlintbaseResearch requests and how to opt out
@techreport{glintbase2026aiusepermissions,
title = {AI Use Permissions on the Web (2026): What 754 Websites Tell AI Systems About Using Their Content},
author = {{Glintbase Research}},
institution = {Glintbase},
year = {2026},
month = {November},
type = {Research Report},
url = {https://glintbase.dev/research/ai-use-permissions}
}Agent-readable version of this report: /research/ai-use-permissions.md