A website can now tell AI systems what they may do with its content. Content Signals split that into three separate permissions: search (index the page and show links), AI input (use it live to ground a chatbot's answer), and AI training (train or fine-tune a model on it). A site can write them in robots.txt, or send them in a Content-Signal HTTP header.
So we asked a simple question: who actually uses them, and when a site states them in two places, do the two agree?
In a pre-registered study, AI Use Permissions on the Web, we checked 1,121 sites: the 103 companies on our agent-readiness leaderboard, the 150 popular sites from our Silent Wall study, and the Tranco top 1,000. For each, we read robots.txt, then fetched the homepage twice: once as a normal browser would, and once the way an AI agent does, asking for Markdown. 754 sites answered.
Almost nobody says anything
93.5% of sites state no Content Signal at all, in either place. Only 6.5% do: 4.2% in robots.txt and 2.9% in a header.
The exception is the companies building for AI agents. 22.5% of our leaderboard states a signal, against 3.9% of the Tranco top 1,000. That's almost six times as often. It isn't about size: within the top 1,000, the top 100 sites (4.8%) are no more likely to state signals than ranks 501–1,000 (3.6%).
Every header said the same thing
Here's the part that surprised us. 22 sites send a Content-Signal header. All 22 send exactly the same value:
Content-Signal: ai-input=yes, ai-train=yes, search=yes
21 of them are on Cloudflare, and 21 send the header only on the Markdown version of the page, the one AI agents ask for.
That matches Cloudflare's documentation for its Markdown for Agents feature. When a client asks for Markdown and the site's own server doesn't set a content-signal header, Cloudflare converts the page and, in its own words, "adds a default Content-Signal: ai-train=yes, search=yes, ai-input=yes".
We can't tell from outside whether any of these sites chose that value. But for 17 of the 22, this header is the only place they state any position on AI use. Their one public statement about AI training may be a default they never set.
Two answers to the same question
Of the 55 sites that serve Markdown to agents, 40% give a different answer on the Markdown response than on the normal page. Almost always that's because the Markdown response carries a signal and the HTML page carries none.
So an AI agent asking for Markdown is told "training allowed". A person, a search engine, or an agent fetching the normal page is told nothing.
Only 5 sites state signals in both robots.txt and a header. Four agree. One contradicts itself: its robots.txt says ai-train=no, while its Markdown response says ai-train=yes. We're not naming it, because everything points to a CDN default the site never chose rather than a decision it made.
When sites write their own policy, training is the fault line
The 32 sites that put Content Signals in their own robots.txt agree on almost everything except training:
- Search is universally allowed. Every site that mentions
searchsays yes. - AI training splits them. Of the 29 that mention training, 16 allow it and 13 refuse it. Seven of those 13 still allow AI input, so they're fine with an agent reading the page to answer a question, just not with a model being trained on it.
- Some go further, crawler by crawler. A handful let assistant and search crawlers such as
ChatGPT-UserandPerplexityBotin for search and AI input, while refusingBytespidereverything. - Five sites use a signal that doesn't exist. They add
use=reference, which the Content Signals spec doesn't define, so a system that follows the spec will ignore it.
Check your own site in 30 seconds
If you're on Cloudflare with Markdown for Agents turned on, your Markdown responses probably tell AI systems that training is allowed, whatever your robots.txt says. Two commands show you:
curl -s https://your-site.com/robots.txt | grep -i content-signal
curl -sI -H "Accept: text/markdown" https://your-site.com/ | grep -i content-signal
If the answers disagree, or the second one says ai-train=yes and you didn't decide that, set a Content-Signal header at your origin server. Cloudflare keeps the origin's value instead of adding its default.
And decide on training on purpose. It's the one use that sites genuinely disagree about. Leaving it unstated while a CDN states it for you is exactly the gap this study found.
What this doesn't tell you
Content Signals are preferences. We didn't test whether any AI company honours them, and nobody has shown that they do. We fetched only homepages and up to two docs pages per site, from one location. And with only five sites stating signals in two places, we can't say how common outright contradictions are; we can only say that the setup for them is everywhere.
The full report, with every number, its confidence interval, the per-site data and the code to re-run it, is at glintbase.dev/research/ai-use-permissions.
Glintbase sells Agent Readiness Score audits. Content Signals aren't part of the current score; a reported, unscored check based on this study is planned for ARS 3.4.


