---
title: "Your Markdown Says Yes to AI Training. Did You?"
description: "We checked 1,121 websites for Content Signals, the robots.txt and header lines that tell AI systems whether they may use a site's content. 93.5% say nothing. And every header signal we found was the same CDN default, telling AI agents that training is allowed."
author: "Victor Okolie"
publishedAt: "2026-11-16"
category: "Research"
tags: ["Content Signals", "robots.txt", "AI Training", "Markdown for Agents", "Glintbase Research"]
canonical: https://glintbase.dev/blog/your-markdown-says-yes-to-ai-training
---

# Your Markdown Says Yes to AI Training. Did You?

> We checked 1,121 websites for Content Signals, the robots.txt and header lines that tell AI systems whether they may use a site's content. 93.5% say nothing. And every header signal we found was the same CDN default, telling AI agents that training is allowed.

Published 2026-11-16 by Victor Okolie · Research · HTML version: https://glintbase.dev/blog/your-markdown-says-yes-to-ai-training

A website can now tell AI systems what they may do with its content. **Content Signals** split that into three separate permissions: **search** (index the page and show links), **AI input** (use it live to ground a chatbot's answer), and **AI training** (train or fine-tune a model on it). A site can write them in `robots.txt`, or send them in a `Content-Signal` HTTP header.

So we asked a simple question: who actually uses them, and when a site states them in two places, do the two agree?

In a pre-registered study, **[AI Use Permissions on the Web](https://glintbase.dev/research/ai-use-permissions)**, we checked **1,121 sites**: the 103 companies on our agent-readiness leaderboard, the 150 popular sites from our [Silent Wall](https://glintbase.dev/research/the-silent-wall) study, and the Tranco top 1,000. For each, we read `robots.txt`, then fetched the homepage twice: once as a normal browser would, and once the way an AI agent does, asking for Markdown. **754 sites** answered.

## Almost nobody says anything

**93.5% of sites state no Content Signal at all**, in either place. Only 6.5% do: 4.2% in `robots.txt` and 2.9% in a header.

The exception is the companies building for AI agents. **22.5% of our leaderboard** states a signal, against **3.9%** of the Tranco top 1,000. That's almost six times as often. It isn't about size: within the top 1,000, the top 100 sites (4.8%) are no more likely to state signals than ranks 501–1,000 (3.6%).

## Every header said the same thing

Here's the part that surprised us. **22 sites** send a `Content-Signal` header. **All 22 send exactly the same value:**

```
Content-Signal: ai-input=yes, ai-train=yes, search=yes
```

21 of them are on Cloudflare, and 21 send the header **only on the Markdown version** of the page, the one AI agents ask for.

That matches Cloudflare's documentation for its Markdown for Agents feature. When a client asks for Markdown and the site's own server doesn't set a `content-signal` header, Cloudflare converts the page and, in its own words, "adds a default `Content-Signal: ai-train=yes, search=yes, ai-input=yes`".

We can't tell from outside whether any of these sites chose that value. But for **17 of the 22**, this header is the **only** place they state any position on AI use. Their one public statement about AI training may be a default they never set.

## Two answers to the same question

Of the 55 sites that serve Markdown to agents, **40% give a different answer on the Markdown response than on the normal page**. Almost always that's because the Markdown response carries a signal and the HTML page carries none.

So an AI agent asking for Markdown is told "training allowed". A person, a search engine, or an agent fetching the normal page is told nothing.

Only 5 sites state signals in both `robots.txt` and a header. Four agree. One contradicts itself: its `robots.txt` says `ai-train=no`, while its Markdown response says `ai-train=yes`. We're not naming it, because everything points to a CDN default the site never chose rather than a decision it made.

## When sites write their own policy, training is the fault line

The 32 sites that put Content Signals in their own `robots.txt` agree on almost everything except training:

- **Search is universally allowed.** Every site that mentions `search` says yes.
- **AI training splits them.** Of the 29 that mention training, 16 allow it and 13 refuse it. Seven of those 13 still allow AI input, so they're fine with an agent reading the page to answer a question, just not with a model being trained on it.
- **Some go further, crawler by crawler.** A handful let assistant and search crawlers such as `ChatGPT-User` and `PerplexityBot` in for search and AI input, while refusing `Bytespider` everything.
- **Five sites use a signal that doesn't exist.** They add `use=reference`, which the Content Signals spec doesn't define, so a system that follows the spec will ignore it.

## Check your own site in 30 seconds

If you're on Cloudflare with Markdown for Agents turned on, your Markdown responses probably tell AI systems that training is allowed, whatever your `robots.txt` says. Two commands show you:

```bash
curl -s https://your-site.com/robots.txt | grep -i content-signal
curl -sI -H "Accept: text/markdown" https://your-site.com/ | grep -i content-signal
```

If the answers disagree, or the second one says `ai-train=yes` and you didn't decide that, set a `Content-Signal` header at your origin server. Cloudflare keeps the origin's value instead of adding its default.

And decide on training on purpose. It's the one use that sites genuinely disagree about. Leaving it unstated while a CDN states it for you is exactly the gap this study found.

## What this doesn't tell you

Content Signals are preferences. We didn't test whether any AI company honours them, and nobody has shown that they do. We fetched only homepages and up to two docs pages per site, from one location. And with only five sites stating signals in two places, we can't say how common outright contradictions are; we can only say that the setup for them is everywhere.

The full report, with every number, its confidence interval, the per-site data and the code to re-run it, is at **[glintbase.dev/research/ai-use-permissions](https://glintbase.dev/research/ai-use-permissions)**.

*Glintbase sells Agent Readiness Score audits. Content Signals aren't part of the current score; a reported, unscored check based on this study is planned for ARS 3.4.*
