Opt out of AI training in robots.txt

What the AI training opt-out in robots.txt does

To opt out of AI training in robots.txt and still stay in search, add a group whose user agent is __usage_ai-training and give it Disallow: /. That group states a preference about how your content is used. It does not stop any crawler from fetching your pages, so search engines can still read and cite them.

This is the opposite job from blocking AI crawlers on your server. Blocking stops the fetch. The training token lets the fetch happen and records what you want done with the content afterwards. If you want a bot to stop requesting your pages, follow that guide instead.

The token comes from an IETF (Internet Engineering Task Force) Internet-Draft, draft-kelley-aipref-training-preference-signal-00. Its author is Kevin Kelley of Anthropic, and it is dated 30 October 2026. It is an individual draft. Its Datatracker page says it is not endorsed by the IETF and has no formal standing. Version -00 expires on 3 May 2027. Before you rely on the details below, check the Datatracker for a later revision.

How the __usage_ai-training group works

The draft calls __usage_ai-training a policy token. You write it in a User-Agent line, where a crawler name normally goes. This is Figure 1 from the draft:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

A consumer is a company that reads your file before it trains a model. It looks up a path in this group with the normal RFC 9309 rules. RFC 9309 is the Robots Exclusion Protocol, the standard that defines robots.txt. The draft keeps its path language and its longest-match precedence. A disallowed path expresses a preference against AI training. In the draft's words, "If it finds that the path is allowed, a preference in favor of that use has been expressed." So Figure 1 says: do not train on anything except the press kit.

Four rules from section 4 of the draft change how you should read your own file:

• Consumers must match the token exactly, apart from case.
• "Wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token."
• If no group contains __usage_ai-training, no preference has been expressed.
• "This policy token is not a directive for crawling behavior." A consumer may use the preference to decide what to fetch, but it does not have to.

The second and third rules have a consequence that surprises people. A file that holds only User-agent: * and Disallow: / keeps polite crawlers out. Under the draft, though, that file says nothing about training at all. If you mean both things, write both groups.

The draft also lets one group name several policy tokens, and lets a policy token share a group with ordinary crawler names. Separate groups are easier to read and easier to audit, so the examples here keep the token in its own group.

Why a stock robots.txt parser can misread the token

RFC 9309 tells a crawler what to do when no group names it: "If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present." The draft says the opposite for the policy token. So a parser written for RFC 9309, with no special case for __usage_ai-training, answers the question with the wrong group.

Python's standard library parser shows this. In the CPython source (the main and 3.13 branches, read on 1 November 2026), the method that picks a group is _find_entry. It first tries an exact name, then a substring match, and its last line is return self.groups.get('*'). That line is the RFC 9309 fallback. Run this on your own machine to see what your installed Python does with a wildcard-only file:

python3 -I - <<'EOF'
import sys
from urllib.robotparser import RobotFileParser
print(sys.version.split()[0])
rp = RobotFileParser()
rp.parse(["User-agent: *", "Disallow: /"])
print(rp.can_fetch("__usage_ai-training", "https://example.com/blog/"))
EOF

The first line of output is your Python version. If the second line prints False, the parser applied the * rules to the token. Under the draft that reading is wrong, because this file expresses no training preference. So do not audit your training preference with a stock robots.txt library. Check that the token group exists by name instead:

curl -s https://example.com/robots.txt | grep -i -A3 '__usage_ai-training'

You should see the User-Agent: __usage_ai-training line and the rules under it. No output means the group is missing, which means the draft treats your site as having said nothing.

A complete robots.txt that stays in search and opts out of training

This file is for a site that wants to appear in search and in ChatGPT search answers, but does not want its content used for model training. Put it at the root of each host, so it is served at https://example.com/robots.txt. Robots.txt applies per host, and Anthropic's ClaudeBot help page says to repeat the rules on every subdomain. If you are still setting up the server itself, start with self-hosting a website on a VPS and come back.

# Shared training preference (draft-kelley-aipref-training-preference-signal-00)
User-Agent: __usage_ai-training
Allow: /press-kit/
Disallow: /

# OpenAI training crawler
User-agent: GPTBot
Allow: /press-kit/
Disallow: /

# Google training control (a product token, not a separate crawler)
User-agent: Google-Extended
Allow: /press-kit/
Disallow: /

# Anthropic training crawler
User-agent: ClaudeBot
Allow: /press-kit/
Disallow: /

# Everyone else, including search crawlers
User-agent: *
Disallow: /admin/

Under RFC 9309 the longest match wins, so Allow: /press-kit/ beats Disallow: / for paths under /press-kit/ in whatever order the lines appear. Drop the Allow lines if you have no press kit.

Search crawlers fall through to the wildcard group. Googlebot, Bingbot, OAI-SearchBot and Claude-SearchBot have no group of their own in this file. So they obey the * group and can crawl everything outside /admin/. Do not add a User-agent: Googlebot group with Allow: / to make this explicit. A crawler that finds its own group ignores the * group completely, so that Googlebot group would drop your /admin/ rule for Googlebot. If you do name a search crawler, copy every * rule into its group.

The per-bot groups stay. They are each vendor's documented training control. OpenAI's crawler docs say "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." The same page says each setting is independent of the others, and that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". That is why OAI-SearchBot is left alone here. Anthropic's ClaudeBot page says ClaudeBot collects web content for possible model training, and that restricting it signals that future materials should be excluded from training datasets. Claude-User and Claude-SearchBot are separate bots with separate jobs.

Google-Extended has one trade-off. Google's common-crawlers page says "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." It controls whether crawled content may be used to train future Gemini models. Google's Search Central blog also says that sites which allow Google-Extended are eligible to appear in Gemini Apps grounding. Grounding is when an AI answer uses your page as a source. So the Google-Extended Disallow may cost you appearances in Gemini answers. The same blog says that when a site uses the shared token to opt out of training and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended. So deleting the Google-Extended group does not get you back into Gemini answers while the token group says Disallow: /. With Google, opting out of training and staying in Gemini answers are one choice, not two.

After you deploy the file, check it for real crawler names:

python3 -I - <<'EOF'
from urllib.robotparser import RobotFileParser
rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
for agent in ["GPTBot", "ClaudeBot", "Googlebot", "OAI-SearchBot"]:
    print(agent, rp.can_fetch(agent, "https://example.com/blog/"))
EOF

GPTBot and ClaudeBot should print False, because they have their own groups with Disallow: /. Googlebot and OAI-SearchBot should print True, because they fall through to the * group. If a search crawler prints False, a rule in your * group, or a group you forgot about, is blocking it. This check is valid for crawler names. It is not valid for the policy token, for the reason in the previous section.

Who has committed to honour the training token

As of 1 November 2026, the vendors' own pages show plans. None of them shows the token in effect. Treat each line below as a plan until that vendor's crawler documentation describes it as live.

• Microsoft says on the Bing blog, dated 30 October 2026: "Starting February 1, 2027, Microsoft will honor the new preferences as it crawls." Its training preference can be set with the User-Agent: __usage_ai-training group in robots.txt. Allow is the default, so you need no rule to allow training.
• OpenAI says: "We aim to implement the signal in the coming months. This new signal will not replace the controls we already provide through robots.txt." That second sentence is why the GPTBot group stays in the file.
• Google says on its Search Central blog: "We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress." Until then, Google-Extended is Google's documented training control.
• Anthropic: Kevin Kelley, the draft's author, wrote on the IETF ai-control mailing list on 30 October 2026 that "Anthropic has already started implementing support for this draft", and that it "complements our existing crawler controls rather than replacing them." That is a mailing-list post, not a help-center statement. Anthropic's ClaudeBot help page does not mention the token, so keep the ClaudeBot group.

When this guide was checked, OpenAI's crawler page, Anthropic's ClaudeBot page and Google's common-crawlers page did not mention __usage_ai-training. Read them again before you remove any per-bot group.

What the training preference cannot do

Know the limits before you rely on this file.

It is a request, not enforcement. RFC 9309 says robots.txt rules "are not a form of access authorization." Any crawler can fetch your pages and ignore every line. If you need a fetch to fail, the server has to refuse it. Blocking AI crawlers at the web server is the guide for that.

It does not reach back in time. A crawler reads robots.txt when it crawls. The file says nothing about copies that are already in a dataset. Anthropic's own wording points the same way: restricting ClaudeBot signals that future materials should be excluded.

A crawler that never implemented the token cannot see it. Such a crawler looks for its own name. If it finds no group, it falls back to *, which is the RFC 9309 behaviour shown above. For that crawler, your token group is a group for some other bot. This is why the per-bot groups are still in the file.

Page-level training opt-out for Bing: meta tag and X-Robots-Tag

Microsoft also accepts a training preference per page. In HTML, it goes inside <head>:

<meta name="Bingbot" content="ai-training:disallow">

The same preference can be sent as an HTTP response header. Your web server can add it to every response without any change to your HTML:

X-Robots-Tag: Bingbot: ai-training:disallow

Microsoft keeps training and grounding apart. Grounding has its own values, ai-grounding:disallow and ai-grounding:disallowextended, and they work only at page level. The Bing blog says "There is no site-wide robots.txt grounding preference." Grounding defaults to Allow. This guide is for a site that wants to stay in AI answers, so set only the training value and leave grounding alone.

If you are not sure which program answers requests on your box, what a web server does explains the role. The two sections below cover nginx and Caddy.

Add the header in nginx

If nginx configuration is new to you, read how nginx server and location blocks fit together first. Add one add_header line to the server block:

server {
    server_name example.com;
    # keep your existing listen, ssl and root lines here

    add_header X-Robots-Tag "Bingbot: ai-training:disallow" always;

    location / {
        try_files $uri $uri/ =404;
    }
}

The nginx docs say that with always, "the header field will be added regardless of the response code." So 404 pages carry it too.

Watch the inheritance rule. The nginx docs say add_header directives "are inherited from the previous configuration level if and only if there are no add_header directives defined on the current level." So a location block with its own add_header, for example a Cache-Control line, drops the server-level X-Robots-Tag for every request it handles. Nothing warns you. Repeat the X-Robots-Tag line inside that location. nginx 1.29.3 added an add_header_inherit directive that can change this behaviour, so run nginx -v before you rely on it.

sudo nginx -t
sudo systemctl reload nginx
curl -sI https://example.com/ | grep -i '^x-robots-tag'

nginx -t should end with test is successful. The curl line should print the header with the value Bingbot: ai-training:disallow. Over HTTP/2, curl prints the header name in lowercase, so expect x-robots-tag. Run the same curl against a path that does not exist, and against a path served by any location with its own add_header. Both should show the header.

Add the header in Caddy

For a static site served by Caddy, add a header line to the site block. Running Caddy as a reverse proxy on a VPS covers the rest of the Caddyfile.

example.com {
    root * /srv/example
    header X-Robots-Tag "Bingbot: ai-training:disallow"
    file_server
}

Caddy's header directive sets a response header field. It also accepts a request matcher before the field name, so you can limit the header to some paths if you want.

caddy version
caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy
curl -sI https://example.com/ | grep -i '^x-robots-tag'

Note the version that caddy version prints, so you know which docs apply to your setup. caddy validate should report a valid configuration. The curl line should print the header with the value Bingbot: ai-training:disallow. Also check the error pages you care about, such as a 404, and confirm the header is present on them before you rely on it.

FAQ

Does User-agent: * with Disallow: / opt my site out of AI training?

Not under the draft that defines the training token. It says a User-Agent: * group must not be read as matching the __usage_ai-training token, and that a file with no token group expresses no training preference. The wildcard group still asks polite crawlers not to fetch your pages, which also keeps you out of search. To state a training preference, add a User-Agent: __usage_ai-training group with its own Disallow lines.

Will the __usage_ai-training token remove my site from Google Search or ChatGPT search?

It is not meant to. The draft says the token "is not a directive for crawling behavior". A consumer may skip fetches made for training, but search crawling is governed by each crawler's own group or by your * group. Google says Google-Extended does not affect inclusion or ranking in Google Search, and OpenAI controls ChatGPT search with the separate OAI-SearchBot token. Keep Googlebot, Bingbot and OAI-SearchBot out of your Disallow groups and they keep crawling under your * rules.

Does any crawler honour the robots.txt AI training token yet?

As of 1 November 2026, no vendor page checked for this guide shows it in effect. Microsoft says it will honour the preference from 1 February 2027. Google says it aims to support it in 2027. OpenAI says it aims to implement it in the coming months. The draft's author, who works at Anthropic, says Anthropic has started implementing support. Keep the GPTBot, Google-Extended and ClaudeBot groups until each vendor's crawler docs say otherwise.

Can I opt out of AI grounding in robots.txt?

Not for Microsoft. The Bing blog says "There is no site-wide robots.txt grounding preference." Grounding is set per page, with a Bingbot meta tag or an X-Robots-Tag header that carries ai-grounding:disallow or ai-grounding:disallowextended. The __usage_ai-training token covers training only. For Google, Gemini Apps grounding eligibility follows the Google-Extended rule in your robots.txt.