By Block AI Crawlers on Your Own Server
Measure how much of your traffic is AI crawlers from your own access log, then block or rate limit them in nginx, Apache or Caddy without losing real visitors.
What blocking AI crawlers actually saves you
You block AI crawlers to cut a bill. Every crawler request costs the same CPU time, the same database query and the same outbound bytes as a real visitor, and a crawler that walks a 4,000 page site does it in an afternoon. On a VPS with a monthly transfer allowance, that is money.
Before you copy a single line of config, measure. On most small sites the honest answer is that AI crawlers are a small slice of traffic, and any rule you add is pure risk for no saving. On some sites they are most of the traffic. You cannot tell which you are without looking, and your access log already knows.
RunCloud published a clear and copy-pasteable guide to the User-Agent blocking technique, and the nginx map block below is the same shape. This guide starts one step earlier, at measurement, and carries on past the point where User-Agent matching stops working.
Measure your bot traffic before you change anything
nginx writes the User-Agent as the last quoted field of its default combined log format. Splitting each line on the quote character puts the User-Agent in the sixth piece, so awk can count them.
sudo sh -c 'zcat -f /var/log/nginx/access.log*' \
| awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -25zcat -f passes uncompressed files through unchanged, so one glob covers today's log and the rotated .gz files together. That matters, because Ubuntu rotates nginx logs daily and keeps 14 days, so a bare access.log is often only a few hours old and tells you nothing about a crawl that ran on Tuesday. Apache's combined format puts the User-Agent in the same place, so the same command works on /var/log/apache2/access.log.
Now the number that decides everything: the share of requests that come from known AI crawlers.
sudo awk -F'"' '
{ total++ }
tolower($6) ~ /gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user|ccbot|bytespider|amazonbot|meta-externalagent|imagesiftbot|omgili|diffbot|timpibot/ { bots++ }
END { printf "%d of %d requests (%.1f%%)\n", bots, total, 100*bots/total }
' /var/log/nginx/access.logThe patterns are all lowercase because tolower() has already lowercased the field. Next, the bandwidth, which is the number your provider actually bills you on.
sudo awk -F'"' '{ split($3, f, " "); bytes[$6] += f[2] }
END { for (ua in bytes) printf "%12d %s\n", bytes[ua], ua }' \
/var/log/nginx/access.log | sort -rn | head -20$3 is the text sitting between the request line and the referer, which holds the status code and the response size. split with a single space uses awk's default whitespace handling, so f[1] is the status and f[2] is the bytes sent. Divide the total by 1048576 for mebibytes.
Write both numbers down. They are your before. Without them you will never know whether the rules you are about to add did anything at all.
Layer 1: robots.txt is a request, not a rule
robots.txt is a plain text file at the root of your site that names crawlers and tells them what not to fetch. Nothing enforces it. A crawler reads it because its operator decided to, which means it works on the crawlers that were already going to behave. Write it anyway, because it is the only layer the major operators have publicly committed to honouring, and the only one that can express a preference a firewall cannot state: index me, but do not train on me.
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow:Two of those names are robots.txt tokens and nothing else. Google's documentation is explicit that Google-Extended "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." Applebot-Extended works the same way. So a Google-Extended pattern in your nginx config matches nothing, ever, because no request ever carries that string. Those two tokens only do work in robots.txt.
Blocking Google-Extended does not cost you search traffic. Google states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." It governs Gemini training, not indexing. Blocking Applebot, which is the real crawler and a different name, does remove you from Siri and Spotlight results. Do not confuse the two.
Check the file is being served before you trust it, since a static site generator will happily overwrite it on the next build.
curl -s https://example.com/robots.txt | head -20Layer 2: how do I block AI crawlers by User-Agent in nginx?
A map block turns the User-Agent request header into a variable, and one if turns that variable into a 403. If you have not met request headers before, the anatomy of an HTTP request covers where this string comes from. map lives in the http context, so put it in its own file at /etc/nginx/conf.d/ai-bots.conf.
map $http_user_agent $ai_bot {
default 0;
"~*gptbot" 1;
"~*ccbot" 1;
"~*claudebot" 1;
"~*bytespider" 1;
"~*amazonbot" 1;
"~*meta-externalagent" 1;
"~*imagesiftbot" 1;
"~*omgili" 1;
"~*diffbot" 1;
"~*timpibot" 1;
}Then one block inside the server you want protected:
if ($ai_bot) {
return 403;
}~* marks the pattern as a case-insensitive regular expression, so GPTBot and gptbot both match. A map is evaluated once per request, and only when the variable is first read, which is cheaper than a chain of if statements. The if above is safe: nginx behaves unpredictably when if wraps most directives, but return is one of the cases nginx documents as always reliable.
Return 403 while you are testing. A 403 appears in your access log and in curl -I, so you can prove the rule fired. return 444 closes the connection with no response at all, which saves a few more bytes and leaves you nothing to debug with. Move to 444 only after the rule has been correct for a week.
Be clear about what this saves. A 403 still costs you the TCP handshake, the TLS handshake and a log line. What it saves is the response body, and that is where the bill lives: a 403 is roughly 150 bytes, and the page it replaced might be 2 MB with images. If your problem is bandwidth, this is the layer that fixes it.
sudo nginx -t
sudo systemctl reload nginxnginx -t parses the config and prints the file and line of any error. If it fails, do not reload. A reload with a broken config is a no-op: nginx keeps serving the old config and stays up, so you will believe the change is live when it is not.
Layer 2 in Apache
Apache does the same job with an <If> expression, which needs Apache 2.4.26 or newer. Ubuntu 24.04 ships 2.4.58, so this works with no extra modules. Put it in the <VirtualHost> block.
<If "%{HTTP_USER_AGENT} =~ /(GPTBot|CCBot|ClaudeBot|Bytespider|Amazonbot|meta-externalagent|ImagesiftBot|omgili|Diffbot|Timpibot)/i">
Require all denied
</If>The i after the closing slash makes the match case-insensitive, and Require all denied returns 403. The older mod_setenvif form reads better once the list grows past a few names, and it works on Apache versions that predate <If>.
BrowserMatchNoCase "GPTBot" ai_bot
BrowserMatchNoCase "CCBot" ai_bot
BrowserMatchNoCase "ClaudeBot" ai_bot
BrowserMatchNoCase "Bytespider" ai_bot
<RequireAll>
Require all granted
Require not env ai_bot
</RequireAll>sudo apachectl configtest
sudo systemctl reload apache2configtest prints Syntax OK when the file parses. Anything else means do not reload yet.
Layer 2 in Caddy
Caddy matches headers natively, so there is nothing to install. A named matcher plus one directive does it. If you are still choosing a server, nginx, Caddy and Traefik compared covers the wider trade-off.
example.com {
@ai_bots header_regexp User-Agent (?i)(GPTBot|CCBot|ClaudeBot|Bytespider|Amazonbot|meta-externalagent|ImagesiftBot|omgili|Diffbot|Timpibot)
respond @ai_bots 403
root * /srv/example.com
file_server
}Caddy's regular expressions use RE2, so the case-insensitive flag goes inside the pattern as (?i) rather than after it. respond sorts before file_server in Caddy's fixed directive order, so the block fires before any file is served and you do not need a route wrapper. Swap respond @ai_bots 403 for abort @ai_bots to drop the connection silently, which is Caddy's equivalent of nginx return 444. Named matchers must sit inside the site block that uses them.
caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddyLayer 3: rate limiting is the only layer that holds a crawler that lies
Everything above trusts a string the client chose. The User-Agent header is set by whoever wrote the client, and changing it is one flag on a curl command. A crawler that respects robots.txt will also name itself honestly. A crawler that ignores robots.txt has no reason to keep its name in the header. So the layers above stop precisely the crawlers that were going to stop anyway.
Rate limiting is different because it counts requests per client address, and the address is the one thing a client cannot simply declare. A scraper pretending to be Chrome still arrives from an IP, and a hundred requests a second from one IP is a crawler whatever the header says.
map $request_uri $limit_key {
default $binary_remote_addr;
"~^/webhooks/" "";
"~^/health" "";
}
limit_req_zone $limit_key zone=perip:10m rate=30r/m;
limit_conn_zone $limit_key zone=conn_perip:10m;
limit_req_status 429;
limit_req_log_level warn;The map is the exemption mechanism, and it is the part people get wrong. nginx has no limit_req off; directive, so you cannot switch limiting off in a location. What the docs do guarantee is that "requests with an empty key value are not accounted", for both limit_req_zone and limit_conn_zone. Mapping a path to an empty string is therefore the supported way to exempt it. Put your payment webhooks and your health check endpoint there before you turn anything on.
Apply the limits in the server block:
limit_req zone=perip burst=20 nodelay;
limit_conn conn_perip 10;rate=30r/m allows thirty requests a minute per address, which is one every two seconds. burst=20 lets a client run twenty requests ahead of that rate, which covers a real browser fetching a page plus its CSS, JavaScript and images all at once. nodelay serves those burst requests immediately instead of spacing them out. Leave nodelay off and an ordinary page load feels broken, because nginx queues the images and releases them one every two seconds.
Set limit_req_status 429. The default is 503, which means server error and invites a well-behaved crawler to retry soon. 429 means too many requests, and it is the code crawlers are built to back off from.
A 10m zone holds roughly 80,000 addresses, since nginx documents one megabyte as about 8,000 states on a 64-bit platform. When the zone fills, nginx evicts the oldest entries and logs a warning.
Two weaknesses you should know before you rely on this. A scraper spread across a hundred addresses gets a hundred separate budgets, and per-IP limiting does nothing to it. And every visitor behind one office NAT (network address translation) gateway, one mobile carrier or one university looks like a single client, so a low rate can throttle a whole company. If nginx sits behind a CDN, $binary_remote_addr is the CDN edge, not the visitor, and you will rate limit your entire CDN into a 429. Set set_real_ip_from and real_ip_header first, then confirm your access log shows visitor addresses, before you enable any limit.
Rate limiting in Apache and Caddy
Apache has no equivalent of limit_req in its core, and mod_ratelimit is not it. mod_ratelimit throttles bandwidth per connection in KiB/s, which is a different control. For request rate you install mod_evasive.
sudo apt install -y libapache2-mod-evasive
sudo install -d -o www-data -g www-data /var/log/mod_evasive
sudo a2enmod evasive
sudo systemctl reload apache2<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 5
DOSPageInterval 1
DOSSiteCount 50
DOSSiteInterval 1
DOSBlockingPeriod 60
DOSLogDir "/var/log/mod_evasive"
</IfModule>The config file is /etc/apache2/mods-enabled/evasive.conf. DOSPageCount 5 with DOSPageInterval 1 means more than five requests for the same URI within one second blocks that client for DOSBlockingPeriod seconds. One caveat that is rarely stated: mod_evasive keeps its counts in a per-process hash table, so each Apache child process counts separately and the effective limit is higher than the number you wrote. Treat these values as a ceiling on obvious floods, not as a precise budget.
Caddy has no rate limiting in the standard build either. The maintained option is caddy-ratelimit, and its README says plainly that it "is not an official repository of the Caddy Web Server organization", so you compile a custom binary.
sudo apt install -y golang-go
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
~/go/bin/xcaddy build --with github.com/mholt/caddy-ratelimitexample.com {
rate_limit {
zone perip {
key {remote_host}
events 30
window 1m
}
}
root * /srv/example.com
file_server
}The module registers itself before basic_auth in the directive order, so no order line is needed. Understand the cost you are taking on: replacing your packaged caddy binary with a compiled one means apt upgrade will never update it again, and every security release is now your job. If you want request rate limiting without owning a custom build, put nginx in front or do it at the edge.
Layer 4: blocking at the edge, if you already use a CDN
Every layer above runs on your server, so the request has already crossed your network interface and spent part of your transfer allowance by the time you reject it. A 403 is cheap, not free. A CDN moves the rejection onto someone else's machine.
Cloudflare has a managed control for this on every plan including the free one. As of August 2026 it sits under Security Settings, filtered by Bot traffic, and it splits AI bots by behaviour into Search, Agent and Training, so you can refuse training crawlers while still letting answer engines through. The same screen can generate a managed robots.txt, which is Layer 1 done for you. Cloudflare's own docs are honest about the ceiling: robots.txt compliance is voluntary and the file expresses preferences without technically preventing access. Cloudflare has also announced new defaults for new domains from 15 September 2026, so read what your zone is actually set to rather than assuming.
You can also drop a crawler's published address ranges at the firewall, which is cheaper still because the packet never reaches your web server. Perplexity publishes its ranges as JSON, and OpenAI and Google publish theirs. This is a standing maintenance commitment, since ranges get reassigned and a stale deny rule eventually blocks somebody else. If you go this way, the ufw firewall basics for a VPS cover the syntax and the order rules matter in.
Blocking a search agent costs you referral traffic
Not every crawler is the same trade. Cloudflare Radar publishes a crawl-to-refer ratio: how many pages a company's crawlers fetch for each visitor that company sends back to you. These are network-wide published figures, not your site's numbers, and the spread between them is the point.
The data behind this chart
[
{
"label": "Anthropic",
"crawls_per_referral_jul_2025": "38,065.7",
"crawls_per_referral_jan_2025": "286,930.1"
},
{
"label": "OpenAI",
"crawls_per_referral_jul_2025": "1,091.4",
"crawls_per_referral_jan_2025": "1,217.4"
},
{
"label": "Perplexity",
"crawls_per_referral_jul_2025": 194.8,
"crawls_per_referral_jan_2025": 54.6
},
{
"label": "Microsoft",
"crawls_per_referral_jul_2025": 40.7,
"crawls_per_referral_jan_2025": 38.5
},
{
"label": "Google",
"crawls_per_referral_jul_2025": 5.4,
"crawls_per_referral_jan_2025": 3.8
}
]In July 2025, Anthropic's crawlers fetched 38,065.7 pages for every referral, and Google's fetched 5.4. Perplexity sat between them at 194.8, up from 54.6 six months earlier. These ratios move fast, so read the shape rather than the digits: Anthropic's own figure was 286,930.1 in January 2025.
What this means for your rules is that a company's crawlers are not one thing. Block GPTBot and you lose bytes only, because that is OpenAI's training crawler. Block OAI-SearchBot and your pages stop appearing as citations in ChatGPT search. Block Claude-User or Perplexity-User and a person who just asked a question about your product gets an answer built without your page in it. Perplexity's docs state outright that Perplexity-User "generally ignores robots.txt rules", because a human requested that specific fetch.
The second class of false positive is worse, because it is silent. A pattern that is too loose matches things you never considered.
~*botmatchesGooglebot, and you have just removed yourself from Google.~*claudematchesClaudeBotand alsoClaude-User, which is the agent a reader triggered.~*pythonor~*go-http-clientmatches your uptime monitor, your deploy script, and a payment provider's webhook retry, because webhooks are sent by ordinary HTTP libraries with default User-Agent strings.- A rate limit applied at
serverlevel counts your webhook endpoint too, so a provider retrying five times in a second gets a 429 and you find out at the end of the month.
Two rules keep you out of this. Anchor patterns on the full bot name and never on a fragment. And exempt the paths that must never be blocked, using the empty-key map above, before you enable anything.
Test a rule before you enable it, and know how to undo it
Run the rule in log-only mode first. The map sets a variable whether or not anything acts on it, so a custom log format can record who would have been blocked while everyone is still served normally.
log_format ai_audit '$time_iso8601 ai=$ai_bot ip=$remote_addr '
'ua="$http_user_agent" uri="$request_uri"';
access_log /var/log/nginx/ai-audit.log ai_audit;Leave that running for a day, then read the result.
sudo grep -c ' ai=1 ' /var/log/nginx/ai-audit.log
sudo grep ' ai=1 ' /var/log/nginx/ai-audit.log | tail -20Read all twenty lines. Every one is a request your rule will reject. If any of them is a search agent you want, a monitoring check, or a browser with an unusual User-Agent, fix the pattern now. Then enable the if and prove it with three requests.
curl -s -o /dev/null -w '%{http_code}\n' \
-A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' \
https://example.com/
curl -s -o /dev/null -w '%{http_code}\n' \
-A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36' \
https://example.com/
curl -s -o /dev/null -w '%{http_code}\n' \
-A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
https://example.com/The first must print 403. The second and third must print 200. The third is the one people skip, and it is the one that catches a pattern like ~*bot quietly deindexing the site. Run all three after every edit to the list.
To check a rate limit, send more requests than the burst allows and watch the codes change.
for i in $(seq 1 60); do
curl -s -o /dev/null -w '%{http_code} ' https://example.com/
done; echoYou should see a run of 200 and then 429 once the burst is used up. If the very first request is 429, your rate is too low or the zone key is wrong.
Rolling back is why the map lives in its own file. The right rollback is to empty the pattern list, not to delete the file: with the file gone, $ai_bot is undefined and nginx -t fails with unknown "ai_bot" variable, taking your reload with it.
map $http_user_agent $ai_bot {
default 0;
}sudo nginx -t && sudo systemctl reload nginxEvery request now maps to 0, the if never fires, and the site is exactly as it was. Test before you reload, every time, because a failed test leaves the old config running and you will spend an hour wondering why nothing changed.
Keep the bot list somewhere you will actually update it
Every name in this guide will be wrong eventually. Crawlers appear, get renamed, and split into separate training and search identities. Meta-ExternalAgent did not exist in early 2024. Anthropic added Claude-SearchBot after ClaudeBot was already widely blocked, and anyone matching on ~*claude swept up both.
Keep the names in exactly one place. In nginx that is the map block. In Apache it is one <If> or one group of BrowserMatchNoCase lines. Never scatter bot names across three vhosts, because you will update two of them and forget the third.
Then read the operators' own pages rather than any blog post, including this one:
- OpenAI documents
GPTBot,OAI-SearchBot,ChatGPT-UserandOAI-AdsBotat developers.openai.com, with the full User-Agent string for each. - Anthropic documents
ClaudeBot,Claude-UserandClaude-SearchBotin its crawler support article. - Perplexity documents
PerplexityBotandPerplexity-User, plus published IP ranges, at docs.perplexity.ai. - Google lists every crawler and control token at developers.google.com.
- The community list at ai.robots.txt generates
robots.txt,.htaccess,nginx-block-ai-bots.confand aCaddyfilefrom onerobots.json, regenerated automatically whenever a crawler is added.
Set a reminder to compare those pages against your config twice a year. That sounds like overhead. It is less overhead than discovering in November that you blocked the crawler that was sending you readers.
One last framing. Crawler traffic is a cost and capacity problem, not an intrusion. Nothing here hardens your server against anyone trying to get in, which is a separate question about how safe VPS hosting really is. Measure first, block the training crawlers, rate limit everything else, and check the referral side of the ledger before you close a door you wanted open.
FAQ
How much of my traffic is AI crawlers?
Your access log has the answer and nobody else's number is relevant. Run awk -F'"' '{print $6}' over /var/log/nginx/access.log, sort and count, and you get every User-Agent by request volume. Sum $body_bytes_sent per User-Agent for the bandwidth figure. Include the rotated .gz logs with zcat -f, because Ubuntu rotates nginx logs daily and a bare access.log may only hold a few hours. If AI crawlers are 2% of requests, adding blocking rules is risk without reward.
Does robots.txt actually stop AI crawlers?
No. It records a preference that the operator chooses to honour, and it has no enforcement behind it. Cloudflare's documentation says the same thing about its own managed version: compliance is voluntary and the file expresses preferences without technically preventing access. The major operators do currently honour it for their training crawlers. Crawlers that fetch on behalf of a person often do not, and Perplexity states directly that Perplexity-User "generally ignores robots.txt rules". Write robots.txt, then enforce with server rules if the numbers justify it.
Will blocking AI crawlers hurt my Google ranking?
Blocking Google-Extended will not. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search", because that token controls Gemini training rather than indexing. The danger is a loose pattern. A rule matching ~*bot also matches Googlebot and removes you from search entirely. Always run curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' against your site after every edit and confirm it returns 200.
Can AI crawlers get around a User-Agent block?
Yes, trivially. The User-Agent is a header the client chooses, so changing it is one flag on a curl command. User-Agent matching only stops crawlers that identify themselves honestly, which are the same crawlers that already read your robots.txt. To constrain a client that lies you need limit_req and limit_conn in nginx, keyed on the client address, since the address is not something the client can simply declare. Per-address limits still fail against a scraper spread over many IPs, so treat them as a cap on any single client rather than a wall.
Why is my rate limit returning 429 to real visitors?
Usually one of three causes. Your server sits behind a CDN or load balancer, so $binary_remote_addr is the proxy's address and every visitor shares one budget: set set_real_ip_from and real_ip_header, then confirm your access log shows visitor addresses. Or you omitted nodelay, so nginx queues a normal page's images and releases them at the configured rate, which looks like a hang. Or the rate is simply too low for a page that loads twenty assets, and burst=20 nodelay fixes it. Check with a for loop of curl calls and watch where the codes flip from 200 to 429.