01 · Symptoms
It rarely announces itself as a bot problem.
On the invoice
An overage that will not go away
You exceed your plan’s visit allowance, pay the overage, and it happens again the next month. The obvious response is to upgrade the plan, which is also the expensive response, and it does not address the cause.
On the server
Slower than the traffic justifies
Analytics shows normal visitor numbers because analytics only counts humans. The server is doing several times that much work.
In the logs
404 noise
Some crawlers hammer URLs that do not exist, generating load for nothing at all. One we found on a client site was repeatedly crawling non-existent pages and had been doing it for months.
On one recruitment client’s site, AI crawlers alone accounted for roughly 7,000 hits in a single week, which works out to about 30,000 visits a month charged against a plan billed by visit. This work usually arrives as part of a wider WordPress security engagement.
02 · The wrong fix
Two common responses, both mistakes.
Upgrading the hosting plan
This pays the bot bill rather than reducing it, and the bots do not stop growing. You are buying capacity for traffic that will never convert.
Turning bot protection up to maximum
This is worse, and it is the more tempting option because it is one setting. Blanket bot protection does not distinguish between crawlers. It catches Googlebot and Bingbot alongside the ones you wanted gone, and it catches the AI assistants that are increasingly sending real referral traffic to your site.
We have seen a site where search crawlers were being blocked by the host own bot handling, which is a far more expensive problem than a bandwidth overage and much harder to detect, because nothing breaks. Rankings just quietly decay.
03 · What we do instead
Measure, sort, approve, verify, review.
Measure before touching anything
We pull the actual traffic composition from Cloudflare analytics: what proportion is human, what proportion is bot, which crawlers specifically, how much each one consumes, and what they are requesting. Until you have that, every decision is a guess.
This usually produces at least one surprise. The bot eating the most resources is frequently not one anybody had heard of.
Sort the crawlers into three groups
Allow, always: search engine crawlers. Googlebot and Bingbot stay on the allowlist and we verify the allowlist rather than assume it. Blocking these costs you more than any hosting overage.
Allow, deliberately: the AI assistants that send referral traffic back. If people are finding you through an AI search tool, blocking its crawler removes you from that surface entirely. This is a commercial decision, not a technical one, and it belongs to the client.
Block, by name: crawlers that consume resources and return nothing. Aggressive regional crawlers with no audience overlap with your business, SEO tool crawlers you do not use, and anything hammering URLs that do not exist.
Get written approval before blocking
We list the crawlers, explain what each one does and what blocking it costs, and get a decision from the client. Nothing gets blocked because we thought it was a good idea.
RuleBlocking a crawler is a business decision about which surfaces you appear on, and it is not ours to make quietly.
Verify the traffic actually passes through the rules
This is the step that gets missed, and it invalidates everything above when it is wrong.
On one client site we deployed the blocks, and the traffic barely moved. The reason was a DNS configuration issue: not all requests were being routed through Cloudflare, so a portion of traffic was reaching the origin server without touching a single security rule. The blocks were working perfectly on the traffic they could see, which was not all of it.
We fixed the DNS, scheduled the cutover for early morning, and had the client team verify the site and its integrations from the office the same morning.
RuleUntil you confirm this, a set of bot rules is a set of assumptions.
Review, and keep reviewing
Crawler behavior changes. New AI companies launch crawlers monthly. A configuration that was right in March is not automatically right in September, so this belongs in an ongoing review cycle rather than being a one-time fix.
Crawler control is one piece of our wider WordPress security work. Because new crawlers appear every month, most clients fold the review into a maintenance retainer rather than treating it as a one-time job.
Common questions
Cannot I just use robots.txt?+
robots.txt is a request, not a rule. Well-behaved crawlers respect it. The ones costing you money frequently do not, and the ones causing the most load are often the least compliant. Blocking at the Cloudflare edge is enforcement rather than a polite note, and it happens before the request reaches your server, so it costs you nothing to serve.
Will blocking AI crawlers hurt my visibility in AI search?+
It can, which is exactly why this is a decision rather than a default. Blocking a crawler means the company operating it stops seeing your content, and if that company’s product sends you referral traffic, you have traded a hosting cost for a traffic cost. We separate the crawlers that feed products people actually use from the ones that do not, and you decide.
How do I know which bots are hitting my site?+
You need edge-level analytics, which means Cloudflare or an equivalent in front of your site. Your WordPress analytics will not show you this, because Google Analytics is designed to filter bots out. That is the first thing we set up if it is not already there.
Is this a one-time job or ongoing?+
The initial audit and configuration is a project. Keeping it current is ongoing, because new crawlers appear constantly. Most clients run the project once and then fold the review into their maintenance retainer.
My host says they handle bot traffic. Is that not enough?+
Hosts do filter some bot traffic, and some handle it much better than others. But host-level filtering is generic, you cannot see what it is doing, and you cannot make exceptions. It is also, in at least one case we have dealt with, capable of blocking search engine crawlers without telling you. Edge-level control is visible and adjustable. Host-level filtering is a black box.
What if the bots are a symptom of something else?+
Sometimes they are. Repeated crawling of URLs that do not exist can point to a sitemap problem, a plugin generating phantom pages, or a bad migration leaving thousands of dead URLs behind. We look for that, because fixing the cause beats blocking the symptom.
