SNACK: 3-line summary
- Cloudflare has introduced a system that continuously evaluates verified bots’ behavior.
- Respect for robots.txt and responses to HTTP 429 requests to slow down shape a bot’s trust assessment.
- The system is internal for now; public Radar scores and selectable trust thresholds for all Cloudflare customers are planned.
Cloudflare announced ongoing behavior checks for verified bots on October 16, 2026, looking beyond who operates a crawler to whether it respects a website’s limits. Cloudflare is already using the system internally; access for customers remains a future plan.
Snackgirls react
AIKO: I’d like to see how the same verified bot’s trust band changes after it repeatedly ignores 429 responses. A valid signature can establish identity; listening skills require separate evidence.
Nea: I’d like small game wikis and fan archives to have their limits respected, too. I’m curious whether a crawler’s longer history will show that kind of care consistently.
Does the crawler respect robots.txt?
The first check looks at robots.txt, where site owners publish preferences about which bots should crawl which paths. Cloudflare compares observed traffic with those preferences over time, looking for repeated disregard rather than treating a one-off violation as decisive.
Those preferences set minimum limits. A missing restriction is not blanket permission, and a site can use other signals to communicate that continued access is unwelcome.
A 429 response should change what happens next
Even a crawler that stays within permitted paths can overload a site. An HTTP 429 “Too Many Requests” response asks the visitor to reduce its request rate, sometimes with guidance on how long to wait. Cloudflare watches whether the bot slows down, pauses or continues at the same pace.
Consistently slowing down or pausing raises trust; repeatedly ignoring the signal lowers it. These observations feed an explainable trust band—a range showing how consistently the bot respects site preferences and the evidence behind that assessment. Risk concerns a particular request at a particular moment, while trust reflects sustained behavior; the band is neither a permanent verdict nor a security guarantee.
Internal checks now, public scores and customer access later
Cloudflare is currently building a history of bot behavior internally and refining how those observations become useful guidance. It plans to publish scores on Radar and make the system available to all Cloudflare customers, who would be able to choose the trust threshold used to handle verified bot traffic.
Customers can already use Bot Management to manage automated traffic and Bot Preference Sync to align published site preferences with configured policies. Radar’s existing bot-population trends and the BotBase directory are separate from the planned public trust scores.
Sources and checked date: October 17, 2026
Comments
0No login needed. Edit or delete your comment from the same browser.
All comments 0
한국어 · English · 日本語No comments yet. Start the conversation.