robots.txt: 5 Mistakes We Found on Our Own Website
We check robots.txt, sitemaps and indexing regularly, on client projects and on our own website alike. At the last check, our robots.txt was partly outdated: 88 lines, extended over the years, and some entries no longer matched what we want today. So on 14 November 2026, we rewrote it. Going through it, we found 5 mistakes, one of which locked out the crawler of a real search engine. The old version from 9 May 2026 is stored in the Internet Archive, and every quote below comes from it.
We are sharing these mistakes because they are typical. A robots.txt goes unnoticed as long as the website works. It gets extended, copied and rarely read again. Every mistake here could just as well have happened on your website.
What robots.txt can and cannot do
The robots.txt tells crawlers which paths they may fetch. It controls crawling, not indexing. That distinction is behind the first mistake. Since 2022, the format has been standardised as RFC 9309, and Google describes its interpretation in the Search Central documentation. We checked every finding against both sources.
Mistake 1: Disallow hides the noindex
The old file blocked 6 pages for all crawlers:
| Line | Entry |
|---|---|
| 3 | User-agent: * |
| 4 to 9 | Disallow: /agb, /impressum, /datenschutz, /termin-buchen, /danke-termin, /rich-text-components |
The intention was clear: legal pages and booking pages should not show up on Google. The block can do the opposite, though. Google says that if pages may not be crawled, Google cannot index their content, but the URL may still appear without a snippet. A crawler only sees a noindex in the page head if it is allowed to fetch the page. So the block prevents exactly the signal that would keep the page out of the index. In our Search Console, the imprint, privacy policy and terms never showed up in 16 months, so no damage can be proven. The block was still wrong.
Now: all pages are open to crawling. The 5 pages that still exist carry a noindex in the page head. For files without an HTML head, such as PDFs, the X-Robots-Tag in the HTTP header does the job.
Mistake 2: A blank line inside a group
This mistake is the least visible. Lines 11 to 15 were meant to lock out the Internet Archive and the MJ12bot crawler:
| Line | Entry |
|---|---|
| 11 | User-agent: ia_archiver |
| 12 | User-agent: archive.org_bot |
| 13 | (blank) |
| 14 | User-agent: MJ12bot |
| 15 | Disallow: / |
Under RFC 9309, the file does exactly that. A group consists of one or more user-agent lines and the rules after them, and it only ends with the next user-agent line after a rule or with the end of the file. A blank line makes no difference. Lines 11, 12 and 14 therefore form one group, and “Disallow: /” applies to all 3 crawlers.
The original standard from 1994 sees it differently: there, blank lines separate the records, and every record needs at least one Disallow line. For a crawler that follows it, lines 11 and 12 are a record of their own without a rule, and therefore invalid. Whether the block applied depended on which standard a crawler reads the file by. The Internet Archive kept saving our homepage anyway, most recently in October 2026. A rule that allows 2 readings is a mistake, even if Google understands it correctly.
Now: there are no blank lines inside a group anymore. Every group has a comment stating its purpose and ends with a rule.
Mistake 3: A real search engine was blocked
Between analysis tools and foreign search engines sat this group:
| Line | Entry |
|---|---|
| 45 | User-agent: DuckDuckBot |
| 46 | Disallow: / |
DuckDuckBot is the crawler of DuckDuckGo, a search engine used by many privacy-conscious people. For an agency that lives on inquiries, there is no reason to lock it out. We cannot measure how much this mattered. The line probably came from one of the many blocklists circulating online. That is exactly the risk of copied lists: hardly anyone checks every entry.
Now: DuckDuckBot is no longer in the file and may fetch everything through the group for all crawlers.
Mistake 4: Bot names that do not exist
A crawler looks for its own name, the so-called product token, in the robots.txt. According to RFC 9309, this name may only contain letters, underscores and hyphens. 2 entries in the old file could therefore never match:
- “SEMrush Sensor Bot” contains spaces and is not a valid name. On its bot page, Semrush lists names such as SemrushBot and SiteAuditBot instead.
- “SEO-Kicks” misses the crawler, which identifies itself as “SEOkicks” according to its operator.
A third entry looked suspicious but was harmless: “DotBot” with 7 spaces at the end of the line. Google ignores spaces at the start and end of a line. Also harmless, but a sign of a file that grew over time: Baiduspider and DotBot each appeared twice.
Now: every name has been checked against the operator's documentation.
Mistake 5: The sitemap line appeared twice
This mistake only crept in at the beginning of November, and it is a Webflow quirk. Besides the Webflow sitemap, we have a second sitemap for our own tools, such as the UTM generator. We added both at the top of the robots.txt. Webflow, however, automatically appends its own line for sitemap.xml. After that, it appeared twice in the file.

It does no harm, crawlers only evaluate duplicate lines once. It is still not clean. Webflow offers the switch “Remove sitemap.xml from robots.txt” for this. If it is on, the file only contains the sitemap lines you enter yourself. If it is off, you only add the additional sitemaps yourself. We switched it on so that both sitemaps sit together at the top of the file.
Now: every sitemap appears exactly once in the file.
The trap during the rebuild: a group of its own overrides *
When rewriting the file, we gave AI crawlers such as GPTBot, ClaudeBot and PerplexityBot their own groups. One rule applies here that many people overlook: each crawler follows only one group, the one with the best-matching name. It ignores all other groups, including the one for all crawlers with “*”.
Had we kept blocking the legal pages with Disallow under “*”, they would have been open to every crawler with its own group. Anyone who gives a bot its own group therefore has to repeat all the rules from “*” there. We solved the problem at the root: under “*” there is now only “Allow: /”, and whatever should stay out of the index is handled by the noindex on the page. That applies to every crawler, whatever group it is in.
How our robots.txt is structured today
The new file has 55 lines in 6 groups, each with a comment:
| Group | Rule | Why |
|---|---|---|
| All crawlers (*) | Allow: / | We control indexing with noindex on the page. |
| AI search and answer engines, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot | Allow: / | We want to be mentioned in AI answers. |
| Training and grounding, such as GPTBot, ClaudeBot, Google-Extended | Allow: / | For visibility in AI answers (GEO, LLMO, AEO), our content should feed into language models. |
| Analysis and competitor tools, such as AhrefsBot, SemrushBot | Disallow: / | They create load and give competitors data about us. |
| Other search engines, such as Baiduspider, SeznamBot, Exabot | Disallow: / | They use bandwidth and bring no visitors from our markets. |
| Uptime checkers, such as UptimeRobot, Pingdom | Allow: / | Monitoring services should be able to check availability. |
The decision on AI crawlers was deliberate. Anyone who wants to appear in ChatGPT, Gemini and Perplexity has to let their crawlers in. How this works together with llms.txt, and what else counts for visibility in AI answers, is covered on our page on GEO and LLMO optimization.
Checklist for your robots.txt
- Do not use Disallow for pages that should only stay out of the index. That is what noindex is for.
- No blank lines inside a group. Not every crawler reads them the same way.
- Know every crawler you block. Is there a search engine in there that brings visitors?
- Look up names with the operator. No spaces, exact spelling.
- Make your own groups complete. What is under “*” does not apply there.
- List every sitemap exactly once. On Webflow, watch out for the automatic line.
You can check all of this in Google Search Console. The robots.txt report shows which version Google fetched last. The status “Indexed, though blocked by robots.txt” in the page indexing report reveals exactly the first mistake.
Our old robots.txt looked harmless and still locked out the crawler of a search engine. That is exactly why it is part of every check. If you want to know where your website stands, let's look at it together. As a Webflow agency, we check robots.txt, sitemaps and indexing on every project, get in touch.
FAQ
Is Disallow in robots.txt enough to keep a page out of Google?
No. Disallow only prevents crawling. Google may still show the URL without a snippet. To keep a page out of the index, it needs a noindex in the page head or in the X-Robots-Tag, and it must not be blocked in robots.txt for that.
Does a blank line end a group in robots.txt?
Not under RFC 9309. There, a group only ends with the next user-agent line after a rule or with the end of the file. The original standard from 1994, however, separates records with blank lines. So that every crawler reads the file the same way, a group should contain no blank lines.
Should you block AI crawlers in robots.txt?
It depends on your goal. Anyone who wants to be mentioned in ChatGPT, Gemini or Perplexity has to let their crawlers in. We deliberately allow AI search and training. Important: a crawler with its own group ignores the rules under “*”, so they have to be repeated there.




















































rev%20komprimiert.jpg)





















