Published:
by Wayne Smith
The robots.txt file is an important component of technical SEO and is often considered part of full-stack SEO practices. While technically optional, it is widely regarded as a best practice. In essence, the file informs web crawlers and search engines which URLs should or should not be indexed.
More comprehensively, the robots.txt file can:
- Use wildcards (e.g., *) for flexible URL targeting
- Support simple pattern matching (note: it does not support full regular expressions)
- Reference sitemaps for better crawler guidance
- Set crawl delay directives (though not all bots honor this)
- Specify rules for individual bots
If no robots.txt file is present, search engines typically assume the entire site is available for indexing. Some believe that using the Allow directive may influence prioritization for indexing, though this remains speculative, and there is no evidence that the Allow directive prioritizes pages for indexing — it merely enables access to otherwise disallowed paths.
A matching User-agent is required for the rule to be enforced. The wildcard * is used to apply rules to all bots.
This example permits all bots to crawl all pages on the site. It functions similarly to having no robots.txt file at all.
Location of robots.txt File
The robots.txt file must be placed at the root of a domain. For example, https://example.com/robots.txt is valid. However, subdomains like blog.example.com or www.example.com require their own separate robots.txt files if they are crawlable. The "www" prefix is technically a subdomain, just like "blog" or "store".
How robots.txt is Processed and Common Mistakes
It's best practice to structure your robots.txt file assuming that rules are processed from top to bottom, as many bots historically apply the first matching directive they encounter. While modern crawlers like Googlebot use rule specificity rather than order, ensuring top-down compatibility helps support a broader range of crawlers, especially older or less advanced ones.
For example, the following is a common mistake. The broad Allow rule appears first, which cause some crawlers to ignore the more specific Disallow rule that follows:
While some bots stop at the first match and ignore the disallow rule, Googlebot uses rule specificity to determine which directive to follow. In this case, it will correctly block crawling of PDF files because the disallow pattern is more specific than the general allow.
To ensure the most reliable behavior across all bots, it's safer to place the disallow rule before the allow rule.
Misconception: Using Disallow Removes a Page from Search Results
A common misunderstanding is that disallowing a page in robots.txt will immediately remove it from search engine results. In reality, the Disallow directive only blocks compliant bots from crawling the page — it does not remove content that has already been indexed, nor does it guarantee that the page won’t appear in search results if it’s linked from elsewhere.
This configuration tells crawlers not to access dontindex.pdf. However, if the file was already indexed — or if it is linked to from other websites — it may still appear in search results, sometimes with minimal or no snippet information.
Over time, search engines may drop pages that haven’t been crawled in a long time (often around 90–120 days), but this is not guaranteed. For immediate removal, use the appropriate removal tools or APIs — such as Google’s Remove Outdated Content tool or Bing’s Webmaster Tools.
Disallow-First Strategy
A "Disallow-first" strategy is commonly used in robots.txt files. Technically, the Allow directive is not required. If a URL does not match any rule in the robots.txt file, it is generally assumed to be crawlable and indexable.
Sitemap and Allow Directives Should Not Reference 404 Pages
There is anecdotal evidence, including discussions on Google’s Webmaster Help forum, suggesting that referencing a sitemap URL in robots.txt that returns a 404 error may contribute to crawling inefficiencies or indexing issues.
Similarly, Allow directives in robots.txt should not point to URLs that return 404 errors. While speculative, doing so may lead to wasted crawl attempts and could signal inconsistencies in your site’s structure to search engines.