Robots.txt is a plain text file placed at the root of a website that instructs web crawlers and automated bots which pages or sections of the site they are permitted or not permitted to access. It is one of the most fundamental tools in a webmaster's toolkit for managing how search engines interact with a website's content.
The file follows a standard known as the Robots Exclusion Protocol, which was established in 1994 and has since been adopted universally by major search engines including Google, Bing, and others. When a crawler such as Googlebot visits a website, one of its first actions is to request the robots.txt file, typically found at a URL like https://example.com/robots.txt. The crawler then reads the file's directives before deciding which parts of the site to crawl.
How Robots.txt Works
The file is structured around two key components: the User-agent directive, which identifies the specific crawler being addressed, and the Disallow directive, which specifies the paths that crawler should not visit. A complementary Allow directive can be used to explicitly permit access to a path that would otherwise fall under a broader restriction. A single robots.txt file can contain multiple groups of rules, each targeting a different crawler or set of crawlers.
For example, a site might block all crawlers from accessing an internal search results page or a staging directory, since indexing such pages provides no value to users and wastes crawl budget. Crawl budget refers to the number of pages a search engine is willing to crawl on a given site within a certain timeframe, making robots.txt a useful lever for directing crawler attention toward the most important content.
Robots.txt and SEO
From an SEO perspective, robots.txt plays an important role in controlling which content appears in search engine indexes. However, it is worth understanding a critical distinction: robots.txt controls crawling, not indexing. A page blocked by robots.txt may still appear in search results if other pages link to it, because search engines can infer its existence without actually visiting it. To prevent a page from being indexed, a noindex meta tag or HTTP header is the appropriate mechanism.
It is also important to note that robots.txt operates on an honor system. Reputable crawlers respect its directives, but malicious bots may ignore the file entirely. Therefore, robots.txt should not be used as a security measure to protect sensitive content - proper authentication and server-level access controls serve that purpose.
Despite its simplicity, robots.txt remains a powerful and widely used configuration file. Errors in its syntax, such as accidentally disallowing the entire site, can have significant consequences for a site's search visibility, making careful management of the file an important responsibility for developers and SEO professionals alike.