What is Robots.txt?

The robots.txt file is one of the first things Googlebot looks for when it visits your website. It follows the Robots Exclusion Standard, a set of instructions that control crawler access.

A robots.txt file can allow all crawlers access to the entire site (the most common configuration for public-facing content sites), disallow crawlers from specific paths (like admin pages, duplicate content directories, or search result pages that should not be indexed), or specify where to find the site's sitemap.

For most small business websites, robots.txt is simple: allow all crawlers, disallow any admin or login pages that should not appear in search results, and point to the sitemap location. Overcomplicated robots.txt files, or accidentally disallowing important pages, can catastrophically harm SEO.

Common robots.txt mistakes include: accidentally blocking entire sections of a site (this happens frequently when staging sites are launched without updating the robots.txt that blocked all crawlers during development), inconsistencies between what robots.txt allows and what canonical tags specify, and adding a robots.txt disallow rule for a page while still wanting that page to rank (which does not work, Google respects the crawl block but the page can still appear in results based on links).

You can view any website's robots.txt file by navigating to theirdomain.com/robots.txt in a browser. Google Search Console shows any crawl errors related to robots.txt under the "Crawl" section.

Robots.txt is not a security tool. It instructs well-behaved crawlers but does not prevent access. Sensitive content should be protected by authentication, not just a robots.txt disallow rule.

Relevant services from FabBuilds

Related terms