What is a robots.txt file?
A robots.txt file is a simple text file located in the root directory of a website. It provides instructions telling compliant search-engine crawlers which URLs they may or may not request.
For example, the file for www.example.com must be available at www.example.com/robots.txt. The rules only apply to the protocol, host and port where that file is published.
Do not confuse these
Crawling, indexing and security are different
This is where most robots.txt mistakes begin.
Crawling
A crawler requests and reads a URL. Robots.txt can allow or disallow that request.
Indexing
A search engine stores a URL and may display it in search results. Robots.txt does not reliably prevent this.
Security
Sensitive content needs authentication or password protection. Robots.txt is public and provides no security.
Important: robots.txt does not guarantee removal from Google
If other websites link to a blocked URL, Google may still discover and display that URL without crawling its content. Use an appropriate noindex instruction, password protection or removal method instead.
Why do you need robots.txt?
For most smaller websites, the file should remain simple. Its main purpose is to guide crawlers away from URLs that provide no value in search.
01
Reduce unnecessary crawling
Prevent repeated requests to administrative areas, internal search results and low-value URL combinations.
02
Manage large websites
Large or rapidly changing websites may use robots.txt to reduce requests to repetitive or unimportant URL patterns.
03
Declare your sitemap
A Sitemap directive can give crawlers the full address of the website’s XML sitemap.
How robots.txt rules work
Each group starts with a user agent. Allow and Disallow rules then control access to specified URL paths. The rules are case-sensitive.
Identifies the crawler that should follow the group of rules.
Prevents the named crawler from requesting a matching path.
Creates an allowed exception inside a broader blocked path.
Provides the complete address of an XML sitemap.
A practical WordPress example
A straightforward WordPress robots.txt file may look like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap_index.xmlThis blocks the main WordPress administration directory while allowing the AJAX endpoint used by some front-end functions. Replace the example sitemap with the genuine sitemap address.
Best practices for using robots.txt
Use `/robots.txt` on the correct protocol and host.
Only block URL paths for a clear and deliberate reason.
Do not block CSS or JavaScript needed to understand a page.
Robots.txt paths and rules are case-sensitive.
Open the live file and check the robots.txt report in Search Console.
Check that staging rules have not been copied onto the live website.
What robots.txt should not be used for
- Hiding confidential information: use password protection, authentication or server permissions.
- Guaranteeing removal from Google: use noindex where the page can be crawled, remove it or use an appropriate removal method.
- Fixing duplicate content: use canonicals, consistent internal links and proper URL handling.
- Protecting a staging website: restrict public access and password-protect the entire environment.
Technical SEO without the confusion
Talk Sense can check whether Google is crawling and indexing the right areas of your website.
Official guidance and further reading
Technical guidance checked October 2026. Search-engine documentation and supported directives can change, so confirm current guidance before changing a live robots.txt file.