Last updated: June 24, 2026

How to Create a Robots.txt File for SEO

A robots.txt file gives crawl instructions to search engine bots. It is usually placed at the root of a website, such as example.com/robots.txt. A simple robots file can allow crawling and point search engines to your sitemap.

Robots.txt is useful for crawl guidance, but it is not a security tool. Do not rely on it to hide private pages. If something must stay private, protect it with proper access control.

Simple example

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This example allows crawling and lists the sitemap location. After publishing, open the robots file in your browser and check it in Search Console.

Common mistakes

• Blocking the whole site by mistake.
• Using the wrong sitemap URL.
• Assuming robots.txt removes indexed pages immediately.
• Forgetting to test after deployment.

Recommended robots.txt workflow

1. Create a simple robots.txt file that allows important public pages to be crawled.
2. Add the correct sitemap URL so search engines can discover clean canonical pages.
3. Open the file at yourdomain.com/robots.txt after deployment.
4. Use Search Console URL inspection and sitemap reports to check whether Google can discover important URLs.

Robots.txt and sitemap best practice

For most small websites, the safest robots.txt file is simple. Avoid blocking CSS, JavaScript, images or tool pages that should appear in search. If a page should not be indexed, use the right page-level method instead of relying only on robots.txt.