What is the difference between Allow and Disallow?

Disallow tells search engine crawlers which pages or directories they should NOT crawl or index. Allow is used to override Disallow rules and specifically permit crawling of certain pages within a disallowed directory. For example, you might disallow an entire /admin/ directory but allow /admin/public-page.html. If you don't specify any rules, search engines will crawl everything by default.

Robots.txt Generator | Control Crawler Access

Q: What is a robots.txt file?

A robots.txt file is a text file placed in the root directory of your website that tells search engine crawlers which pages or sections of your site they can or cannot access. It's part of the Robots Exclusion Protocol, a standard used by websites to communicate with web crawlers and other web robots.

Q: Where should I place my robots.txt file?

The robots.txt file must be placed in the root directory of your website. For example, if your website is https://example.com, the robots.txt file should be accessible at https://example.com/robots.txt. It cannot be placed in a subdirectory and will not work if placed elsewhere.

Q: Should I block all search engines from my website?

No, in most cases you should not block all search engines. Blocking search engines will prevent your website from appearing in search results, which means you'll lose organic traffic. Only block search engines if your site is in development, contains sensitive information, or you specifically don't want it indexed. For production websites, you typically want to allow search engines to crawl most or all of your content.

What is a robots.txt File?

A robots.txt file is a simple text file that tells search engine crawlers which pages or sections of your website they can or cannot access and index. It's placed in the root directory of your website and follows the Robots Exclusion Protocol.

This file is crucial for SEO because it helps you control how search engines interact with your website, prevent duplicate content issues, manage crawl budget, and protect sensitive areas of your site from being indexed.

Basic Syntax

User-agent: Specifies which crawler the rules apply to (e.g., Googlebot, Bingbot, or * for all)

Disallow: Tells crawlers which paths they should not access

Allow: Explicitly permits crawling of specific paths (used to override Disallow rules)

Sitemap: Points to your XML sitemap location

Crawl-delay: Sets a delay between requests (not supported by all crawlers)

Frequently Asked Questions

Robots.txt Generator

Quick Presets

Sitemap URLs

AI Bot Control

Crawler Rules

Generated robots.txt

What is a robots.txt File?

Basic Syntax

Frequently Asked Questions

What is a robots.txt file?

Where should I place my robots.txt file?

Should I block all search engines from my website?