Rule Group 1

Sitemaps

Generated robots.txt

0 lines
# Empty robots.txt

Templates

Quick Reference

User-agent

Specifies which crawler the rules apply to. Use * for all bots.

Allow

Explicitly allows crawling of a path. Overrides Disallow if more specific.

Disallow

Blocks crawling of a path. Use / to block everything.

Crawl-delay

Seconds between requests. Supported by Bing and Yandex (not Google).

Sitemap

Points crawlers to your XML sitemap. Must be an absolute URL.

Wildcards

*Matches any sequence of characters
$Matches end of URL

# Block all PDFs

Disallow: /*.pdf$

# Block query params

Disallow: /*?*

About Robots.txt Generator

Create a properly formatted robots.txt file for your website using our visual builder. Add multiple user-agent groups, set allow/disallow rules, and include your sitemap URLs — all without writing a single line manually.

✓Visual rule builder with bot presets
✓6 ready-to-use templates
✓17 common bot presets including AI crawlers
✓Crawl-delay & sitemap support
✓Copy to clipboard or download file
✓Wildcard path suggestions

Complete Guide to Robots.txt

Free Online Robots.txt Generator

Create a valid robots.txt file for your website with our free visual builder. Choose from pre-built templates for WordPress, e-commerce, or AI bot blocking, or build custom rules from scratch. The generator produces properly formatted robots.txt content ready to deploy to your website.

What Is robots.txt?

The robots.txt file is a plain text file placed at the root of a website (e.g., https://example.com/robots.txt) that tells web crawlers which pages or sections of the site they are allowed or not allowed to access. It follows the Robots Exclusion Protocol, a standard that has been used since 1994 and was formalized by Google, Bing, and other search engines.

Robots.txt Syntax

# This is a comment
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Crawl-delay: 10

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Robots.txt Directives

📋 Core Directives

  • User-agent: Target specific crawler (or * for all)
  • Allow: Permit crawling of a path
  • Disallow: Block crawling of a path
  • Sitemap: Location of XML sitemap
  • Crawl-delay: Seconds between requests

🔤 Pattern Matching

  • * — Wildcard (any characters)
  • $ — End-of-URL anchor
  • /*.pdf$ — Block all PDF files
  • /dir/*?* — Block URLs with query params
  • /page* — Block paths starting with /page

When to Use robots.txt

✅ Good Uses

  • Block admin, login, and private areas
  • Prevent crawling of duplicate content
  • Reduce server load from aggressive crawlers
  • Block AI training crawlers
  • Hide staging or test environments
  • Point to your sitemap

⚠️ Limitations

  • Not a security mechanism — malicious bots ignore it
  • Does not remove pages from search indexes
  • Not all crawlers respect Crawl-delay
  • Blocked pages can still rank if linked externally
  • Cannot block specific IP addresses
  • Use noindex meta tag to prevent indexing

Blocking AI Crawlers

Many website owners want to prevent AI companies from using their content for training. Common AI crawlers include GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended (Google AI), and anthropic-ai (Anthropic/Claude). Use our “Block AI Bots” template to quickly set up rules that block these crawlers while keeping search engines allowed.

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

How to Deploy robots.txt

  1. Generate: Use the builder above to create your robots.txt content.
  2. Download or Copy: Click the download button or copy the output to your clipboard.
  3. Upload: Place the file at your website's root directory so it's accessible at https://yourdomain.com/robots.txt.
  4. Verify: Visit the URL in your browser to confirm it's served correctly.
  5. Test: Use Google Search Console's robots.txt tester to validate your rules.

robots.txt Best Practices

Always include a Sitemap: Adding a Sitemap directive helps search engines discover your content faster, especially new pages.

Be specific with Disallow: Avoid blocking entire directories unless necessary. More specific paths give you finer control.

Test before deploying: An incorrect robots.txt can block search engines from your entire site. Always test with Google's robots.txt tester.

Use Allow to override: When a broader Disallow exists, use Allow for specific sub-paths that should still be crawled.

One file per domain: Each subdomain needs its own robots.txt. The file only applies to the domain it's hosted on.

Perfect For

  • Web developers
  • SEO specialists
  • Site administrators
  • WordPress site owners
  • E-commerce store owners
  • DevOps engineers
  • Digital marketers
  • Content creators
  • Startup founders
  • Freelance developers
  • Agency teams
  • Students learning SEO

🔒 Complete Privacy Protection

All robots.txt generation happens entirely in your web browser using JavaScript. No data is sent to any server. Your website configuration and rules remain completely private and secure.

Learn Robots.txt in Depth

Want the full technical explanation first? Read our complete guide on robots.txt syntax, crawler behavior, and SEO-safe best practices.

Read: What is Robots.txt?

Related SEO & Web Tools

<meta>

Meta Tag Generator

Generate optimized meta tags for SEO, Open Graph, and Twitter Cards.

Open Graph Preview

Preview how URLs look when shared on Facebook, Twitter, LinkedIn, and Discord.

* * * * *

Cron Expression Generator

Build cron expressions visually for scheduling tasks and jobs.

DNS

DNS Checker

Look up DNS records for any domain — A, AAAA, MX, TXT, CNAME, NS.

Domain Age Checker

Check when a domain was registered and how old it is.

S

SSL Checker

Verify SSL certificate details, expiry dates, and security configuration.