Robots.txt Generator & Tester + XML Sitemap Generator

Generate, test, and validate technical SEO crawl directives (robots.txt) and structured XML sitemaps to control search engine access, protect crawl budget, and accelerate page indexation.

Robots.txt Directives

Quick Presets:

Target User-Agent *

Disallowed Paths (One path per line)

Allowed Paths (Overrides disallows, one per line)

Crawl Delay (Seconds, optional)

XML Sitemap URL Directive

Add AI Crawler Disallow Rules (GPTBot, ClaudeBot, PerplexityBot, ByteSpider)

Robots.txt Output & Offline Tester

⚡ Offline Directive Tester Console

Test any URL path against your active directives to verify crawler access:

Test URL Path

ALLOWED User-Agent: * is permitted to crawl /wp-admin/plugins.php

XML Sitemap Configuration

Website Base Domain / Homepage *

Website Page URLs (Paste one URL per line) * Enter absolute HTTPS URLs to compile into valid XML sitemap standard.

Default Change Frequency

Default Priority (0.1 to 1.0)

Lastmod omitted (real page modification dates are required) Image entries are not supported in this basic sitemap (xmlns:image)

Compiled sitemap.xml Output

5 URLs

100% Client-Side Generator: All crawl rules, directive calculations, and sitemap XML files are generated locally in your web browser. No site data or server paths are recorded or stored.

Technical SEO: Robots.txt vs. XML Sitemaps Explained

Robots.txt and XML Sitemaps are the foundational protocols governing how search engine web crawlers (like Googlebot and Bingbot) discover, crawl, and index website pages.

🤖 Robots.txt (Crawl Directives)

Function: Specifies which paths crawlers are allowed or disallowed from accessing.
Placement: Must reside at the root directory: https://example.com/robots.txt.
Purpose: Conserves search engine crawl budget and blocks private admin pages.

🗺️ XML Sitemaps (Indexing Roadmap)

Function: Lists all canonical, indexable URLs on your site for fast discovery.
Placement: Referenced in robots.txt or submitted to Google Search Console.
Purpose: Ensures newly published content and deep page links get indexed quickly.

Instructions

How to Use the Technical SEO Workstation in 4 Steps

1. Select Preset or Mode

Choose between the Robots.txt Generator or XML Sitemap Generator tab, or select a pre-configured CMS preset.

2. Configure Rules & Paths

Add disallowed paths, allow overrides, set crawl delays, or paste bulk page URLs.

3. Test Edited Rules Offline

Use the Offline Tester Console to verify whether specific URL paths are allowed or blocked for crawlers.

4. Download & Deploy

Download your ready-to-use robots.txt or sitemap.xml files directly to your web server root.

How Robots.txt Matching & Crawl Budget Allocation Work

Search engine spiders process robots.txt files top-to-bottom according to the Robots Exclusion Protocol standard. Crawlers select the user-agent block that matches their bot identity or fall back to the global wildcard (User-agent: *).

When evaluating paths, search engines compare character length: if both an Allow: and a Disallow: directive match a path, the most specific (longest) path pattern takes precedence.

Technical Directives Do's & Don'ts

Recommended Practices

Always reference your XML sitemap URL inside robots.txt.
Disallow internal site search result URLs (e.g. /search?) to prevent infinite loops.
Include explicit x-default or language paths when managing multi-regional sites.
Request that listed AI crawlers avoid your pages. Robots directives are voluntary crawl guidance, not access control.

Practices to Avoid

Do not disallow CSS or JavaScript files needed for Google visual rendering.
Do not use robots.txt to hide sensitive user data—use authentication instead.
Do not disallow pages that you are attempting to remove via noindex meta tags.
Avoid relative URLs inside XML sitemap <loc> tags.

Applications

Common Technical SEO Scenarios

WordPress & CMS Optimization

Block admin directories like /wp-admin/ while allowing critical admin-ajax.php resources.

E-Commerce Crawl Protection

Request crawl restrictions for cart, checkout and filter paths. Robots rules do not reliably prevent indexing or unauthorized access.

AI Scraper Management

Generate separate disallow groups for listed AI crawlers. Compliance depends on each crawler; this does not protect private content or guarantee search visibility.

FAQ

Frequently Asked Questions

Where should I upload my generated robots.txt file?

Upload the robots.txt file directly to the root directory of your website domain (e.g., https://yourdomain.com/robots.txt).

Does robots.txt guarantee a page won't be indexed?

No. Robots.txt prevents crawling, but if other external websites link to a disallowed URL, Google may still index the URL without reading its content. For indexing control, provide a noindex directive on a crawlable page; protect private pages with authentication.

How many URLs can a single XML sitemap contain?

The sitemap format supports larger files, but this browser tool intentionally limits each export to 1,000 unique URLs including the homepage and 250,000 input characters.

How do I block AI bots like GPTBot or ClaudeBot?

Check the "Add AI Crawler Disallow Rules" option in this tool to automatically append disallow blocks specifically for GPTBot, ClaudeBot, PerplexityBot, and ByteSpider.