robots.txt Builder

Configure crawler access rules with templates, test paths, and export a valid robots.txt file.

robots.txt Configuration

Site Root URL

Quick Templates

User-Agent Rule

User-Agent

Allow

Disallow

Sitemap URLs

robots.txt Preview

Copy this content to your robots.txt file

# robots.txt for https://example.com
# Generated with Webrenew SEO Files Builder
# 2025-11-09

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Path Tester

Test if a path is allowed for a specific user-agent

Path

User-Agent

User-Agent Rules

Control which bots can access different parts of your site. Use wildcards (*) and anchors ($).

AI Crawler Control

Block or allow AI crawlers like GPTBot, ClaudeBot, and others with one click.

Path Testing

Test any URL path to see if it would be allowed or blocked for different crawlers.