Technical SEO

robots.txt generator

Build a valid robots.txt with per-agent allow/disallow rules, a Content-Signal directive for AI usage preferences, and a sitemap reference.

Sitemap & signals

Sitemap URL

Tells crawlers where to find your XML sitemap.

Content-Signal (optional)

contentsignals.org — declare AI usage preferences. Leave blank to skip.

Group 1

User-agents (one per line)

Allow paths (one per line)

Disallow paths (one per line)

Group 2

User-agents (one per line)

Allow paths (one per line)

Disallow paths (one per line)

Group 3

User-agents (one per line)

Allow paths (one per line)

Disallow paths (one per line)

robots.txt·TEXT

# robots.txt — generated by sitehealth.octagramlabs.com/tools/robots-txt-generator

Content-Signal: ai-train=no, search=yes, ai-input=yes

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /checkout
Disallow: /login

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: Applebot-Extended
Disallow: /admin/
Disallow: /cart
Disallow: /checkout

User-agent: CCBot
User-agent: Bytespider
User-agent: Diffbot
Disallow: /

Sitemap: https://example.com/sitemap.xml