Which AI crawlers should I allow in 2025?

Allow ClaudeBot, GPTBot, GoogleBot-Extended, PerplexityBot, Anthropic-AI, and standard search bots (Googlebot, Bingbot). These are verified AI agent crawlers as of December 2025.

What is agent-discovery.json?

/.well-known/agent-discovery.json is a manifest that guides AI agents through your site's capability space, semantic coordinates, and priority-weighted navigation paths. It goes beyond basic agent.json by providing structured discovery guidance.

Do I need special hosting for AI agent optimization?

No. Basic AI agent optimization works with any hosting platform that lets you control robots.txt, serve static JSON files, and add HTTP headers. Most modern hosting providers (Cloudflare, Vercel, Netlify, traditional servers) support these features.

How long does it take for AI agents to discover my site?

After submitting your sitemap to Google/Bing and allowing AI crawlers in robots.txt, initial discovery typically happens within 1-7 days. ClaudeBot and GPTBot usually discover well-structured sites within 24-48 hours.

How AI agents discover your website

Last updated: December 16, 2025

This guide covers the fundamentals of making your website discoverable by AI agents like ClaudeBot, GPTBot, Perplexity, and other AI crawlers. These steps work for any website, regardless of your tech stack or hosting platform.

Step 1: Allow AI crawlers in robots.txt

Update your /robots.txt file to explicitly allow AI agent crawlers. While most default to allowing all agents, being explicit ensures compatibility.

# AI Agent Crawlers (2025)
User-agent: ClaudeBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Anthropic-AI
Allow: /

User-agent: GoogleBot-Extended
Allow: /

User-agent: ChatGPT-User
Allow: /

# Standard search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

# Link to your sitemaps
Sitemap: https://yourdomain.com/sitemap.xml
Sitemap: https://yourdomain.com/sitemap-pages.xml

💡 Tip: If you want to block AI training bots while allowing search/discovery bots, you can selectively disallow certain agents. However, most modern AI crawlers respect robots.txt and are safe to allow.

Known AI Crawlers (December 2025)

Bot Name	Company	Purpose
`ClaudeBot`	Anthropic	Discovery, indexing, training
`GPTBot`	OpenAI	Training, search integration
`ChatGPT-User`	OpenAI	User-initiated searches
`PerplexityBot`	Perplexity AI	Search indexing
`GoogleBot-Extended`	Google	AI training (Gemini, Bard)
`Anthropic-AI`	Anthropic	Research crawling

Step 2: Create and submit XML sitemaps

Sitemaps are the primary way AI agents discover your content structure. Create a comprehensive sitemap that includes all important pages.

Basic sitemap structure:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.com/</loc>
    <lastmod>2025-12-16</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://yourdomain.com/about</loc>
    <lastmod>2025-12-10</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <!-- Add more URLs -->
</urlset>

Submit to search engines:

Google Search Console: Add https://yourdomain.com/sitemap.xml
Bing Webmaster Tools: Add https://yourdomain.com/sitemap.xml

💡 Tip: For large sites, split into multiple sitemaps (sitemap-pages.xml, sitemap-blog.xml, etc.) and create a sitemap index that references them all.

Step 3: Create agent discovery manifest

The /.well-known/agent-discovery.json file provides AI agents with structured guidance about your site's capabilities and important endpoints.

See full examples and implementation details on the page...

Verification Checklist

☐ robots.txt allows ClaudeBot, GPTBot, PerplexityBot
☐ robots.txt links to sitemap.xml
☐ Sitemap submitted to Google Search Console
☐ Sitemap submitted to Bing Webmaster Tools
☐ /.well-known/agent-discovery.json is publicly accessible
☐ JSON-LD structured data on homepage
☐ JSON-LD structured data on key content pages
☐ Semantic HTML structure throughout site
☐ Clear heading hierarchy on all pages
☐ AI agent visits visible in server logs

Resources

ClaudeBot Documentation: anthropic.com/bot
GPTBot Documentation: OpenAI GPTBot
Schema.org Reference: schema.org
Sitemaps Protocol: sitemaps.org

Key Terms

Agent-First SEO Agent Engine Optimization (AEO) AI Agent Discovery Agent Discovery Manifest ClaudeBot GPTBot AI Agent Crawling Structured Data JSON-LD Semantic HTML

Last updated: December 16, 2025 | Open-source guide for AI agent optimization fundamentals