# LLMs and Crawlers Policy for windsurfandsailpools.com  
# Goal: Maximize marketing reach while reducing junk bot traffic  

### 1. ALLOW AI CRAWLERS (FOR FREE MARKETING & TRAFFIC) ###  
User-agent: GPTBot              # Let ChatGPT cite/reference our site  
Allow: /  
User-agent: Google-Extended     # Allow Google's AI (Gemini) to index  
Allow: /  
User-agent: anthropic-ai       # Allow Claude to surface our content  
Allow: /  

### 2. ALLOW ALL MAJOR SEARCH ENGINES ###  
User-agent: Googlebot  
Allow: /  
User-agent: Bingbot  
Allow: /  
User-agent: Applebot  
Allow: /  
User-agent: DuckDuckBot  
Allow: /  

### 3. BLOCK LOW-QUALITY/SCRAPER BOTS ###  
User-agent: CCBot               # Common Crawl (often used for spam)  
Disallow: /  
User-agent: MJ12bot             # Known for aggressive scraping  
Disallow: /  
User-agent: SemrushBot          # SEO spam bots  
Disallow: /  

### 4. BLOCK SENSITIVE AREAS (EVEN FROM AI) ###  
User-agent: *  
Disallow: /wp-admin/           # WordPress backend  
Disallow: /private/            # Any internal tools  
Disallow: /tmp/                # Temporary files  

### 5. ENCOURAGE INDEXING ###  
Sitemap: https://windsurfandsailpools.com/sitemap.xml