(￣_,￣ ) 針對 AI，再次更新 robots.txt

今天再次更新 robots.txt 了。

因為現在 AI 愈來愈多，所以再次調整了自家的 robots.txt

針對 AI 的部分，參考Vivek Gite 的《How to block AI Crawler Bots using robots.txt file》這篇文章和robots.txt Validator and Testing Tool，將 robots.txt 改成了

關於 AI 的 robots.txt 部分

# Disallow AIs
# ###################################

# Chat GPT

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /


# Google Gemini

User-agent: Mozilla/5.0 (Linux; U; Android 12; zh-tw; Gemini 1.0 Pro; Build/T5) AppleWebKit/537.36 (KHTML, like Gecko) Version/10.0 Chrome/100.0.4896.127 Mobile Safari/537.36
Disallow: /
User-agent: *Gemini*
Disallow: /
User-agent: Google-Extended
Disallow: /

# Copilot

User-agent: *Copilot*
Disallow: /

# ByteDance (TikTok)

User-agent: Bytespider
Disallow: /


# Applie AI

User-agent: Applebot-Extended
Disallow: /

# Grok

User-agent: *Grok*
Disallow: /

# Perplexity AI

User-agent: PerplexityBot
Disallow: /

# Anthropic AI

User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: ClaudeBot
Disallow: /

現在完整的 robots.txt 為：

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

# Disallow AIs
# ###################################

# Chat GPT
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /


# Google Gemini

User-agent: Mozilla/5.0 (Linux; U; Android 12; zh-tw; Gemini 1.0 Pro; Build/T5) AppleWebKit/537.36 (KHTML, like Gecko) Version/10.0 Chrome/100.0.4896.127 Mobile Safari/537.36
Disallow: /

User-agent: *Gemini*
Disallow: /

User-agent: Google-Extended
Disallow: /

# Copilot

User-agent: *Copilot*
Disallow: /

# ByteDance (TikTok)
User-agent: Bytespider
Disallow: /


# Applie AI

User-agent: Applebot-Extended
Disallow: /

# Grok
User-agent: *Grok*
Disallow: /

# Perplexity AI
User-agent: PerplexityBot
Disallow: /

# Anthropic AI
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: ClaudeBot
Disallow: /