# /robots.txt
#
# Layout: comments only up here, and each group's lines kept together (no
# blank or comment lines inside a group) — some parsers end a group at a
# blank line. No "Allow: /": everything not disallowed is allowed, and
# first-match parsers would let it override the Disallows.
#
# Group 1 (everyone): /search stays out of indexes (the page noindexes itself
# too) and /api/ is internal. No blanket "Disallow: /*?" — the sitemap lists
# query-string ranking views (/benchmarks?view=socs, ?type=tablet) and
# filtered pages noindex themselves. Search and AI answer services crawl
# under this group: Googlebot (also Google's AI Overviews), Bingbot (also
# Copilot), OAI-SearchBot and ChatGPT-User (ChatGPT search), Claude-SearchBot
# and Claude-User, PerplexityBot and Perplexity-User.
#
# Group 2: crawlers that collect data to train AI models are not allowed
# (see /terms). Google-Extended and Applebot-Extended only control training
# use; they don't affect Google or Apple search results.

User-agent: *
Disallow: /search
Disallow: /api/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: cohere-training-data-crawler
Disallow: /

Sitemap: https://devcheck.app/sitemap.xml