# One group, deliberately.
#
# A crawler obeys ONLY its most specific matching User-Agent group and ignores every
# other group in the file — that is the robots.txt spec, not a quirk. This file used to
# carry fourteen named groups (GPTBot, ClaudeBot, PerplexityBot, Applebot, CCBot,
# Bytespider, Claude-Web, anthropic-ai, cohere-ai, …) whose entire body was `Allow: /`.
#
# Because those groups did not repeat the Disallow lines below, every one of those
# fourteen agents was exempt from them. Verified 2026-08-24:
#
#   /api/posts/   200 application/json  164057 bytes
#   /api/pages/   200 application/json   90255 bytes
#   /admin/       200 text/html          51896 bytes
#
# So the crawlers that matter most were the only ones invited to spend their budget on a
# JSON mirror of every page and post, plus the Payload admin shell. The named groups also
# bought nothing they did not already have: `User-Agent: *` allows everything here, so a
# per-agent `Allow: /` is a no-op with a footgun attached.
#
# Deleting them is the fix AND the policy. Every crawler — named or not, existing or not
# yet invented — now inherits this group: full access to the site, none to the API or
# admin. Do not re-add a per-agent group unless it genuinely needs different rules from
# this one, and if it does, repeat these Disallow lines inside it.
# `robotsPolicy.test.ts` fails the build otherwise.
#
# Two traps, recorded so nobody "optimises" this back:
#   - Google-Extended and Applebot-Extended govern TRAINING only. Absent = allowed, so
#     they need no group. Blocking Google-Extended does NOT remove you from AI Overviews;
#     blocking Googlebot does.
#   - GPTBot and OAI-SearchBot share crawl results, so allowing one and blocking the
#     other is a leaky distinction. Both are allowed here.
User-Agent: *
# Cloudflare Content Signals — a declaration of preference, not an enforcement mechanism.
# search: appear in search indexes. ai-input: be quoted/cited in AI answers (the whole
# point of the AEO work). ai-train: permitted — Parakeeto has no ads and no paywall, so
# blocking training buys nothing and risks fat-fingering the search crawlers.
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /admin/
# Payload serves every uploaded image from /api/media/file/<name>, so a bare
# `Disallow: /api/` blocks the site's own images. That is the cause of the "Blocked by
# robots.txt" bucket in the 2026-08-23 Search Console domain-property export. Google
# reported 3 URLs, but /blog/ alone links 50 distinct /api/media/file/ URLs and a GET on
# one returns 200 image/png — so 3 is a sample, not the population. Most-specific-rule-wins
# means this Allow un-blocks media without exposing the rest of the REST API.
# Keep this line ABOVE the Disallow it narrows.
Allow: /api/media/
Disallow: /api/

Sitemap: https://www.parakeeto.com/sitemap.xml
Sitemap: https://www.parakeeto.com/pages-sitemap.xml
Sitemap: https://www.parakeeto.com/posts-sitemap.xml
Sitemap: https://www.parakeeto.com/glossary-sitemap.xml