# https://www.robotstxt.org/robotstxt.html # # Note on /go/:slug — these are outbound-redirect interstitials with click # tracking. They serve `x-robots-tag: noindex, nofollow` and the internal # tags that point to them carry rel="nofollow", so search engines do # not queue them, do not index them, and do not pass link equity through # them. We intentionally do NOT Disallow them in robots.txt because that # would only generate "blocked URL" warnings in webmaster tools without # adding any real protection beyond what noindex/nofollow already provide. User-agent: * Allow: / Disallow: /dev/ # AI search assistants — indexing allowed. # An LLM-friendly site summary is at /llms.txt; the full content export is at /llms-full.txt. User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Googlebot-Extended Allow: / User-agent: Google-Extended Allow: / User-agent: cohere-ai Allow: / User-agent: Cohere-AI Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: meta-externalagent Allow: / User-agent: DuckAssistBot Allow: / User-agent: YouBot Allow: / User-agent: Amazonbot Allow: / User-agent: MistralAI-User Allow: / # Bing / Microsoft Copilot — explicit allowlist parallel to the AI-bot block # above. These are technically permitted by `User-agent: *` already, but # named entries let Bing Webmaster Tools audit the site as # explicitly-allowed and surface it in Copilot AI answer panels with the # same posture as ChatGPT and Perplexity. User-agent: bingbot Allow: / User-agent: msnbot Allow: / # Block mass-harvest training crawlers (model-training scrapers without # the courtesy of an opt-out signal in their UA-named bot). User-agent: CCBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: ia_archiver Allow: / Sitemap: https://slypanorama.com/sitemap.xml