SilverStripe Robots.txt Module

A modern, dynamic robots.txt module for SilverStripe that works correctly with multi-domain, multilingual, and Subsites-based setups.

Features

• Dynamic Generation: robots.txt and llms.txt are generated on-the-fly, allowing for host-specific rules.
• Automatic Exclusions: Automatically disallows pages where ShowInSearch is set to false in robots.txt.
• Integration with SEO Sitemap: If wilr/silverstripe-googlesitemaps is installed, it can automatically include a Sitemap: directive and respect its exclusion rules.
• LLMs.txt Support: Generates an llms.txt file for Large Language Models.
• AI-Powered Summaries: Automatically generates page summaries for llms.txt using GPT (OpenAI).
• Elemental Support: Correctlty extracts content from Elemental areas for AI processing.
• CMS Configurable: Editors can manage all settings via SiteConfig.
• Sensible Defaults: Comes with a pre-configured set of common blocked paths (e.g., /cms/, /resources/).
• Developer Friendly: Highly extensible via an interface-based provider system.

Installation

composer require innomedia/silverstripe-robots-txt

Configuration

CMS Settings

Robots.txt

Under Settings > Robots.txt, you can:

• Toggle default disallowed paths (technical paths like /cms/).
• Toggle automatic exclusion of ShowInSearch = false pages.
• Toggle Sitemap module integration (excludes pages that are not in the sitemap).
• Toggle Sitemap reference line.
• Add raw custom robots.txt lines.

LLMs.txt

Unter Settings > LLMs.txt (oder Einstellungen > LLMs.txt), können Sie:

• Eine allgemeine Seitenbeschreibung angeben.
• Kontaktinformationen für die llms.txt hinterlegen.
• Ihren OpenAI API Key für die Generierung von Zusammenfassungen hinterlegen.
• Bis zu 10 Kapitel definieren, jeweils mit einem Titel und einer Auswahl relevanter Seiten.
• Das System ruft beim Speichern automatisch die Inhalte ab (inklusive Elemental-Blöcken) und generiert KI-Zusammenfassungen via GPT.

Die llms.txt wird nach einem modernen Standard formatier inkl. Seitentitel, Beschreibung als Zitat, Sitemap-Link und Kontaktinfo.

YAML Configuration

You can override the default blocked paths or add your own disallow providers:

Innomedia\RobotsTxt\Builders\RobotsTxtBuilder:
  default_disallowed_paths:
    - '/cms/'
    - '/resources/'
    - '/my-custom-blocked-path/'
  disallow_sources:
    - 'Innomedia\RobotsTxt\Providers\ShowInSearchDisallowProvider'
    - 'MyCustom\Provider\DisallowProvider'

Extensibility

Custom Disallow Rules

Implement Innomedia\RobotsTxt\Interfaces\DisallowSourceInterface and register it in YAML:

namespace MyCustom\Provider;

use Innomedia\RobotsTxt\Interfaces\DisallowSourceInterface;

class DisallowProvider implements DisallowSourceInterface
{
    public function getDisallowedPaths(): array
    {
        return ['/private-area/'];
    }
}

Custom Sitemap Resolver

Implement Innomedia\RobotsTxt\Interfaces\SitemapResolverInterface and register it in YAML:

Innomedia\RobotsTxt\Builders\RobotsTxtBuilder:
  sitemap_resolver: 'MyCustom\Provider\SitemapResolver'

License

See LICENSE (BSD-3-Clause).