Create a robots.txt file and reference your sitemap

A robots.txt file serves as the first point of contact between your site and any crawler — search engine, AI system, or other automated visitor — attempting to understand what content you want crawled and indexed versus what should be left alone, making this foundational file's accuracy genuinely important for how your entire site gets discovered and processed.

Referencing your sitemap directly within this file provides crawlers with an explicit, efficient path to your complete list of indexable pages, rather than requiring them to discover your site's structure purely through following links — this represents a genuine efficiency gain for crawl discovery, particularly valuable for larger sites or sites with content that might not be easily discoverable through link-following alone.

The genuine risk of getting this file wrong deserves real respect — an overly restrictive robots.txt can inadvertently block crawlers from content you actually want indexed, silently undermining your search visibility without any obvious error or warning, while an absent file leaves crawlers to make their own default assumptions about what should be crawled, which may not align with your actual genuine intentions.

This file requires periodic review as your site evolves — content and site structure changes over time, and a robots.txt file that was accurate at initial creation can become outdated as new sections get added or existing ones get restructured, making this a file worth revisiting rather than treating as a permanent, one-time configuration.