Update robots.txt to disallow sensitive paths and application files

Beyond the basic robots.txt setup, this specifically means disallowing sensitive paths, admin directories, configuration file locations, and similar application-specific paths that should never be crawled or indexed, reducing the chance search engines accidentally surface something sensitive.

While robots.txt is not a genuine security control on its own (a determined attacker can simply ignore it), it does prevent accidental indexing of sensitive paths by legitimate search engines, closing an easy, low-effort information disclosure gap.

How to do it

1. 1
   Identify your genuinely sensitive paths

   Admin directories, configuration file locations, internal API endpoints, anything that should never appear in search results.
2. 2
   Add specific Disallow rules for each

   Beyond the general robots.txt setup, targeting these specific sensitive paths explicitly.
3. 3
   Remember this is not a real access control

   robots.txt only asks well-behaved crawlers not to index these paths, it does not actually block access, real protection still requires proper authentication.
4. 4
   Verify important content is not accidentally blocked

   Confirm you have not overly broadened a rule to block legitimate content along with the sensitive paths.