Update robots.txt to disallow sensitive paths and application files

Updating robots.txt to explicitly disallow sensitive paths and application files extends the basic robots.txt configuration discussed elsewhere with a specific security dimension — while the primary purpose of this file is guiding legitimate crawler behavior for SEO purposes, it can also serve a secondary function of discouraging legitimate crawlers from indexing paths that shouldn't appear in search results.

This deserves an important, honest caveat — robots.txt provides no genuine access control or security protection, since it's purely an advisory instruction that well-behaved crawlers voluntarily respect, while a malicious actor probing your site can simply ignore these instructions entirely, meaning this file should never be relied upon as an actual security measure for genuinely sensitive content.

The genuine value here is specifically limited to preventing accidental indexing of paths you'd prefer not to appear in search results — administrative interfaces, internal tools, or other paths that, while perhaps not carrying severe security risk if discovered, you'd still prefer weren't prominently indexed and discoverable through search engine results.

Any path containing genuinely sensitive information or functionality requires actual access control — authentication requirements, IP restrictions, or other genuine security measures — rather than relying on robots.txt exclusion, which provides no protection against anyone who directly navigates to or specifically probes for these paths regardless of crawler exclusion instructions.