Create a robots.txt file and reference your sitemap

20 min Impact: high Effort: low ✓ Scan-verified — no manual checkbox

Robots.txt is the first file crawlers check before touching anything else on your site — it tells them what they're allowed to access, and a misconfigured one can silently block your entire site from being indexed with no other visible symptom.

Robots.txt tells crawlers what to index. A missing file is fine — but a misconfigured one can silently hide your entire site from Google.

How to fix it

1

Check what your current robots.txt actually says

Visit yoursite.com/robots.txt directly and read it — confirm it isn't accidentally blocking everything with Disallow: /.
2

Reference your sitemap

Add a Sitemap: line pointing to your full sitemap URL — this helps crawlers discover your content faster.
3

Only block what genuinely shouldn't be crawled

Admin areas, internal search results, and duplicate-content paths are reasonable to block — your actual content pages should not be.

Common mistakes

A blanket 'Disallow: /' is often left over from a staging environment that got copied to production without anyone updating this one file — it's a common, easy-to-miss cause of a site being invisible to Google.

How you'll know it's done

robots.txt exists, doesn't block legitimate content, and references your sitemap.

H.I.V.E. checks this automatically

Fix it, then re-scan — the check confirms itself. No manual checkbox, the scan is the truth.

Run this check in H.I.V.E. →

Create a robots.txt file and reference your sitemap

How to fix it

Common mistakes

How you'll know it's done

Related checks

H.I.V.E. checks this automatically