A short definition of robots.txt and how it helps communicate crawl rules to automated agents.
Published | Updated
Short answer
robots.txt is a public text file that gives crawl instructions to automated agents at the site level. It helps communicate what may or may not be crawled, but it is not a private access control system.
Why it matters
Because the file is public and easy to inspect, it is one of the first places to check when a site unexpectedly blocks or exposes crawl paths.
How to verify
Open /robots.txt directly.
Confirm the intended crawl rules are present and readable.
Check whether the sitemap location is declared when appropriate.
Sources and notes
Google Search Central documentation is a primary reference for crawler-facing robots handling.