Operational meaning
Cooperative crawlers use its allow and disallow rules when deciding which paths to request. The file is public and should never be treated as a place to hide sensitive routes.
Common misunderstanding
robots.txt is not authentication and does not reliably keep a public URL out of search results.
References
No external reference is claimed for this plain-language definition.