Direct answer

Add one fully qualified Sitemap directive per sitemap or sitemap index to the root robots.txt file. The directive is not tied to a user-agent group, may appear multiple times, and helps crawlers discover sitemap locations without overriding crawl rules or guaranteeing that any listed URL will be indexed.

What to remember
  • The Sitemap directive requires an absolute URL including the protocol and host.
  • Multiple Sitemap lines are allowed, with one sitemap or sitemap index location on each line.
  • The directive is independent of user-agent groups and does not grant access to a file blocked from crawling.
  • Robots.txt discovery complements Search Console submission and does not guarantee sitemap processing or page indexing.

01

Definition

A robots.txt file can tell supporting crawlers where a site's sitemap or sitemap index is located.

The directive uses this form:

Sitemap: https://www.example.com/sitemap.xml

The value must be a fully qualified URL containing the protocol and host. A relative value such as /sitemap.xml is not the documented syntax.

Google says the Sitemap field is case-insensitive, while the URL value remains case-sensitive. The field is supported by Google, Bing, and other major search engines under the sitemap protocol. It is not tied to a particular User-agent group. How Google interprets the robots.txt specification[3]

The directive helps a crawler discover the sitemap location the next time it fetches robots.txt. It is not an indexing command. The crawler still has to request the sitemap, parse it, evaluate the URLs, and decide whether to crawl or index any listed resources.

A site can also submit its sitemap in Search Console. Search Console adds reporting about fetch history and processing errors, while the robots.txt directive gives crawlers a durable public discovery location. Google documents both methods as valid ways to make a sitemap available. Build and submit a sitemap[1]

02

Mechanism

Place robots.txt at the root of the host it governs:

https://www.example.com/robots.txt

A basic file can contain crawler rules and a sitemap declaration:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The Allow: / line is optional because crawling is generally allowed when no applicable disallow rule exists. The sitemap directive can appear before or after user-agent groups because it is not scoped to one group. Google's robots.txt creation guide uses the same basic structure. Create and submit a robots.txt file[2]

For multiple sitemaps, use multiple lines:

Sitemap: https://www.example.com/sitemaps/articles.xml
Sitemap: https://www.example.com/sitemaps/products.xml
Sitemap: https://www.example.com/sitemaps/images.xml

Google states that there is no limit to the number of sitemap fields that can be included in robots.txt. The practical limit is maintainability: a long manually edited list creates more opportunities for stale paths, duplicates, wrong hosts, and missing files. How Google interprets the robots.txt specification[3]

When many child files belong to one inventory, declare a sitemap index instead:

Sitemap: https://www.example.com/sitemap-index.xml

The index can list the child sitemap files. See XML Sitemap Size Limits for file limits and index organization.

The sitemap URL does not have to use the same host as the robots.txt file. Google documents cross-host sitemap directives, such as a sitemap served from a dedicated CDN or another verified site. This flexibility does not make every cross-host configuration wise. The sitemap still needs a stable public response, valid contents, appropriate ownership, and correct URL scope.

A sitemap field does not override a robots rule. If the sitemap file itself is blocked from crawling, declaring its URL does not magically make it fetchable. Google notes that the field may be followed by crawlers provided the sitemap location is not disallowed. How Google interprets the robots.txt specification[3]

03

Examples

One sitemap

Sitemap: https://www.example.com/sitemap.xml

Use this when one valid file contains the complete preferred URL inventory.

One sitemap index

Sitemap: https://www.example.com/sitemap-index.xml

Use an index when the site operates several child sitemaps and wants one stable discovery address.

Several independent sitemap files

Sitemap: https://www.example.com/articles-sitemap.xml
Sitemap: https://www.example.com/products-sitemap.xml
Sitemap: https://www.example.com/video-sitemap.xml

This can be useful when different systems generate independent files or when Search Console reporting is intentionally separated.

Different sitemap host

Sitemap: https://sitemaps.example-cdn.com/example-com-index.xml

A cross-host location can be valid, but test it as an external dependency. A certificate, DNS, access-control, cache, or deployment failure on the sitemap host can prevent discovery even while the main site remains healthy.

Host variants

These hosts have separate robots.txt files:

https://example.com/robots.txt
https://www.example.com/robots.txt
https://shop.example.com/robots.txt

If alternate hosts redirect to one preferred host, test how robots.txt is served on every publicly reachable variant. Do not assume one file automatically governs unrelated hosts.

Common syntax mistakes

Avoid:

Sitemap: /sitemap.xml

The value is not absolute.

Avoid:

Sitemap: https://www.example.com/a.xml, https://www.example.com/b.xml

Use one directive per line.

Avoid a stale environment URL:

Sitemap: https://staging.example.com/sitemap.xml

Production robots.txt should not advertise a preview or development inventory.

Avoid pointing to a redirect when the final sitemap URL is known. Declare the final public location directly.

04

Boundaries

A sitemap directive does not control page crawling.

This line:

Sitemap: https://www.example.com/sitemap.xml

does not allow or disallow any page URL. Crawler access is controlled by applicable User-agent, Allow, and Disallow rules. Search exclusion requires an appropriate index-control or access-control mechanism rather than a sitemap declaration.

A sitemap directive also does not establish that the file is valid. The location can still return:

  • 404;
  • 403;
  • 429;
  • 5xx;
  • a redirect loop;
  • an HTML error page with 200;
  • invalid XML;
  • an oversized file;
  • URLs on the wrong host;
  • redirects and noindexed pages.

Use Sitemap Could Not Be Read when Google cannot fetch or parse the declared file.

Do not hide important pages behind sitemap-only discovery. Google can use sitemaps to find URLs, but internal links place pages in the site's navigation and contextual structure. See Internal Links vs. XML Sitemaps.

Do not confuse robots blocking with removal from search. A blocked page URL may still appear based on other signals because the crawler cannot retrieve its content. See URL Blocked by Robots.txt.

Validate the public robots.txt response after every routing, CDN, framework, or hostname change:

Correct host
HTTP 200
Plain-text response
Expected crawler rules
Absolute sitemap URLs
No staging references
No duplicate obsolete directives
Every declared sitemap returns successfully

The directive is valuable because it is simple and durable. Keep it that way. A crawler needs a correct absolute location, not a sitemap scavenger hunt assembled from redirects, environment aliases, and three generations of forgotten plugins.

References

Sources behind this record

  1. Build and submit a sitemap — Google (accessed July 31, 2026)
  2. Create and submit a robots.txt file — Google (accessed July 31, 2026)
  3. How Google interprets the robots.txt specification — Google (accessed July 31, 2026)

Corrections

Correction history

No corrections recorded.

To report an error, use the public corrections path.

Claim limit

Robots.txt behavior differs among crawlers, and declaring a sitemap location does not establish ownership, successful fetching, canonical selection, crawling, indexing, ranking, or traffic.