Direct answer

Redirect an obsolete URL to a real replacement, canonicalize an accessible duplicate, noindex an accessible page that should not appear in search, use robots.txt only to manage crawling, require authentication for private content, and return 404 or 410 when a resource is genuinely gone without a replacement.

What to remember
  • Start with the intended user, crawler, and search outcome instead of selecting a familiar SEO tag.
  • Canonicals consolidate accessible duplicates, while redirects replace the source URL with another destination.
  • A robots.txt block does not reliably remove a URL from search and can prevent a crawler from seeing noindex.
  • Private information requires access control rather than search-engine directives.

01

Definition

Decision tree comparing redirects, canonicals, noindex, robots.txt, authentication, and 404 or 410 responses
Choose a URL control by deciding whether the original URL should remain accessible, searchable, and crawlable. Credit: Rank Builder SEO Research Desk

A page can be duplicated, moved, hidden from search, blocked from crawling, protected from unauthorized users, or removed entirely. Those are different states. They require different controls.

The quickest way to create a technical mess is to begin with a favorite tag rather than the desired outcome. A team adds a canonical because the URL looks duplicated, adds noindex because it wants the page gone, blocks the URL in robots.txt because crawling seems wasteful, and then redirects it because somebody finally notices that users should reach another page. The resulting signals do not describe one coherent state.

Use this operating rule:

  • Redirect when the original URL should give way to another destination.
  • Canonicalize when the original URL must remain accessible but represents duplicate or substantially similar content.
  • Use noindex when the page must remain accessible but should not appear in supporting search engines.
  • Use robots.txt when the objective is to manage crawler access.
  • Require authentication when the information is private.
  • Return 404 or 410 when the resource is gone and has no suitable replacement.

Google treats redirects and rel="canonical" annotations as strong canonicalization signals, while sitemap inclusion is weaker. None of those mechanisms is a privacy system. How to specify a canonical URL with rel=canonical and other methods[1]

02

Mechanism

Begin with three questions.

Should users still access the original URL?

If the answer is no and a genuine replacement exists, redirect the original URL.

A permanent server-side redirect tells clients that the resource has moved. Google describes 301 and 308 as permanent redirect types and uses a permanent redirect as a canonicalization signal toward the target. Temporary redirects such as 302 and 307 are appropriate when the alternate destination is temporary. Redirects and Google Search[2]

At the HTTP level, 307 and 308 require the client to preserve the request method during automatic redirection. Historical behavior permits a client to change POST to GET after a 301 or 302. That distinction matters for forms, APIs, uploads, and other non-GET requests, although ordinary page navigation usually uses GET. RFC 9110: HTTP Semantics[5]

If users must continue reaching the original URL, do not redirect merely to manipulate search behavior. Decide whether the accessible page is a duplicate, an independently useful page, or a page that should be excluded from search.

Should the page remain independently eligible for search?

An accessible duplicate can declare a preferred representative with rel="canonical".

Typical cases include:

  • Tracking-parameter versions
  • Print views
  • Alternate sorting views
  • Duplicate product routes
  • Syndicated copies
  • Hostname or protocol variants during cleanup

A canonical is not a command to delete the alternate URL. Users and crawlers can still request it. Google can also select another canonical when the declared target conflicts with content, links, redirects, sitemaps, or other signals.

If the page must remain accessible but should not appear in search, use a robots meta tag or X-Robots-Tag header containing noindex. Google must be able to crawl the resource and extract the rule before it can process it. Block Search indexing with noindex[3]

Should crawlers request the URL?

Robots.txt governs crawler access. It does not reliably remove a web-page URL from search results. A blocked URL can still be discovered through links and can appear without a normal description because the crawler was not allowed to retrieve the page. Google explicitly recommends noindex, password protection, or removal when the goal is exclusion from search. Introduction to robots.txt[4]

This creates an important sequence problem:

Page is indexed
        ↓
Site adds noindex
        ↓
Site also blocks the page in robots.txt
        ↓
Crawler cannot retrieve the page
        ↓
Crawler cannot see the new noindex rule

When an already known page must be removed from search and later excluded from unnecessary crawling, allow the crawler to retrieve the noindex first. Confirm that the rule was processed before applying a crawl block, if a crawl block remains necessary.

Is the content private?

Use authentication and authorization.

A robots.txt rule is public. A noindex directive is a request to cooperating search engines. A canonical is a duplicate-content signal. None prevents a person, scraper, or disobedient crawler from requesting an otherwise public URL.

Private dashboards, medical records, customer documents, internal reports, and paid resources must be protected before the application returns their contents.

03

Examples

SituationPrimary controlSupporting cleanup
HTTP URL moved permanently to HTTPS301 or 308Update internal links, sitemap URLs, and canonicals
Old article replaced by a materially better guidePermanent redirectLink directly to the replacement
Printer-friendly duplicate must remain usableCanonical to the principal versionKeep links consistent
Tracking parameter returns the same pageCanonical or strip the parameterStop generating tracked internal links
Account dashboardAuthenticationAdd noindex as secondary defense if publicly reachable shell exists
Internal search resultsCase-specific noindex or crawl controlBound query-space generation
Service temporarily unavailable302 or 307Restore the original route when service returns
Expired product with a direct successorPermanent redirectExplain the replacement to users
Removed page with no replacement404 or 410Remove internal links and sitemap entry
PDF duplicates an HTML guideCanonical header or separate treatmentDecide whether both formats deserve search visibility
Filter combinations with no search valuePrevent unnecessary crawlable URLsNormalize and constrain filter links
Product variants with distinct intentSeparate stable URLs when justifiedUse consistent canonicals and variant links

A 404 means the server did not find a current representation or is unwilling to disclose one. A 410 is more explicit when the server knows the condition is likely permanent. Neither response needs to be “fixed” when it accurately describes the resource.

Do not redirect every removed URL to the homepage. A redirect should lead to a destination that meaningfully replaces the original request. Otherwise the response can confuse users and search systems while hiding the fact that the content is gone.

04

Boundaries

A URL-control decision is not complete until the surrounding signals agree.

Check all of these:

  • HTTP response
  • Redirect destination
  • Canonical annotation
  • Robots meta tag
  • X-Robots-Tag
  • Robots.txt rule
  • Internal links
  • Sitemap inclusion
  • Structured-data URLs
  • Alternate-language annotations
  • Navigation and breadcrumbs

Avoid these combinations:

  • A robots-blocked URL that relies on a newly added noindex
  • A redirecting URL that remains in the XML sitemap
  • A canonical target that is itself noindexed
  • A canonical that points through one or more redirects
  • A chain of canonicals
  • Different canonicals in HTML and HTTP headers
  • A canonical target with materially different content
  • A redirected URL that still receives prominent internal links
  • A private page protected only by robots.txt

Use the canonical fundamentals guide when the only question is how canonical annotations work. Use the page-with-redirect guide when Search Console reports a redirecting URL. Use the noindex guide or robots.txt blocking guide when one of those controls is already producing an indexing status.

For query-string inventories, continue with URL Parameters and SEO. During a migration, use the SEO migration launch checklist to align redirects, links, sitemaps, and canonicals at release time.

The durable rule is not “always use a canonical” or “redirect everything.” It is simpler: define the intended state of the original URL, choose the mechanism that directly creates that state, and remove signals that describe a different one.

References

Sources behind this record

  1. How to specify a canonical URL with rel=canonical and other methodsGoogle (accessed July 29, 2026)
  2. Redirects and Google SearchGoogle (accessed July 29, 2026)
  3. Block Search indexing with noindexGoogle (accessed July 29, 2026)
  4. Introduction to robots.txtGoogle (accessed July 29, 2026)
  5. RFC 9110: HTTP SemanticsRFC Editor (accessed July 29, 2026)

Corrections

Correction history

No corrections recorded.

To report an error, use the public corrections path.

Claim limit

Canonical annotations and redirects are signals processed by search systems rather than guarantees of immediate canonical selection, deindexing, recrawling, or ranking behavior.