Search Engine Rankings.COM.AURequest an audit

Technical SEO

Robots.txt, noindex and canonical tags

Choose a control based on whether the goal is crawl access, index eligibility or duplicate consolidation.

How to approach it

Write down the desired result first. Allow crawlers to see a noindex instruction on a page you want removed from search. Use a canonical signal for similar pages and robots rules for crawl management.

Test one representative URL per rule before applying it to a whole directory. A robots rule can also block assets or templates if the pattern is broader than expected. Inspect the live response, the relevant robots rule and any header-based directives. Keep staging behind access control where content needs to remain private.

Worked example

An internal search results page you want excluded may need a crawlable noindex. For confidential staging, use access controls. An existing indexed URL can still appear if crawling alone is blocked.

How to verify the change

Write down how you will verify the intended result. Check access to a crawlable noindex page after deployment and monitor whether an already indexed page drops out. For a consolidation issue, compare the preferred URL with the duplicate and verify navigation, sitemap and canonical all point consistently.

Keep a copy of old rules before editing and test where broad patterns overlap. In a redesign, a rule copied from staging may unintentionally reach production. Include the robots file and template directives in your release checklist.

Official reference ↗

Related guides