Search visibility
Robots.txt
What it means, how it is used and what to know for your project
Understanding the term
It is an instruction, not a lock. Reputable crawlers respect it; others do not. Never rely on it to protect anything confidential: a truly private page must require authentication.
Watch out for the most common misunderstanding: preventing a page from being crawled does not necessarily stop it appearing in search results. Removing a page from results requires a noindex tag, and the crawler must then be able to read the page to see it.
Related terms
Your turn
What if our next project
were yours?
Imagine your website
Your free mockup, with no obligation.