What Is Robots.txt?
Van Isle SEO Knowledge Hub
Robots.txt is one of the oldest and most widely misunderstood technical SEO controls.
Its primary job is crawl management. It tells compliant crawlers which areas they may or may not request.
That is different from telling Google whether a page should appear in search results.
How Does Robots.txt Work?
The file normally exists at a domain's root, such as example.com/robots.txt.
Crawlers can request the file and interpret rules applying to their user agent.
Rules can permit or restrict crawling of URL paths, but implementation should be handled carefully because one incorrect directive can make important content inaccessible to crawlers.
What Does a Robots.txt File Look Like?
Content Asset 1
Basic Robots.txt Concepts
| Directive | Purpose |
|---|---|
| User-agent | Identify which crawler the following rules apply to |
| Disallow | Request that the crawler not crawl a path |
| Allow | Permit a path within a broader restricted structure where supported |
| Sitemap | Reference an XML sitemap location |
Does Robots.txt Prevent Indexing?
Not reliably.
Robots.txt controls crawling. A blocked URL can still be discovered through links or other references.
If the objective is to keep an accessible page out of search results, an appropriate indexing directive may be required instead.
Crawling and indexing are different
If Google cannot crawl a page because robots.txt blocks it, Google may also be unable to see a noindex instruction located inside that page.
What Is the Difference Between Robots.txt and Noindex?
Content Asset 2
Crawl and Index Control Comparison
| Method | Primary Purpose |
|---|---|
| Robots.txt | Manage crawler access |
| Noindex | Request exclusion from search indexing |
| Canonical | Indicate a preferred representative among duplicate URLs |
| Password protection | Restrict access to private information |
These controls solve different problems and should not be treated as substitutes.
Can Robots.txt Contain an XML Sitemap?
Yes. A robots.txt file can reference one or more XML sitemap locations.
This provides crawlers with another route to discover the sitemap but does not guarantee indexing of the URLs contained within it.
Should CSS and JavaScript Be Blocked?
Important resources required to render and understand webpages should generally remain crawlable.
Blocking critical CSS or JavaScript can prevent search engines from seeing the same meaningful page experience available to users.
This is part of broader technical SEO and rendering analysis.
What Are Common Robots.txt Mistakes?
One of the most serious mistakes is accidentally blocking a large section or an entire production website after launch.
This can happen when staging rules are copied into production during a migration.
Other mistakes include blocking resources needed for rendering, assuming robots.txt creates privacy, or using crawl blocking when noindex or canonicalization is actually required.
Can Robots.txt Protect Private Information?
No. Robots.txt is publicly accessible and should never be treated as an access-control system.
Private or sensitive content should use proper authentication, permissions, or other security controls.
Listing a secret directory in robots.txt can actually reveal that the path exists.
How Do You Audit Robots.txt?
Content Asset 3
Robots.txt Audit Checklist
- Confirm the file loads at the root.
- Review all user-agent groups.
- Check important paths for accidental blocking.
- Review staging-related directives.
- Check CSS and JavaScript access.
- Confirm sitemap references.
- Separate crawl controls from indexing controls.
- Retest after migrations or platform changes.
Frequently Asked Questions About Robots.txt
What is robots.txt?
Robots.txt is a file containing crawling instructions for compliant web crawlers.
Does robots.txt remove a page from Google?
Not reliably. Blocking crawling and preventing indexing are different objectives.
Where is robots.txt located?
It normally sits at the root of the hostname, such as example.com/robots.txt.
Can robots.txt block the whole website?
Yes. An incorrect broad disallow rule can prevent compliant crawlers from crawling an entire site.
Is robots.txt a security feature?
No. Sensitive information should be protected through proper access controls.
Can I include my sitemap in robots.txt?
Yes. Sitemap locations can be referenced within robots.txt.
Use Robots.txt for Crawling — Not for Every Technical SEO Problem
Robots.txt is a useful crawl-management tool when its purpose is understood. It should not be used as a substitute for noindex, canonicalization, redirects, or genuine access control.
Explore the Technical SEO guide, use the free SEO audit, or request technical SEO support.