robots.txt, noindex and the tags that control search

Two controls, two jobs

The 2025 Web Almanac states the split plainly. Robots directives influence indexing and serving: whether a page can appear in results and how. The robots.txt file governs crawling: whether a bot may fetch a URL at all. For a directive on a page to take effect, the crawler must be able to reach the page, so a page blocked in robots.txt may never have its directives seen.

A crawler that is not allowed to fetch a page cannot read a noindex on it. MDN says the same: if a resource is blocked from crawling through robots.txt, any indexing rules in a robots tag or X-Robots-Tag header will not be detected and will be ignored. MDN also notes that a page blocked from crawling may still be indexed if other documents refer to it.

Neither control is enforced. MDN stresses that only cooperative robots follow these rules, and that they do not stop bad actors such as email harvesters.

robots.txt

robots.txt is a text file placed in the root of a website, for example https://www.example.com/robots.txt. MDN describes it as saying whether crawlers may access the whole site, or specified resources. A restrictive file can also prevent bandwidth being used by crawlers.

The simplest examples from MDN:

# Keep every crawler out of the whole site
User-agent: *
Disallow: /

This is what a staging site might have, and what a live site must never have. A file with no rules at all lets crawlers in.

What MDN says not to do

MDN's implementation guide says to use the file to reduce site load and to stop unsuitable content appearing in search results, and that it should not be used to hide private information. The file is publicly readable. Listing a sensitive path in it shows that path to anyone who looks, including attackers. Some robots, such as malware robots, ignore the file. Protect private pages with a login, not with a rule in this file.

What the file's status code means

The 2025 Web Almanac, which checked the files of a very large number of sites, adds some detail about how crawlers treat the file itself:

• 200 means a valid file was served. 85% of the requests returned it. A 200 only proves that the file exists, not that its rules are right.
• 404 means there is no file. Without one, the default is unrestricted crawling. 13% of requests returned it.
• 5xx is the risky one. The Almanac says a server error on the file can cause search engines to temporarily treat the site as blocked, until a cached copy or a later successful fetch is available. Fewer than 1% of sites showed it, but a hosting problem can cause it to happen to yours.
• An empty file is served by about 2% of sites. Most major crawlers treat it as permissive, but the standard, RFC 9309, does not define that behavior clearly, so lesser-known bots may differ. The safer choice is a valid file, or a 404 if you want no restrictions.
• Size. Google enforces a 500 KB parsing cutoff. 98% of files were under 100 KB, so the limit is rarely a problem.

The RFC the Almanac refers to is the Robots Exclusion Protocol, standardized by the IETF as RFC 9309.

What real files look like

The catch-all rule for *, meaning every crawler, appeared in 77% of the files. Rules for named crawlers are rarer. The Almanac found that rules naming AI crawlers grew fast in a year. GPTBot was named in about 4% of files, ClaudeBot in about 3.5%, and others in a similar range. The Almanac's reading is that robots.txt now has two jobs: managing visibility in search, and deciding whether content may be used to train AI models. Those are separate decisions, and the file lets you answer each.

Crawlers also differ in how they read *. The Almanac notes that Google's AdsBot ignores it, and that another crawler falls back to Googlebot's rules before applying it, so some bots need to be named.

noindex

There are two ways to ask that a page stay out of search results.

A robots meta tag, in the page's head:

<meta name="robots" content="noindex">

An `X-Robots-Tag` header, in the HTTP response:

X-Robots-Tag: noindex

The header works for files that have no HTML, such as PDFs and images. MDN calls it a de-facto standard, not part of any specification.

Both take comma-separated rules. MDN lists the common ones: index and follow are the defaults, noindex asks the robot not to index the page, nofollow asks it not to follow the links on it, and none equals noindex, nofollow. There are also rules for the preview shown in results, such as nosnippet, max-snippet and max-image-preview. Conflicting values, such as index with noindex, have undefined behavior.

To remove a page from results

MDN's advice: X-Robots-Tag: noindex will typically work, but a robot must first revisit the page to see it. So the page has to be crawlable. Do not block it in robots.txt at the same time. If the page should disappear for good, removing it so the address returns an error, or redirecting it, is another route. See redirects and status codes.

How common each is

The Almanac found a robots meta tag on 47% of pages, while the header was on under 1%. Both were present at once on 0.4% of pages, which it does not recommend, since two methods raise the chance of conflicting signals. The rules index and follow are unnecessary, because they are the defaults, yet they appeared on about 60% of pages. The Almanac suggests a popular SEO plugin that outputs them by default as a likely reason.

Which to use

The speed angle

Crawlers are visitors. They make requests to your server, and some are heavy. MDN notes that a restrictive robots.txt can reduce bandwidth use, and the Almanac observes that SEO tools can generate significant crawl activity, which is why so many sites name those tools in the file. A slow server also affects crawlers. See server response time.

Mistakes that hide a site

• A staging block that went live. A Disallow: / that stays after launch tells crawlers to stay out of the whole site.
• A site-wide `noindex`. WordPress has a setting for discouraging search engines. In current versions, Site Health includes a check that reports when it is on, labeled "recommended" in that case. Plugins have their own switches too.
• Blocking a page you want removed. The crawler can then never see the noindex.
• Using `robots.txt` as a lock. It is public.
• Two methods that disagree. A page with both a meta tag and a header should say the same thing.
• Tags in the wrong place. The Almanac notes that an invalid element in the head, such as an img or a div, can make the browser treat the head as ended early, which puts later tags into the body, where they stop working. About 10% of pages had invalid head elements. Check that your robots tag and canonical link are before any such element.

How to check

1. Open `/robots.txt` on your site and read it. Check that the status is 200 and the rules are what you meant.
2. View the page source for a robots meta tag, and look at the response headers for X-Robots-Tag in the Network panel. See the Network panel.
3. Run Lighthouse. Its Page is blocked from indexing audit fails when a header or tag blocks all crawlers, and the robots.txt is valid audit checks the file. In Lighthouse 13.5 the first of these carries by far the largest weight among the SEO checks. It does not fail for crawler-specific rules, such as one aimed at Googlebot, though it may warn.
4. After a change, allow time for crawlers to come back.

Check your own page

Run the free audit and read the SEO findings. Then open your robots.txt and your home page's source. Confirm that you are not blocking anything you want found, and not leaving anything open that you want private.

Sources

• SEO chapter, Web Almanac 2025: the split between crawling and indexing, the status codes and size of real robots.txt files, the rules used, robots directives, the canonical data and invalid head elements.
• robots meta tag, X-Robots-Tag, robots.txt and the robots.txt configuration guide on MDN.
• Page is blocked from indexing in the Lighthouse documentation, and the Lighthouse 13.5 source for the SEO audit weights.
• WordPress core: Site Health: the search engine visibility test.