Introduction

The robots meta tag is an optional HTML tag which can be set in the <head> part of any HTML document. It is a mean to provide information to web crawlers visiting the page and looks like this:

  <meta name="robots" content="index,follow">

By default, web crawlers index the pages they visit and follow all the links they can find on these pages. In this case, links are considered as DOFOLLOW links. It is possible to prevent web crawlers from indexing a page or to follow its links by setting the robots meta tag value to noindex and nofollow like this:

  <meta name="robots" content="noindex,nofollow">

It is also possible to control the following or not of a single link in a page (regardless of the robots meta tag value) with the rel attribute. For example:

  <a href="mysite.com/mypage.html" rel="nofollow">my link</a>

Even if the page is set as FOLLOW (by the way, robots meta tag values are case insensitive), the above link will not be followed by web crawlers.

When to use NOINDEX and NOFOLLOW?

One must distinguish between internal links (i.e., links pointing to pages belonging the same website) and external links (i.e., links pointing to pages located on other websites).

At the page level:

- There is no known situation where setting NOFOLLOW at the page level brings a benefit. In fact, even if a page is not indexed, it will accumulate PageRank (a.k.a. link juice). It will pass it to other pages, unless it is marked as NOFOLLOW.
- NOINDEX should be used when the page is still work in progress, or when it suffers from quality issues which need to be fixed or when all its links are pointing to external websites one does not trust or when its content is not very useful to search engine users or when it is obsolete. It can also be used to remove a page from search engine indexes or to avoid duplicate content.

At the link level:

- If a page is set as NOFOLLOW, it is pointless to set one of its link as FOLLOW, since i) NOFOLLOW has the priority and ii) when not specified, a link is always considered as FOLLOW by default.
- NOFOLLOW should be used on a link when the target page (or website) is not fully trusted or when it is a sponsored link (i.e., advertising).

Bing and Google allow additional values for the robots meta tag. Follow the links for more details.