Meta tags can be used to replace the robots.txt file, however this method is much more limited.

You can't specify a robot to exclude this way, nor can you prevent access to directories in an easy manner.

Four directives can be used in the robots meta tag.

The below example would tell the robot it may indeed index this page and follow links on this web page.

<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">

To allow all robots to index a page but not follow links on it, use the html meta tag line shown below.

<META NAME="ROBOTS" CONTENT="INDEX,NOFOLLOW">

An optional directive which may or may not work with most popular search engines is the revisit-after one.

<META NAME="revisit-after" CONTENT="15 days">

Replace the value in red with the number of days after which you want the robot to visit your web site again.

These meta tags are not recognized by all search engines, use a robots.txt file instead when possible.