Search engine spiders and similar robots will look for a robots.txt file,
located in your main web directory.
This is a plain text file. Create or modify it with a text editor
and be sure to upload (FTP) it in ASCII mode.
If your web site doesn't work properly after adding or modifying this file,
remove it and check it for errors.