Just though I'd give you a few tips on the robots.txt which all major search engines now recognize.

1. Creating your robots.txt

Simply create a text document and save the new document as robots.txt Do not use a html editor to create the file unless is has the ability to create a plain text document (ASCII). Most computers will allow you to create a text document using notepad.

Right click on your desktop
*

Choose new
*

Choose text document
*

Open the document you just created
*

Insert instructions to robots
*

Click on save as
*

Save document as robots.txt

2. Always validate your coding (http://tool.motoricerca.info/robots-checker.phtml)

3. What is a robot.txt?
It simply allows the search engines bots to crawl and index your site.

4. You should add these lines to your code if you want your whole site included in any search engines.

User-agent: *
Disallow:

5. Alternatives

This allows all robots to crawl all files except the images file.

User-agent: *
Disallow: /images/

This allows all robots to crawl all files except the images file and the contents.

User-agent: *
Disallow: /images/
Disallow: /content/

6. Disallowing a bot to search your site.

http://www.robotstxt.org/db.html

This shows a list of bot names for example this is how to block the google bot (this is just an example for all you people who want traffic do not do this!)

This denies access to Googlebot-image to any files in your domain

User-agent: Googlebot-Image
Disallow: /

This specifically denies Googlebot-image to your images file

User-agent: Googlebot-Image
Disallow: /images/

7. A robots.txt is not required on any site
It is highly recommended but always ensure you have done it correctly using the validator!

__________________