Just though I'd give you a few tips on the robots.txt which
all major search engines now recognize.
1. Creating your robots.txt
Simply create a text document and save the new document as robots.txt Do not use a html editor to create the file unless is has the ability to create a plain text document (ASCII). Most computers will allow you to create a text document using notepad.
Right click on your desktop
*
Choose new
*
Choose text document
*
Open the document you just created
*
Insert instructions to robots
*
Click on save as
*
Save document as robots.txt
2. Always validate your coding (
http://tool.motoricerca.info/robots-checker.phtml)
3. What is a robot.txt?
It simply allows the search engines bots to crawl and index your site.
4. You should add these lines to your code if you want your whole site included in any search engines.
User-agent: *
Disallow:
5. Alternatives
This allows all robots to crawl all files except the images file.
User-agent: *
Disallow: /images/
This allows all robots to crawl all files except the images file and the contents.
User-agent: *
Disallow: /images/
Disallow: /content/
6. Disallowing a bot to search your site.
http://www.robotstxt.org/db.html
This shows a list of bot names for example this is how to block the google bot (this is just an example for all you people who want traffic do not do this!)
This denies access to Googlebot-image to any files in your domain
User-agent: Googlebot-Image
Disallow: /
This specifically denies Googlebot-image to your images file
User-agent: Googlebot-Image
Disallow: /images/
7. A robots.txt is not required on any site
It is highly recommended but always ensure you have done it correctly using the validator!
Robots.txt Get the lowdown!
Linear Mode
