How do I use a Robots file or do you have an example of one?

dealsforums · #1 (**permalink**) 08-25-2009, 04:27 AM

Hi everybody,
Can anyone please post the reply that
How do I use a Robots file?
or do you have an example of one please reply me
Thanks in Advance

Donk · #2 (**permalink**) 08-28-2009, 12:08 AM

You'll find instructions for the robots txt file at robotstxt.org

Bob

dealsforums · #3 (**permalink**) 08-28-2009, 04:06 AM

ya i got it.. thanks for the link..

Donk · #4 (**permalink**) 08-28-2009, 04:14 AM

No problems.

One thing that was not included on that page was that you can use robots.txt to point to you (Google) xml site-map.

Code:

Sitemap: http://www.yoursite.com/sitemap.xml

Bob

Bill the Builder · #5 (**permalink**) 08-28-2009, 05:31 PM

I usually put the following text in a note pad file and save it as robot.txt. The robot.txt file is then uploaded to the primary page directory of your website. My understanding is that you are now telling the search engines that there are no restrictions on any of your pages for them to crawl. Obviously, if you want to block some pages, don't use it.

This is the text:

User-agent: *
Allow: /

It seems simple, but has worked for me.
Bill the Builder

shakir · #6 (**permalink**) 08-29-2009, 04:07 AM

Quote:

Originally Posted by dealsforums

Hi everybody,
Can anyone please post the reply that
How do I use a Robots file?
or do you have an example of one please reply me
Thanks in Advance

if you wants to hide any files / folders from Google bots , can use robot.txt. Just join with Google webmaster tool, and their u can create robot.txt

whitelee5 · #7 (**permalink**) 08-31-2009, 02:59 AM

You can control how search engines "crawl" your website by using the right commands on Robots.txt. For learning more about commands.

TechieGuy · #8 (**permalink**) 08-31-2009, 06:07 AM

I usually add such things as my Wordpress system files/directories in the robots txt file, as well as other directories and file types I don't want index. I probably have a larger robots txt file than most people.

joker · #9 (**permalink**) 11-06-2009, 03:48 AM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

here's how to create robots.txt... http://www.javascriptkit.com/howto/robots.shtml

WB Sponsors