robots.txt

[lenesyn]

What's the importance of robots.txt? How does it work?

[getreffs]

It tell search engine to index a page or leave it.
Using Dofollow will index your Pages
Using Nofollow will leave the page which has Nofollow index.
Index means that search engine will not pick these pages in SERP

(I am not correct, plz any one correct me)

[cal]

howd u add robots.txt 2 your site to make it do follow??

[fluff22]

You can't use robots.txt to make your whole site dofollow, you can only use it to make things nofollow. If you leave it blank, it will be dofollow but if you then have nofollow on your links your robots.txt will not override it. If you are trying to make your site dofollow, just don't put nofollow tags anywhere.

[brayanC]

A robots.txt file is used to tell search engine spiders (or bots) what content they should access or stay out of. For example you have two directories on your website like so: example.com/public and example.com/private and you only want the search engines to crawl and index the pages and content in the public folder you would use the robots.txt file. Personally I would also block the private file from being accessed by anyone by using some server side security such as the .htaccess file.

[stephen186]

Well there is very good explanation about robots.txt here at mattcutts blog
http://www.mattcutts.com/blog/robots-txt-remove-url/

[nicolson]

Robot.txt is server side file. and this is a notepad file.......this file we put in CGI Bin directory
Here's a basic "robots.txt": code

If you want to allow Google crawler then write this

ser-agent: Gooebot
allolow: /

If you don't want to allow Google Image crawler then write this

User-agent: Googlt-Image
DeDisallow: /

[nachiketkhanna]

A robots.txt file is used to tell search engine spiders (or bots) what content they should access or stay out of. For example you have two directories on your website like so: example.com/public and example.com/private and you only want the search engines to crawl and index the pages and content in the public folder you would use the robots.txt file. Personally I would also block the private file from being accessed by anyone by using some server side security such as the .htaccess file.

[Klingon]

You can get detailed explanation of robot.txt at http://www.robotstxt...robotstxt.html. I had the same question few weeks ago and was looking for answers. I found this website very useful.

[jamesmarnley]

Robot.txt is text file. this file we put in CGI Bin directory
Here's a basic "robots.txt": code

ser-agent: search engine crawler name
allolow: /

[heartofluck]

A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file.