Setting up a new site or have planned out customization of your wordpress blog. Now that you taken care of your title and meta tags and other basic SEO stuff. Make sure you do get your robots tag in place too.

Robot.txt plays a crucial role in Search Engine optimization. There are two ways in which one can use robot.txt that is control it directly though individual pages and the second one is folders. We can write the robots tag in the Meta tag of the page or upload a robots.txt to the root folder of the site.

A webmaster can specify whether he desires that a particular page be indexed or not by the search engine bots by placing the robots Meta tag in the HTML section of the site.
The following are few commonly used robots tag:

< content="NOINDEX" name="ROBOTS">
- It follows only links and doesn’t index content.
< content="NOFOLLOW, INDEX " name="ROBOTS">
- Does not follow links but indexes content
< content="NOINDEX,NOFOLLOW" name="ROBOTS">
- Does not index content and does not follow links.
< content="INDEX,FOLLOW" name="ROBOTS">
- Indexes content and follows links.
< content="NOARCHIVE" name="ROBOTS">
- Cache link is not shown in Search result pages.
< content="NOODP" name="ROBOTS">
- The tag is for Open directory projects- the title and description that is being used in ODP should be displayed in the search results.
< content="NOYDIR" name="ROBOTS">
- The yahoo directory title and description for the pages should be displayed in search results.

The above examples are of robots Meta tag that is when we write the robots in the HTML section. Another way of using robots is by uploading robots.txt on the root folder of the site.
An example of how the robots.txt file looks like is:
User-agent: *
Allow: /
Disallow: /admin*
Allow: *content*
Disallow: /test/
Disallow: /credit/
Disallow: /cgi/

This means that the search engine bots should not crawl admin files; folders named credit, test and cgi and should crawl content folder.

Hopefully this has helped you all clear the doubts about robots.txt

Tags: customization of blog, robots tag, robots.txt