Controlling the Googlebot

How to control what gets indexed by Google and when? Most of the time, we want Google to snarf up as many pages as possible. However the indexing is not something you always want and then you have to go back to Google to actually have pages removed.

Before going into the deails, you need to know the different types of Googlebots

MediaBot - used to analyze AdSense pages
user agent “Mediapartners-Google”

ImageBot - crawling for the Image Search
user agent “GoogleBot-Image”

AdsBot - checking AdWords landing pages for quality
user agent “AdsBot-Google”


Finally here are the quick tips on how to control the Googlebot’s indexing.

1. At a site or directory level, use .htaccess to add password protection.

2. At a site or directory level, make use of a robots.txt file.

3. At a page level, use the noindex < meta> tag.

4. At a link level, use a nofollow attribute.

5. If the content has already been crawled, use the Google url removal tool as a last resort.