Robots.txt Optimization tips for bloggers

The robots.txt file is used to control the crawlers activity on a website/blog. It will help you to keep some directories away from crawling while allowing some. For example if yu have two folders 1.Articles and 2.Javascripts – and if you wish to exclude Javascripts from crawling by robots, then you can command it on the robots.txt file.

A few basics about what the robots.txt file is -
- It is found in the root folder, Ex:-www.google.com/robots.txt
- It’s a text file and can be edited
- It is used to command the robots what to crawl and what not
- It is used to help the crawlers locate the sitemap on your site

If you are on blogger platform, then you can’t upload the robots.txt file. Panic not – there is another option which you can utilize. I’ll discuss it towards the end of this article. First let’s discuss a normal robots.txt implementation on a hosted site.

Implementing the Robots.txt file on a web-hosted site(Wordpress)

Pre-requisites – I assume that you have a wordpress hosted site with Cpanel/FTP access.

- Find the file at your public_html folder. If it isn’t there, create a blank text document.

Excluding a folder from crawling by SE bots.
Suppose you don’t want Google to index one of your folders.
In the robots.txt file, you have to specify two things – which crawler agent(Google, Yahoo, MSN) do you want to keep out and – which folder/folders you want to exclude.

The general syntax to be written in the Robots file is this.

User-agent: *
Disallow: /yourfolder/

Here, user-agent:* means all search agents(Google,MSN,Yahoo etc).
/yourfolder/ restricts that folder from crawling. Note that the sub-folders will not be crawled too.

In order to keep all agents away from crawling ALL folders, use this code.

User-agent: *
Disallow: /

You can specify individual crawler agents with their names(replacing *) like google bot,lycra etc.If you are following a general command to all search engine crawlers, keep the * in the user-agent line.

Specifying a sitemap with the Robots.txt file
Due to the recent agreement with the major search engines, they have come up with a common command that they will follow to detect sitemaps from robots.txt file. The command is –

Sitemap: Sitemap url here

Robots.txt for Blogger users.

Blogger users cannot upload the robots.txt file instead, they can use the robots meta tag to control the crawling of bots on particular files.

These codes should be included in the HEAD section of the particular page template.(Enclosed in arrow brackets)

META NAME=”ROBOTS” CONTENT=”NOINDEX”

This command will not index the current page in which this code is included.

META NAME=”ROBOTS” CONTENT=”NOFOLLOW”

This command will not follow/parse the links present on the particular page where this code is present in the head section.Blogger users can use this option to their advantage when making posts.If you want every new page to be crawled by the bots, include the following code to head section of your blogger template.

meta name=”robots” content=”index, follow”

Happy driving the robots.

Possibly related SEO & Social Media Articles

• SEO Errors that you might have missed checking, but still showup
  There are of course a bunch of basic SEO elements that you should check your site for occasionally like the W3C validation, redirect errors, sitemap generation errors etc. These crop up unknowingly and are identified only when a problem occurs. Example – If you are using the All in One SEO plugin, in the settings [...]...
• Control robots.txt from your wordpress dashboard
  When robots (like the Googlebot) crawl your site, they begin by requesting http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and edit your robots.txt file from within Wordpress (using Options -> Robots.txt). We have already seen how to control robots crawling your website using optimised robots.txt useage and..Avoiding duplicate content using robots.txt [...]...
• Ensure effective Google crawling on blogs
  If you don’t know nothing about robots.txt and controlling robots through Meta tags -and don’t want to take the pain of editing the txt file in your root folder and disallowing each search engine crawler manually – worry not! There is an easy way to get this done in Wordpress. Goto Dashboard> Options> Privacy settings. You will [...]...
• How to submit a sitemap for Yahoo!
  Google isn’t the only search engine online – remember? Yahoo is somehow neglected while optimising the blog/site by majority of webmasters. Google seems to be the favourite of many, for searching as well as optimising their sites.The reason – the major part of online traffic goes through Google and Yahoo shares only 1/4th of it. Does that [...]...
• Create a free Google Sitemap
  Google Sitemaps is an excellent way to submit your site to Google. If you are not sure of what a site map is - Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that [...]...