The robots.txt file is used to control the crawlers activity on a website/blog. It will help you to keep some directories away from crawling while allowing some. For example if yu have two folders 1.Articles and 2.Javascripts – and if you wish to exclude Javascripts from crawling by robots, then you can command it on [...]
Posts Tagged ‘robots.txt’
How to submit a sitemap for Yahoo!
Google isn’t the only search engine online – remember? Yahoo is somehow neglected while optimising the blog/site by majority of webmasters. Google seems to be the favourite of many, for searching as well as optimising their sites.The reason – the major part of online traffic goes through Google and Yahoo shares only 1/4th of it. [...]