How to build a sitemap for blogger and large websites?
| Tweet | Share |
Today, let’s see more about sitemaps. Every webmaster must have a sitemap ready for his site and submit it to google in order to get all the pages listed on Google.Sitemaps are of two types, as you know the HTML sitemap you use to navigate a site and second the sitemap used to help crawlers crawl the pages more effectively.
Why are they necessary?
Sitemaps are not necessary.(Yep i said that) Even if you don’t have sitemaps the crawlers will crawl your pages and find the content. But, it is like letting them crawl in a dark room. What if you had a well lit room with all navigation and helpers around which will take them to each room? It will be more effective right? Sitemaps serve this purpose.
It has the site structure ready giving indication to the crawlers as to which are the folders/files that are important, which are not, which are the folders/files that are to be visited frequently, and which are the ones to be visited only once. This helps the crawlers to undersand your site more effectively.
Now, how to build a sitemap for blogger?
It’s very simple in Blogger. It only requires you to go to the Google Webmaster Central and add your site feed and the sitemap is automatically created. You can get detail instruction on this here.Make sure that you submit your full feed and not partial one.
Which is the best sitemap generator program around?
There are lot of free online and downloadable sitemap generators.
Here’s a simplified listing of what is best.
1- Python Scipt – This is the most difficult one to install. But if you are familiar with python, then this is the best one around.It’s automated and requires no additional support.I don’t recommend it for a beginner.Requires technical knowledge.
2 – Online sitemaps - This is best for small websites. It’s easy, simple and online.Just go to this site and submit your url.Fill in some basic details like time and priority settings for the files and click go!The whole sitemap will be generated online.You will get both ROR file and the Google sitemap XML file.If you are interested only in Google, use the XML sitemap.The format is according to Google sitemap protocol and is faultless.
Best choice for beginners and small websites of less than 500 pages.
3 – Gsite Crawler - This is a downloadable application. If your website is a bit large and you have time to tweak some settings and is serious about sitemap, then i would recommend this guy for you.
It requires you to give the website url, then select the types of files to be scanned from it, priority settings are automatically detected, and you can create bot Google sitemap and Yahoo url.
It has report generation as well that will give you an idea of how many urls were crawled and broken links etc.This is very useful while handling large sites.
How to make sitemap for large sites?
If you have really large websites for instance a one million page one, then it’s really going to be tough creating a sitemap. Practically this is possible with the Python script but if you are not okay with the technical stuff then you got to depend on sitemap generator programs.(If you don’t have a really large website the follwing piece of information may not help you.)
Step 1 - Download a free sitemap generator program like Gsite crawler.
Step 2 - Use it to crawl each folder of your website as separate projects.Make sure that you create a new database each time a new project is opened.
Step 3 - Now you have separate sitemaps for each folder.
Ex:- yourdomain.com/folder1 has a sitemap called folder1.xml and yourdomain.com/folder2 has a sitemap called folder2.xml
Step 4 - Download this simple index generator program.
Step 5 - Copy paste all the folders (containing the sitemaps) from thh projects folder of Gsite crawler(C:program files…) and put it into one single folder.
Step 6 – Run the index generator program against this parent folder.
Step 7 – Now a sitemap index would be created with links to all the child sitemaps but one problem, since in Gsite Crawlers projects folder(C:Program Files) each crawled folder will be named with underscore replacing the forward slash.
Ex:- yourdomain.com/folder will be named as yourdomain.com_folder
Therefore the sitemap index produced will have the links too this way.
Step 8 - Use notepad/wordpad to open the sitemap index file. Find and replace all the underscores with forward slash.
Step 9 - Upload the child sitemaps in the respective folders online.
Ex: – yourdomain.com/folder1..folder2 etc.
Step 10 – Upload the sitemap index file to the root folder and submit it to google.
Bingo! There you go you have now created a sitemap index and child sitemaps for a large website. Now submit it trough the webmaster central window and keep waiting!
What if its a dynamically updating site, and we want a sitemap? i.e. Its a site which is updated, say 20 times a day, but is not a blog, just a custom designed website. Is there an automated script available? Or what are the steps to keep in mind while making such an automated sitemap script?
Mohan’s last blog post..Knol… from Google. Art thou game?
Mohan, for a dynamically updating site, be it php, asp or anything..using a MOD Rewrite to make SEO friendly pages itself is half the task done. Then depending upon which pages carry important information, we can automate the sitemap generation process. For this Google suggests a Python script that can be installed in your servers. It has the risk of slowing down your server because of automation, but if you want to, give it a try.
https://www.google.com/webmasters/tools/docs/en/sitemap-generator.html#download
This and other topics here have helped me a lot. Thanks. There’s still a lot to improve with my blog but I’m slowly learning. And there’s still a lot to learn.
mama meji’s last blog post..So You Want To Switch from Blogger to WP…
thanks
Thanks for the tips on sitemaps.
thanks for that. What if you have giant size website, where pages will be auto-generated based on query of the users. And then you have millions of these pages? what will be the size of the sitemap file? and no matter what sitemap you create, you still dont get it complete because pages will be dynamic.
@Peter, Dynamic pages with “non indexable” URL’s are clearly out of the brackets – to be indexed. In such cases, we’ll have to use MOD rewrite to convert those dynamically generated pages to indexable format. And then run the sitemap generator program.
Once when the pages are huge in number, you’ll have to do them in batches and link the DB to an SQL database, or the program won’t hold it.
No matter how many pages you’ve got, it’s possible to get them all indexed in batches. At Alamy.com we’ve got millions of pages indexed with the same logic.
Very, Very Useful Tips, Information. Thank’s a Lot.
Mani, the link on number 2 is not working.
Thanks Ryan. Fixed now.
thanks too mani, built my site map already. nice guide.
Thank you for this!
Have to try now. This post seems the better one to build a sitemap.
Thanks for the thorough and detailed information.
Thanks for the tips. I’ve been facing difficulties in creating and installing sitemap for my blogger account. Unlike to wordpress that it’s easy and simple and there are plugins for this purpose
I can’t begin to tell you how much putting a sitemap on our site helped with SEO – it made a massive leap in keyword search rankings.
Underneath the heading “Now, how to build a sitemap for blogger?” You said
“It only requires you to go to the Google Webmaster Central and ass your site feed”
I think you meant that we should ADD and not ASS.
Your welcome
lol
Crap, how did I miss it ??
Thanks a lot David. Corrected now.
I am using Google sitemap generator wordpress plugin.
Is that good enough?
hi all i need to create a sitemap for my site http://www.diyanswerdirect.com so i can send it to google so my blog http://www.diyanswerdirect.com/blog can get indexed . i hope iam asking in the right place any ideas or sites i can get this information from thankyou
sitemap is very useful for blog life..I have wrote step by step to make google sitemap
Really interesting post also useful.Thanks
In your opinion… would you frown on including tags and categories from wordpress in your sitemap. I’ve heard a few people say that search engines might see that as “spammy”? Just curious.
Thank you for a great post! As an alternative to the the “2 – Online sitemaps” mentioned above I’d recommend http://www.web-site-map.com/ generator. It’s free too but can be used for much bigger websites
I’m wondering if its happening to others. Sitemaps via feeds submitted through the Google’s webmaster tools seem not to index all published articles.
I already have more that 140 published articles but “Sitemap URLs submitted on Google’s webmaster tools state” that it has only 26 URLs submitted with only 19 URLs indexed. My website has been online since 2008. I would guess that it shouldn’t only 26 URLs indexed, thus, only 19 indexed.
Is there a way to add a sitemap from third-party providers for blogger.com platform users? Thanks!
I didn’t know about the ability to create a sitemap index and child sitemaps.
Thanks for your suggestions…
I will give a try to the generator recommended by Brat, I have some problems with a large Website!
Thanks it was really helpful and I would like to know about sitemap just for blogger.. beacause I don’t create website other than wordpress.
Any way I liked what you have said to built sitemap