So I just checked Google Web Master Tools to see how the site was doing [looking for any issues]. Plus I had just up-loaded a new xml sitemap. The last update of the xml site-map was in December, so this new one covers the 20 odd pages added in the last few months.
Anyway, because the site was off-line for a few hours the other day, I checked the crawl rate. I no longer have the option of changing the Google crawl rate [for Googlebot]. Instead of a crawl faster or crawl slower selection I see this message;
Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.
Under the Crawl errors page I don't see any issues. There are 16 pages not found, but those are all mis-spelled page addresses from other web sites [incoming links], which I can't do any thing about. So I check how GoogleBot is crawling the Engineering web site, all looks well. In fact it seems that over the last three years around 500 pages are crawled per day. Here is the site crawl rate history over the last few years [Engineering Blog with a search term of Crawl]. So what is the deal and what does Special Crawl Rate mean?
I tried a Google web search for the terms 'assigned special crawl rate' and I get a Google News group with dozens of people asking the same thing. Well News-Groups are not that great a place to get information, only because so many people post a reply, just to post with out ever answering the question. Some people said it was for large sites, others replied that they had a small site. Then people would say that's why 1000's of their pages are not indexed [which is different all together].
I could add to that and say my site does not even require a sitemap, any new page gets spidered with in about a month. Of course this blog gets spidered today, remember blogger is owned by Google. Oh if I didn't already say Google indicates 1,653 urls submitted in the xml sitemap, with 1,322 urls indexed.
8 comments:
3/29/2010 I just did a google search for the term "been assigned special crawl rate settings" to see how this post would show up in the listings. Page two, third down, or 13 out of 608,000. That's good but I don't think many people will get to the second page of the search. Even if they are like me and could not find an answer on the first page of results. I'm not even sure how fresh this topic is any more. Ah, indicates the page was spidered on March 27. And the sitemap has not been downloaded by Google again, still says Mar 26....
4/1/10 Just checked Google webmaster tools again, the crawl rate selections are back. So now I can change the crawl rate again ~ not sure why.....
4/7/10 Now Google indicates that only 1,309 pages have been indexed.
5/22/10 Google now indicates that 1,394 pages have been indexed, so the number of pages in the index is growing.
Google webmaster tools indicates that this blog has been assigned special crawl rate settings. You will not be able to change the crawl rate. But it should not matter much any new blog posting get spidered within a few hours, but I'm not really sure how long it takes to get these comments indexed.
5/31/10 Now there are 1,400 URLs indexed by Google. So over the last thirty days Google has indexed another 80 pages, using the same sitemap data. I can't tell which pages are now included, but any increase is good news.
Looks like there have only been about three or four new pages added to the web site over this time range, so still no need to generate a new sitemap yet, maybe next month.....
6/7/10 yet one more up-date with 1,411 indexed URLs.
Hi,
Did you notice a change in your rankings after Google assigned "special crawl rate"?
I put my under development new site live (with single page) and put some backlinks, and google assigned special crawl rate.... I am afraid it is penalizing the site!
No there was no change or has been no change in the page rank of the site. The index page has been a 5 for as long as I can remember.
With only one page, you should not care what the crawl rate is for your site, the bot only has to check that one page.
Post a Comment