What is Googles Farmer Update

The Farmer update, or Panda update was one of a few hundred algorithm changes Google makes to its search each year. The Farmer update got its name because the change was aimed at removing content farms from the search results.
A content farm is a web site set up just to trap key-word searches, but really has little or no content. Maybe hundreds or thousands of articles designed to bring in search traffic. However the articles are usually poorly written and are just filled with key words to attract traffic.

The Farmer algorithm update went live in the US on February 24th, with an update to non-US searches some time later but has yet to be announced. So if you have a site in the US and saw a decline in search traffic from Google starting on or after the 24 of last month you could be effected by the Farmer update.

One of the few comments Google has made regarding the update is that a weak page on your site could effect the entire site [which is some what new]. In the past it seemed that a weak page with little or no content would just never be indexed by Google, but never seemed to draw down other pages in the web site. Any way the forums are full of people giving advice, but the poster are just guesses.

So any way I wanted to detail a few changes that I made to the web site, not that I know I've been effected.
1. Deleted a dozen orphan pages from the server. The orphan pages had been out there for years and were mis-spelled page addresses or abandon pages that were no longer used. At one point they had news groups or what ever linking to them, so I saw no reason to show a 404 page not found. But after a few years, I'm sure that no one is reading those old news group posts and clicking in.
2. Redirected another dozen pages to something else. These page were being used, but had more weight years ago. As time has past they slowly lost content due to links going bad, or no interest from me in trying to fix them; however 10 years ago they were valid pages.
3. I removed about a dozen pages relating to advice on using Adsense, as their page views had been falling over the last few years and they did not have any relation to an engineering topic.
4. I removed a few ads on some of the pages, only because the content had decreased over time, because links on the pages were no longer working. Removing an ad was part of a normal page review and had nothing to do with the algo update. How ever some people in the forums are saying to many ads could effect a pages ranking, and pages with ads above the fold ~ I think that's garbage.
5. I fixed a number of bad links on the site, but I do that all the time. Out-going links are always going bad
6. I removed dozens of blog posting from early years on this blog. I've also done this from time to time in the past. Some times I'll write a blog post to keep the blog going, but in the big picture the post is of no interest to any one but the few people following the blog. But if Google is going to rank the entire blog based on these thin posts, then it's better just to remove them. This blog had 534 posts in Dec, but now has 512.
7. I've been optimizing HTML code on the site to reduce loading time, but I've been doing that for a year now.
8. Removing pic files from Google Picasa and storing them on the server, which should help with page loading and I've also been doing that for a year now.
9. I removed the Google tool bar because it seemed it was tracking me more than the site visitors. The tool bar measures page loading time and was showing wide variations when the load times should have been more even. I figured the tool bar was tracking 'my' page loads bringing up slower pages looking to optimize. In other words I would open a 'slow' page to fix, but showing Google a slow page. So Google was seeing more slow pages because of my tool bar usage than it normally would.
10. Updating pages with the new faster Google Analytics tracking code. I was doing that any way, and again to decrease page loading time.
11. Updating or enhancing page content ~ I do that any way.
12. Deleted or removed dozens of out going links, that were valid but seemed to be outdated ~ maybe they had a copy right of 2004 on their page. Normally the page had to many links any way, so no harm done.
13. I added a few more no-follow tags to some out going links, again their pages seems outdated.

I've been making these changes either before or after the 24th of last month and I've seen no difference in page views. However any decrease in page views that has occurred has really only occurred to a few high page-view pages, other pages are up or down as normal.

Comparing Feb to March the site is down 10%
Comparing Jan to March the site is down 0%
However March is normally higher than either of the previous months.

At the same time it could be that my site was not hit by the Algo change, rather some other sites are just doing better in the SERPS now ~ there's no way to tell. In any event these changes are normal Search Engine Optimization or SEO type changes.

URLs in Web Index

This would be the third revision or up-date to the posting of the web stats showing URLs in Web Index.
The stats relate to this Engineering Web Portal.

The table of data below shows the number URLs or pages in Google's web index. First of all I don't like to generate new site maps that often because of the bandwidth requirements of the sitemap program. That is the program I run needs to check every page I have on the server, so it's a big hit to my server. But if you look at the data and the increasing number of URLs in the web index you will notice that generating a sitemap over and over again is not required.

However I do recommend running a new sitemap if you have a large amount of new pages. The number of URLs in the web index always increases after a site-map is generated. However you'll notice that in some cases the number of indexed pages falls off a few days later. I assume the program reading the sitemap finds the new pages and adds them to the web index, but sometime later another computer algorithm determines the page should not be index and they drop off the index.

01/21/11 = 1,938 urls in web index [Site-map loaded]
11/30/10 = 1,875 urls in web index.
11/04/10 = 1,854 urls in web index.
10/23/10 = 1,805 urls in web index.
07/24/10 = 1,779 indexed pages.  [Site-map loaded]
07/01/10 = 1,535 indexed pages.  [Site-map loaded]
06/30/10 = 1,504 urls in web index.
06/24/10 = 1,455 urls in web index.
06/18/10 = 1,426 urls in web index.
05/31/10 = 1,400 urls in web index.
05/22/10 = 1,394 urls in web index.
04/07/10 = 1,309 urls in web index.
03/27/10 = 1,322 indexed pages. [Site-map loaded]
12/19/09 = 1,481 indexed pages. [Site-map loaded]
12/13/08 = 1,318 indexed pages. [Site-map loaded]

There are two reasons why the number of URLs in the web index are increasing. First I generate at least two or three new pages every month, sometimes many more. So you would except to see an increasing number of pages included in the web index as those new pages are found and included. Secondly I'm always updating preexisting pages already residing on the web-site. So after time the up-dates may allow a page to be indexed, usually because it has more content [text]. Many times I'll add a new page topic that is little more than a graphic and a small description. But over time I get back to up-dating the page to include more data and so on. Once the page gets 'the required amount' of content Google included the URL into the web index.

How soon your new page shows up in the web index, with out a sitemap, depends on how often your pages are crawled. The chart shows that my site has 500 pages crawled every day, although you can't tell from that how many pages are re-crawled. So for my site any new page added is found within a few days, I assume 15 days ~ so I don't really need a sitemap.

Although it has little to do with the number of URLs in the web index, over and over I see new pages not receiving any incoming hits for months. That is even as a new page gets included in the web index, it takes three months for the page to really start getting any visits. Web Masters call that the Google sand box. So don't think that just because a page gets indexed that it will bring in a larger number of hits the next day. Also I have groups or sections of pages that have dropped of the index, for years now. Currently I have 1,946 URLs submitted and 1,875 URLs included. That three month "down-time" is also the same amount of time it takes a page to get a Google Page Rank. [if it ever gets one].

2N3485 Transistor Derating. 3 year old page with zero page rank.
Semiconductor Manufacturers 'L'. 5 year old page with zero page rank.
Electron Tube Classification 2 week old page with 0 page rank.
74L121 Monostable Multivibrator IC. 1 week old page with 0 page rank.

Note always run your site-map generator program at night or when you expect low incoming traffic. Remember that program is talking to your server, which would be in direct competition with your visitors.

Also I have no idea how to determine which pages of my site are not included in the web index, other wise I might fix those pages.

The number of web pages; URLs in web index change every week or so, the numbers listed above are just the ones I wrote down.
. err click the title to read the comments, if you didn't come to this specific topic. The blog compresses the comments section unless you're on that particular page. The comments are updates to the blog post.
.......

Disallow access to server stat files

So after 10 years on the web the spiders have found my server stat files, in this case AWSTATS. This is the program output I use to see data regarding site visitors; number of visits, page views, visit duration, key words and on and on. The program stores the data in large text [txt] files, but outputs the data to me in the form of charts and graphs. In fact the txt files are very large, multi-Mega Byte files, the graphs are some what smaller and more readable.

I would recommend that every web master add a 'Disallow' line to the robots.txt file to stop the web spiders from reading your stat files. In my case the line looks like this; Disallow: /awstats/.

The bottom curve is server bandwidth ~ the increase occurs as my stat files started showing up in my search results. Because the graph is set up to show number of visitors, the bandwidth is normalized. So the 100,000 horizontal bar which indicates 100,000 visits or page views... indicates 10GB of server bandwidth for the bottom curve. The 200,000 line indicates 20GB for bandwidth

I only noticed the server text files showing up in search result about a month ago, because normally I don't need to search my own engineering site ~ right, I wrote it. So I only added a 'block' to the robots file a few weeks ago; however I recommend that you block access now even if you don't have an issue. It only takes a few minutes to add and if you pay for bandwidth or blocked if you exceed bandwidth it may well be worth the time.

If you look back to 2006/2007 you can see that bandwidth tracked unique visits, but by 2008 the gap stated to widen. Two years before Google started to rank pages on down-load speed I had already started to make the web site more efficient.

Unfortunately now the bandwidth data is meaningless, because it only shows these large txt statistic files being downloaded. For example one 2.52MB txt file was downloaded 230 times last month, a 4.11MB file was downloaded 98 times. That's 328 visitors that used the search bar and received bogus results, are they going to come back for a second visit? Really its much worse, before I stopped counting, there were 1,206 people last month who thought that one of those text files was a valid search return.

Download Speed vs Site Performance

I check Google's data on the Engineering Site all the time, again today I rechecked the Site Performance. At least for the last few weeks the time it takes to down-load a page has decreased [always a good thing]. The attached graphic shows the amount of time it takes Google to down-load pages on the site. The numbers don't represent a single page but the average of some number of pages Googlebot tried to read. I don't really think the data is that accurate, because it never takes 4 seconds to download a page, but I read the data because that's what Google is looking at to determine site performance.

There is no way to tell why my site is coming in faster. The server could be working faster, Google pulled fewer pages or Google pulled small pages; who knows? Well I can check and see how many pages Google crawled per day and it's about the same between now and mid June. Google Crawl Stats indicate an average of 673 pages per day over the last few months.

However; I've been working on making the site faster for months, but not really getting anything to work. The basic problem is that any time I update a page and make the HTML text smaller [less code], I add more data making the page larger. So I may make the html code more efficient [decreasing download times], but I add more human readable text increasing the download time.

The current data [below] indicates the average time to download pages on the site takes 2.6 seconds. This is only important because Google rates site by download speed, so it's Search Engine Optimization [SEO].

Related Blog Posts;
Page Download Times [7-15-2010] 3.1 seconds to download
Speed Performance Overview [6-16-2010] 2.8 seconds to download
Web Site Performance [4-22-2010] 3.7 seconds to download
Google now ranks pages by speed [4-14-2010] No speed data.
Website Speed Performance [4-3-2010] 3.7 seconds to load.

Blogger has some new features

Blogger is finally coming out with some new settings, and it's about time.
I've been on Blogger for about 5 years now and it seems that Google never adds any thing new, at least until this year. It does add new templates from time to time, but I've never been interested in changing. Now at the beginning of the year Google allowed Amazon to display ads on Blogger, so that's different.

Now Google has added Stats, so you can tell who visits the blog or what page the visitor views. Up until now I had used the stats from the ads that I run to tell how often that the blog was being visited. I also used Google Analytics to see who then visited www.interfacebus.com and what page they came from. They're both round-about ways to tell how the blog is doing, but I never got around to adding a free counter to the blogger template.

Blogger only just started counting pages last month, so I only have one month of data. The page with the most visits was written in 2008, followed by one from 2010, 2006, and 2005.

I just tried to up-load an image and I see they changed that function too. Instead of 'up-load a file' I get add from Blogger, Picasa, or from an html address ~ so I'm lost and will not be adding a picture from the stats counter??

URLs in Web Index

I'm going to partially re-post a blog I wrote just a few weeks ago regarding web-pages included in Google's Index. I've only added a dozen pages in the last two months so the increase in the number of pages indexed has to be due the SEO changes being made to pre-existing pages already on the web. The Index History relates to an Engineering Portal on the web.

Index History:
6/30/10 = 1,504 indexed pages.
6/24/10 = 1,455 indexed pages
6/18/10 = 1,426 indexed pages
6/9/10   = 1,411 indexed pages.
5/31/10 = 1,400 indexed pages.
5/22/10 = 1,394 indexed pages.
4/7/10   = 1,309 indexed pages.
3/27/10 = 1,322 indexed pages. [Site-map loaded]
12/19/09 = 1,481 indexed pages. [Site-map loaded]
12/13/08 = 1,318 indexed pages. [Site-map loaded]


 That's not to bad, 12 new pages added, but 100 more pages indexed by Google.

Does the increase in indexed pages imply that all of a sudden the site will start receiving a great many more site visitors, not really. But it does mean that the newly indexed pages will show up in a Google search for that topic, maybe not first in the list but at least they will show up now.

 No Page Rank; Company index 'M'.

I think I'll run a new site map later tonight, once the traffic slows down. I'll get those dozen or so new pages included, and remove a few FAQ pages that have been de-linked over the last few weeks. I did have a few pages that are not being used but still had links pointing to them, but Google dropped them from the index long ago ~ time to removed the link and orphan the pages.
I'll add a comment to this post once Google has time to read the new sitemap.

Graphic; Number of visits to the web site 1/1/10 to 6/29/10 [mid-year update].
Year-to-date = 17.48% increase in visits over the same time period last year [192,858 more visits].

Pages Crawled Per Day

Crawl Stats;

Looks like Google is finally slowing down checking my Engineering Site. After 2 months of heavy Googlebot activity the spider seems to be taking a break, which is good. There's really no reason Google should be reading 2,000 pages a-day. The numbers shown in the graph are High number of pages, Average number and low number of pages spidered each day.

HTML Suggestions;

I finally got down to the point that Google does not have an suggestions of fixing any meta-tags. It seems like there were a few duplicate pages that were unused that I never fixed ~ so they may have been de-listed.Here is what Google has to say;
"We didn't detect any content issues with your site. As we crawl your site, we check it to detect any potential issues with content on your pages, including duplicate, missing, or problematic title tags or meta descriptions. These issues won't prevent your site from appearing in Google search results, but paying attention to them can provide Google with more information and even help drive traffic to your site. For example, title and meta description text can appear in search results, and useful, descriptive text is more likely to be clicked on by users."

Sitemaps;

1,653 URLs submitted, 1,426 URLs Indexed.

Google Caffeine and Indexing

So Google came out with a new indexing system which updates the Google index much faster than in previous years. The new indexing system is called Caffeine [Google Caffeine], and allows for a change in its index each night.

It was common knowledge that once every thirty days Goggle would shuffle it's indexed pages, so if you were number 2 in the search engine position [index] one month that same page would change position the next month. Some times the position would get a higher position, and sometime the index change would leave the page in a lower position.

However at the same time many of the new pages I generated would show up in the index the next day because I would blog about the new page address in the New Engineering Pages Blog. Google operates these blogs [blogspot] so they read or spider them all the time. Any new page mentioned in blogger gets noticed by Google much faster before it reads any external web site [I assume]. Before I started blogging I would wait 2 to 4 weeks before a page would get indexed, but after I started blogging any new page would get picked up within several hours.

So for Search Engine Optimization [SEO], blog about any new page you generate because it gets picked up much faster that waiting for the spider [Googlebot] to find it, and much faster than having to generate a new site map for each new page addition.

So what is Caffeine? Well it's Google changing your search engine position ever day instead of every month. However; because your page position might change ever day as it was displaced by one of my new pages, I'm not really sure I see the difference. However Google does indicate a recent drop in the number of my page that are being displayed in the search engine listing, or really the number of pages that are being clicked on.

So this graph depicts the the number of times my page is shown in the search engine results [blue] and the number of times somebody clicks on one of my pages [yellow] in the search results. A drop in click-through rate [yellow] in the last few days should indicate that although my pages are still showing up in the search results, they now appear lower down in the results ~ maybe page 2 of the results instead of page 1.
But at the same time Google Analytics reports that there has been no drop in visits, and I trust the Analytics report more than I trust the report from Google webmaster tools [graphic above].

If I run the above report for the term Can Bus I see no real reduction in the click through-rate.
While the graph shows all search queries [all search terms] that relate to my website.

Indexed URLs by Google

I'm always watching how many of my pages are indexed in Google's search engine, and a few days ago it reached 1,411 pages indexed. Here's a few data points over the last few months [years];

6/9/10   = 1,411 indexed pages.
5/31/10 = 1,400 indexed pages.
5/22/10 = 1,394 indexed pages.
4/7/10   = 1,309 indexed pages.
3/27/10 = 1,322 indexed pages. [Site-map loaded]
12/19/09 = 1,481 indexed pages. [Site-map loaded]
12/13/08 = 1,318 indexed pages. [Site-map loaded]

The number of pages most go up just after I add a sitemap, and then drop down a few weeks later after Google finishes reading them. However as I 'fix' the pages they slowly start to re-appear as indexed, but at a slower pace. Fixing a page is adding more data so it no longer appears as the page it was copied from.
No page rank; Component Manufacturers. Just one example of a page not indexed.....

I'm not going to show the graph but Google indicates that on average Google will spider 862 pages per day ~ every day. What is up with that, the site only has 1,600 pages? Google used 537MB of server bandwidth as the spider Googlebot down-loaded from the website.

Search Key Word vs. Search Key Phrase

For a number of years now I've noticed that many of my pages do not do well in a Google search when a one word phrase is used. I've never been able to figure out why my pages do better with two or more key words rather than one key word, but I guess I can't show up first for every key word.

So one example is the search term VME, which would be a one-keyword search. The VME page shows up in the ninth position on the first page. When the phrase is changed to VME Bus, the page shows up in the forth position [same for VMEbus]. When I add another key phrase, VME Bus Pinout, the page shows up as the first listing on the first page. First listing for the key-words VME Backplane, and VME Chassis also. Hmm, looks like VME Connector places first too, in both a text and image search.

I'm using the VME Bus as an example, but I could have used any computer bus because the same thing happens. I just don't under stand how Google determines what page should show up first when only one key-word is used.

Yes I know the rules, those other pages must be using the key terms more often at the beginning of their article. I' don't know, my pages do show up but I would rather have them show up higher for a few one word searches. I guess I need to somehow optimize the VME interface page up that it shows better in a Google search, maybe rewrite some of the text.

Blog Comments change

I made a change to how comments are added to a posting within this blog. Previously there was a 'Leave a Comment' line at the end of the posting. I think that text is gone now and replaced by a '# comments' which needs to be clicked on. So that text needs to be clicked, regardless of the number of current comments, it may say 0 comments. Now this only holds true if your on the main page or some archive page. If your within the posting than you will see the comments, if any, and a comment box.

So that's what I changed, I added the Comment Box to each posting. Now the comment box is within the blog post instead of on a pop-up page. However the comments are still moderated, as they always have been. Some people don't realize the comments need my approval before they get posted, because I see identical redundant comments come in.

Most of the comments in this blog are from me, because I use the comment field to up-date the blog postings. Also, like any other blogger I get comments that say very little but have a link embedded within the comment ~ I don't post those comments. Those comments are just spammers looking for a free link. Anyway it may be a few days before the links gets moderated and posted, or not.

Visitor Demographics, Web Site

Here is what Google Analytics has for the Engineering Site visitor demographics [year to date]. The graphic indicates visitors from the United States [US] only. Data is available for many other countries as well, but not all countries display this chart. There are also a number of additional tables of data that go along with graph, but there not as interesting.

Now I did get this chart from the Google Analytics, but it appears that the data was generated by Doubleclick [which Google purchased a year ago]. So I might assume that the data is not as good as if it really came from Google, because they track my visitors. However there are other sites that produce more or less the same data; but they use different methods of producing their data.

For example, Alexa produces some of the same graphs, but you would have to be using an Alexa tool-bar to be counted in their data. They indicate a small amount of Female visitors, while Doubleclick indicates none. Alexa also indicates many more visitors in the age range of 18-24, again doubleclick indicates zero visitors. I just looked at the Alexa data, but I do have a posting from 7/1/09 which shows what their charts look like [Alexa Demographics].

Quantcast also has some of the same data, but I'm not really sure how they get their information [see note below]. Here is another previous posting from 10/2/07 showing one of their charts [Quantcast data]. Their data is also different. I think it's so different I'll post their data to the right.  The time ranges between the two sets of the charts are different, but I'm just working with what I have. Either way, the two graphs show how or what different companies collect and how they display the data.

The point is who do you believe, and why doesn't Google use their own data. I guess Google is using the data from doubleclick because they already had the program. Then why doesn't Google send over the right data, they know every thing about my web site. Then again how would Google know if a visitor was a man or woman ~ there most have been some kind of sigh-up with doubleclick? 

Note: One method to generate information for Quantcast data is to add a small amount of code to your web page(s). For several months I had that code on my home page and a few other high traffic pages [out of 1,300 pages]. However I removed the code to increase page loading time. The real point is before I had added the code Quantcast had my monthly traffic at 9800 visitors, while after I added the code to those few pages the traffic went up to 39000 visitors. Currently it appears to be running around 22000, but that's not the point. How accurate is the data these sites provide.

Of course I don't care what my site's visitor demographics is, unless some company emailed and said they would only advertise if I had a particular kind of visitor ~ which has never happened.

I need to promote a few web pages that are getting no page views [less than 10 this year], sorry;
2N3867 Transistor derating curve............... 2N2944 Transistor derating curve.
2N6762 FET derating curve..................1N4454 Diode derating curve.
PCM Waveforms, IRIG.

Graphic: McDonnell Douglas KC-10 Extender in flight

Page De-listed and not found

I was working on updating a page the other day and I did a Site Search for it, so I could see how it displayed in the Web Browser. Well guess what, the page did not come up in the Google Search bar I have on the site. Two other pages that pointed to it showed up, but not the page it-self.
The page in question shows a Derating Curve for a 2N3762 Transistor, UA Package. However the two pages that showed up in the listing were for a Temperature Derating Curve for 2N3765 Transistor and a list of PNP Transistor Derating Curves. So the assumption is that the page relating to the 2N3762 PNP Transistor was delisted. Now I noticed that there is another related page; the main 2N3762 Transistor Design page and the 2N3762 Transistor in a U4 package which also should have showed up in the search results.

Now if I put a Google Site Search on my Engineering web site, I would expect Google to search my site ~ that's the point of the search bar. What Google does on their own site is up to Google, but my pages should come up on my site.

I was also working on another page [IRIG PCM Waveforms] and I checked Google Analytics stats for that page and only a few page views showed up. So I again did a Google site search and that page does not show as listed either.

These pages exist on the site, but many people may not navigate to the correct page to see the data, they may just use the search engine. I assume they click away when the correct page does not show up in the search.

Attached picture; A-10 Thunderbolt II In-flight, US Air force [USAF].

Page Optimization SEO

Just to follow up with the last posting regarding click-through rates from Google searches ~ I did look at a few of the Google search queries and worked on those related pages. Now I did not update the pages just to optimize them for the search engine. I used the list or my page in question and reviewed the web page to see if any additional work or data could be added to the page.

So in fact I did optimize several pages for the search engine, but only because I added more text to the page. I really just addressed page issues, if the page didn't require any work I left it alone. So I guess it was a good tool to use to find pages that need work. Well there are always many pages that need work, or maybe more detail, but there are always trade offs between what pages get up-dated.

Every Query in the list has a little star which I can high-lite so I'll know which keyword or query I've already worked. Of course there are a few thousand search terms in the list so it might take a little bit of time to work the entire list ~ like I'm going to do that.

Like any page up-date I make, I won't see any response or change in the page visits until the page gets spidered by GoogleBot [which could be a few weeks]. So that fact makes it hard to track page enhancements vs increased page views; as I made 14 page updates today, 30 pages yesterday and 40 pages the day before. What would I do keep a rolling list of page changes for 15 days until Google spidered them. Among other pages I updated include the 7 Segment Driver page and the 15 pin D-sub Connector page to name a few....

Oh, I added a comment to the previous post; I hadn't noticed but many of those numbers in the graph are the same [flat lines]. So the graph must be generalizing data, it's just not possible to have the same click-through rate each day.


Graphic: Iraqi T72 Tank Firing in the Desert [Soviet Designed Main Battle Tank].

Clickthrough Rate

Over the last month or so I've been updating pages to display the new Google Search bar, primarily because it uses 1kB less html code. So far just over a thousand pages have received the up-grade with the new code. But what about all the other page metrics I have to worry about, maybe it's time to work on a few of those; as in:

Page Bounce Rate

Pages with low Pageviews

Pages with a high Exit Rate
And on and on; there's just so many different ways to measure how a page is doing on the internet.

One metric I don't look at that often is search engine Click-Through Rate, or the amount of times a page shows up in a Google search but is not clicked on. Now the search term used or the query could be any term as long as it causes my engineering website to show up in the results returned in a Google search.

Of the 6,091 queries, Google provides no data at all for the lower 4,500 searches. I assume because the page impressions are to low [a percentage] to track; however I really don't know. There are another 1000 queries that display an impression but no Clickthrough data, maybe because the clickthrough rate is below 1%. Because the first term with a clickthrough shows up with a 1% clickthrough rate; which would be the worst search term with any data. The 'worst' Google search term in the report is "DVI" which I assume relates to the page I have on the DVI interface. That DVI page shows up on the first page as #8 out of 32,500,000 results. The DVI search term causes my page to be displayed in the Google search results 6,600 times with only 58 people clicking on the link [over the last month].

So there are thousands more search terms with different impressions and click-through rates. For example the term 'Derating' had only 58 search impressions but 12 click-throughs. So the question is, should I work on pages that have no or low search impression or web pages with a low clickthrough rate? What-ever; working any page based on this data would be Search Engine Optimization [SEO].

Related Blog posts 

[Custom Search Bar 4/2/10]
[Web Site Speed Performance 4/3/10]
[Web Site Speed Enhancements 4/15/10]
[Web Site Performance 4/22/10]

Graphic: Top Search Queries as Page Link Impressions vs. Clickthrough Rate.

Spider Crawl Data

So I've been blogging about page speed, loading time, or down-load time and how Google rates web sites. Well what is up with Google's spider. Google Bot has been hitting my site pretty hard. Now as I look at the Crawl errors report I find 12 pages that Google indicates as 'unavailable' ~ maybe the server is getting over loaded.

Now I have to ask if Google is the one that's slowing down my site as they continually pull down page after page. AWSTATS indicates that Googlebot has used up 198.13MBytes of bandwidth so far this month. The Yahoo spider was the next worse offender with 125MB of bandwidth used.

Was this a Catch 22 situation? I was updating hundreds of pages to reduce down-load times by adding a new search bar. GoogleBot picked up on the page up-dates and tried to retrieve them all within a few days ultimately slowing down my server. What is up with that? It may well be that I'll have to wait a few weeks before the data comes back to normal.

I mean look at the change in the time it required to download a page; a low of 68mS all the way up to 1,608mS. And if that data is true then why does the site performance tab indicate a 3.7 second average load time while the crawl rate indicates a 186mS average load time [0.186 seconds] ~ that's a pretty big difference..... I can only conclude that the page takes .186 seconds to download, but the Java Script for Google Analytics and any pic file from Google Picasa take up the rest of the time.

Web Site Performance

I've got a few more weeks of data from Google's Site Performance. The data seems to indicate that the site speed is about the same it was in the last posting. However now Google indicates I'm 7% slower, so I have to assume that the other sites being used as a comparison are also getting faster? Because it still indicates 3.7 seconds to download a page.

Their text; "On average, pages in your site take 3.7 seconds to load (updated on Apr 20, 2010). This is slower than 62% of sites. These estimates are of medium accuracy (between 100 and 1000 data points). The chart below shows how your site's average page load time has changed over the last few months. For your reference, it also shows the 20th percentile value across all sites, separating slow and fast load times.".

So far 833 pages have received the newer smaller html code for Google's Search Bar, but I guess the 1k of text reduction per page doesn't seem to be helping any. Looks like this is a scrolling graph, as Nov has fallen off the end and replaced by newer data in Apr. The Page Speed suggestions do not seem to be updating, as Google still shows the pages and suggestions as before.

Web SIte Speed Enhancments

So to follow up with 3 of the last four postings regarding page downloading times and so on. I'll go ahead and detail a few of the things I've been doing to the web site to speed up down-load times. Or really to decrease page down-load times, depending on the page that received a change or not. I really can't make a single change that would effect the entire web site, each page is completely separate from another page.

So the new Google search bar is now on 684 pages. Each page that gets the new search code sees a reduction in html code or text of 1,480 characters [1,480 bytes] ~ I started replacing the search bar code a few days before I posted about it with Custom Search Bar. Just 1k Byte may not sound like a lot of data, but in fact it is when you consider how often these pages are down loaded, perhaps 15,000 page requests per day. It's a big saving on server bandwidth [over time] and Google sees the page as 1K smaller too, which was the point. People really pay for server bandwidth by the month so my reduction would be 400,000 pages x 1,480 Bytes, once all the pages get switched over. The second benefit is that the old search code from Google used an Engineering site logo which Google saw as another DNS look up that was a drain on the page loading time [logo stored off-site].  So this change saves the site 1 DNS look up and 1KB per page.

Using data from Feedburner [the blog feed] and Adsense [advertiser] I determined that there were not that many people reading the blog as a news feed. Plus the news feed was not generating any revenue, so I decided to remove the feedburner banners from the web site. Right, why publicize; the banners take up space, slow the page down and produce no income from the site. In addition the banners required 663 characters of html text and required an additional DNS look-up. The down side is that the banners were only on about 6 pages, so the savings is small, but those 6 pages should make the entire site appear faster [to a small degree].

Five gif files have been removed from the site, two were reinserted into this blog. The attached graphic shows monthly traffic to the web site for 2009. In addition to the page losing the graphic and seeing the size reduction this blog gets a link from the web site indicating the new location of the graphic. The 5 pages also no longer require another DNS look-up because the graphics were out on Google Picasa. Now the FAQ pages never received that many hits and the gif's were out on Picasa so my server sees no change. However Google will see the loss of a DNS look-up and the disappearance of five 80K Byte pic files.

In addition to removing those pic files I also reduced the size of another 14 gif files, saving between 20 and 30K Bytes per file. Yet another small change, but these files were local so the server will also see a reduction.

Any single change is small but the aggregate speed increase to the web site should help. I'll find out in a few weeks when Google up-dates the Site Performance report again. It's hard to tell, but this is Search Engine Optimization [SEO] because Google uses down-load time as part of its Page Rank algorithm.

Google now ranks pages by Speed

Seems hard to believe, but they do indicate that one of the 200 different criteria for Page Rank is page loading time. Recall that just a few posts ago I indicated that their own speed rating was complaining about their own Google products [Web Site Speed Performance] being used on my pages. Even back in 2007 I had complained about the (then new) W3C strict coding style with the Physical Page Size post, as even the smallest function required a large amount of coding.

Anyway most of the comments (including mine) to Google's blog posting were negative, and for good reason. How do you trade off page content with page loading speed. Many people mentioned Google Analytics code [tracking] or Adsense code [ads] as issues with loading time. However there were two comments I would like to bring over from two different posters [each from a different Google Blog];

"Doesn't this punish the small operator who has less control over their, usually shared, hosting? Or those in countries that have lesser infrastructure? At the same time, allowing bigger business to throw money at the speed problem and gain a better ranking?"

"With the recent court ruling with the FCC vs. Comcast speed might be tiered or throttled in the future. Is that a concern?"

Anyway it's already been said on those blogs. It's the Gmail Buzz debacle 1 month later, I just don't get it ~ as I remove Google product the rest of the night.

The attached graphic is daily stats for the engineering website for March 2010 [generated by AWSTATS]. Nice to look at, but the real reason it's here is because it was just deleted from my site ~ replaced with a link to this posting. It was a large graphic that required an additional DNS look-up [because it was located out on Google Picasa]. The next few posts will also contain one of these FAQ pics as they are moved off the site to increase the speed of those pages. The pics also get dumped from Picasa.

Web Site Speed Performance

I figured I would follow yesterdays posting regarding adding the new, smaller, script for the Google search bar with how Google sees my site when down-loading. Remember their web spider, GoogleBot, reads 500 of my pages every day. See a previous blog posting on Special crawl setting. So Google would know if my server was slow or not.

Here is what Google had to say; "Performance overview
On average, pages in your site take 3.7 seconds to load (updated on Mar 26, 2010). This is slower than 55% of sites. These estimates are of medium accuracy (between 100 and 1000 data points). The chart above shows how your site's average page load time has changed over the last few months. For your reference, it also shows the 20th percentile value across all sites, separating slow and fast load times."

Google Webmaster Tools gives a lot of page examples and what I could do to speed them up.
Their first suggestion is to 'Enable qzip compression' to reduce the page size. That's a nice idea but it makes working on the page a bit hard. Why don't I just save the 2k and continue to replace the Google search bar. I mean I am careful about up-loading large graphic files. In fact for pic files that can't be reduce, get uploaded to GoogleSites, and I only use a link to the file.

Their second suggestion is to 'Minimize DNS lookups', well guess what the DNS lookups are being used to access Google products. There are three common look-ups that they are referring to.
A logo used with the old Google search bar, which gets removed as the new search bar replaces it.
The Google Analytics code that I use as the site counter, provided by Google.
Finally, Google is complaining about pic files that I'm storing on Google Picasa that it has to down load. I just posted about Google Picasa off-line the other day too and how I used Picasa to save bandwidth.

So there are three of the four things Google says is slowing down my site, and there all Google products, does that make any sense. The fourth compression issue may not be an issue at all if the rest of the Google code on my page was a tad bit smaller. For example the new search bar code that is much smaller than it's been over the last five years. The adsense code also got smaller a few years ago, but could also be smaller.