Download Speed vs Site Performance

I check Google's data on the Engineering Site all the time, again today I rechecked the Site Performance. At least for the last few weeks the time it takes to down-load a page has decreased [always a good thing]. The attached graphic shows the amount of time it takes Google to down-load pages on the site. The numbers don't represent a single page but the average of some number of pages Googlebot tried to read. I don't really think the data is that accurate, because it never takes 4 seconds to download a page, but I read the data because that's what Google is looking at to determine site performance.

There is no way to tell why my site is coming in faster. The server could be working faster, Google pulled fewer pages or Google pulled small pages; who knows? Well I can check and see how many pages Google crawled per day and it's about the same between now and mid June. Google Crawl Stats indicate an average of 673 pages per day over the last few months.

However; I've been working on making the site faster for months, but not really getting anything to work. The basic problem is that any time I update a page and make the HTML text smaller [less code], I add more data making the page larger. So I may make the html code more efficient [decreasing download times], but I add more human readable text increasing the download time.

The current data [below] indicates the average time to download pages on the site takes 2.6 seconds. This is only important because Google rates site by download speed, so it's Search Engine Optimization [SEO].

Related Blog Posts;
Page Download Times [7-15-2010] 3.1 seconds to download
Speed Performance Overview [6-16-2010] 2.8 seconds to download
Web Site Performance [4-22-2010] 3.7 seconds to download
Google now ranks pages by speed [4-14-2010] No speed data.
Website Speed Performance [4-3-2010] 3.7 seconds to load.

IE 8 takes forever to shut down

I'm not really sure why but Internet Explorer takes a long time to close down. When I close down a single tab, the tab closes right away. However when I close down the program it takes forever to shut down, with the first few tabs taking the longest amount of time. I always have three browsers open; IE, Firefox and Chrome.

I tried looking up reasons on the web, but no luck. Just a bunch of real old IE crashing issues or out of date info.

Anyway here is how the top five browsers are doing on the engineering site: Explorer usage continues to decline, while Firefox seems to have stalled. Both browsers seem to be getting hurt by the increase in Chrome usage. Chrome did not exist before 2009.



1/1/2010 to 8/21/2010
Internet Explorer = 49.08%  [800,493 visitors]
Mozilla Firefox = 35.15% [573,338 visitors]
Google Chrome = 8.66% [141,218 visitors]

Link Checker Firefox Extension

I went ahead and hand-checked about 20 pages in the OEM Manufacturers section. Now I did just check the entire web site the other day, but there's a catch. Those automated programs do not always find 'bad-links'.

If a site goes bad or out of business and their web site payment becomes due, then ads could pop-up instead of the old engineering information. Seems right, once the payments run out the domain registrar takes over and runs their own ads, and the site appears "good" to the automated software checker. So you still need to hand check links. Any way I only found one 'bad' link which pointed to an ad, out of 20 pages and 2 hours of checking.

I found Link Checker on the web today and I down loaded the program (but have not used it yet). What I have used is the Firefox Browser extension for Link Checker that checks links on a per page basis. Yes it's another automated program, but now if needed I can check a single page with out checking the entire site. Link Checker seemed to work well. I used it to check each of the 'A' pages in the OEM Company section.

While testing it I noticed that it checked all the internal and external links on a page. However when I went to the next page all my internal links went 'green' right away, telling me that the program was not rechecking them ~ which is good. Link Checker uses color to indicate if a link is good or bad, so some what easy to use. I just tried three other pages but all the links are coming up good. Maybe I'll post a comment in a few days if I do find a bad link with the program, or if I load and run the entire .exe version of the program. The best I can get is a Yellow link which I assume is a 'no robots allowed', not checked link.....

So it's not all about adding new pages, sometimes you have to insure that the pages that are already out there are good, or are otherwise not hurting the web site. I also should have said that the Link Checker program is fast, but only as fast as the site it's checking; right, because it has to wait until it gets a response from the external site.

How often should site links be checked

I normally try to check external links on the site every 3 months or so. I'd like to check more often but I just see it as a severe hit to my server, because the link checker program has to read every page. At the same time it's a lot of work to use this program because at any given time there are always a few dozen websites off-line for what ever reason, so I have to recheck those. Of course how many companies could go out of business or change hands in a three month time period.

Any way this time around there were about 20 bad links including 3 from me, pointing to a misspelled address on my own site. I still have a few more links in my to-do box, waiting a few days to see if they come back on-line. The attached graphic shows the data produced by the link checker program. Over 99.5% of the links were good, but I didn't track all the link conditions. The report indicates I have 7,815 external links and 1,917 internal [page-to-page] links; because there are 1,917 pages that make up the web site.

So if you have external links it's always a good idea to keep them up to date. You never know who buys up the link once its gone bad. Search engines don't like broken links and of course visitors never like bad links, they just make the site seem abandoned. Oh I should also say I tried using the program over the last few week but just kept getting to many broken internal links in the report making the report impossible to use ~ the program was over-loading my server. The program and server worked ok today....

At the same time I fixed those bad links I also made what ever updates were required on the 20 pages that were 'fixed', making the process take twice as long as it needed.

Click to enlarge the graphic.

Computer Generated Pages

So I don't really know how many engineers read this blog, but I really need to comment on all these auto generated pages on the internet. Any time I try to look up a part on the internet all the search results that come back are just PC generated pages with nothing more than 100's of part numbers. Now a few of the pages do have a data sheet but they make you click through three pages to find the pdf version. So it's a bit time consuming to find a data sheet for a part that's less than common.

Up until a few months ago a search on Google would let you select the bogus site so they would not reappear in the search results. So by the next search I wouldn't let many of these sites show up. However instead of the de-select function Google changed that to a 'vote-for' star. When I'm looking for these components I don't see many sites to vote for. I just tested out Yahoo and they did a bit better, but not by much.If these guys would just add some content they wouldn't need 10,000 auto generated pages with every part number known to man....

Anyway I vented ~ I was looking for a Zener diode LVA62 which was on page 1 of Yahoo and 4 or 5 on Google. But every week I'm looking up part numbers....

I should also say this occurs much more often when looking for a part that may no longer be second sourced or in production decline, a part that is no longer in full production. I don't really have the same problem with newer parts, never if I know the current manufacturer ~ I can just go to the OEM's site and get the data sheet from there.

Digital Living Network Alliance Products

I mentioned this DLNA certified BluRay player yesterday, thinking I might need it. I'm sure I would like to have DLNA but that may be years away. Because except for the BluRay player which I haven't even purchased yet no other gear that I have is compatible and I see no need to change any thing out until they break. Wireless stuff would be nice but I don't really think I would ever use any of it. I gave away the spare PC I had connected to the HDTV because I never used it, that was wireless too, but not to the TV. I assume the BluRay player would be called a Digital Media Player [DMP].

I would like a new PC but I've been waiting to figure out what they were going to do with USB 3.0 and for PCI-Express 3.0 to be released So I'm about a year away from getting a new PC, but maybe I don't need one [New PC Posting 4/23/10]. The Digital Living Network Alliance indicates that I would need a DLNA certified Network Attached Storage [NAS] device, which I could get instead of a new PC (I guess). I'm not really sure why I would need a network drive when a DLNA certified PC should work, I assume. I'm not really sure if a NAS is the same as a DMS [Digital Media Server] [HDD Vendors].

So the 2 year old 47" flat screen I just blogged about the other day does not appear to be compatible. The 53" or 57" floor model out in the front room has to be 10 years old and the 37" CRT in the back room is even older so this whole idea of caring about DLNA is out the window. I could be years away from getting a new TV.

So it would appear this blog posting is pointless, unless it helps someone like me to understand that unless their buying a whole new system DLNA is just not required.

DVD Player No Disk

I just tried out my KLH 221 DVD player after years of having the thing sitting in my closet. The company is either called KLK Audio or KLH Audio Systems, but I'm not sure of the correct name. Any way I haven't used the thing for about 5 years, and I have no idea when I purchased the player. I had the DVD player setup in a back room that I never used, so it never got a lot of play time, but it was working several years ago.

I tried 3 or 4 DVDs in it yesterday but each time the LCD screen would just display 'No disk', which I assume means it could not read the DVD. However now that I'm thinking about it I don't recall the thing even spinning up. I tried rebooting a few times by pulling the plug but nothing seemed to work. Maybe I'll try it out one more time. Now I did try a quick web search for the model KLH 221, but I couldn't find much or a manual. Most of the web posting were just 'trash' sites with the same re-posts over and over. It could be that this DVD player is almost ten years old, I think it was the first one I ever purchased. Could it be because the CMOS battery went bad and the player no longer remembers how to work, maybe I need some reset code? Oh the remote was still working, that was a surprise.

I do have another DVD player in the front room that I've wanted to replace with a BluRay player. So I guess it's time for a new BluRay for the front room and moving that DVD player into the back room, replacing the one that no longer works. I would have rather purchased a new BluRay in my own time and not because some other system failed. I already have a BluRay in my office, and I may just re-purchase another new Sony BluRay player. I like Sony gear for the front room because that would match the other Sony gear and should work with the Sony remote I use out there.

I see a few reviews that the newer BluRays are loading the disk faster, it's about time. I also see some that have DLNA capability, I guess I want that. I'll have to look around for a few more days.


Just in case the chart in the previous post is a bit busy or hard to read I show the same data a different way. It should be a little easier to see that the site is still doing better than any previous year, but down these last few months because that's the mid-year trend. Of course the visits are way down, but I've always found that the higher the hits the bigger the drops. Maybe one more month before hits start to increase again. Click the image for a larger view of the data.

Server Bandwidth increasing

Even as visitors to the web site have been decreasing over the last few months bandwidth has increased. Now it's normal for site hits to decrease during this time of the year, normal for my site.

In March there were 190,622 unique visitors to the site using 14.64G Bytes of bandwidth.
In June there were 156,355 unique visitors to the site using 16.15G Bytes of bandwidth.
In July there were 150,100 unique visitors to the site using 17.03G Bytes of bandwidth.

Checking Googlebot Crawl Stats the amount of data downloaded did double for the end of July, but only to a high of 28kBytes per day. However the number of pages crawled stayed about average at 812 pages per day. So the only assumption that can be made is that Google was reading pages with more pictures than normal. Some pages have more graphics than normal; also, some of the graphic files are local to the server and some are held out on Picasa. Of course the ones located on Google Picasa do not effect my bandwidth.

Now I have removed a few dozen pic files from Picasa over the last few months. Google started to rank web sites based on down-load speed and they considered getting the pic files from 'Google' Picasa as slow. Not because Picasa is slow, although it maybe, but because they consider looking up the DNS [address] of another web site as being inherently slow. The most relevant posting was Web Site Speed Enhancements.

Last month I did up-load a new site-map to the server which Google has been reading every few days. The site map is 300k Bytes which is the size of about 60 html files, or maybe 30 files if you were to count the pic files too. The reason for up-loading the sitemap was to try and get more pages included in Google's index, which I have but maybe at the cost of server bandwidth. June 30 had 1,504 pages in Google's search index [URL's in Web Index], and as of July 28 there were 1,782 files included in Google's Index.

The server counter AWSTATS indicates that Googlebot used 285.97MB of server bandwidth, and the spider from Yahoo used 227.35MB of bandwidth. I have a [BaiDuSpider] spider from China that used 681.53MBytes of bandwidth. I noticed that a number of internet posts have had issues with the amount of data being indexed by this particular spider. It would appear that over the previous few months BaiDuSpider was only reading about 15MB/months so maybe it just got around to reading the entire web-site.

Now checking Google Analytics I see no real increase in visits from China, a little over 3,000 a month for the last three months. The top 10 robots used almost 3G Bytes of bandwidth, it's to bad their not sending me more traffic.

Even Alexa used over 329MBytes to spider my site, to bad my traffic rank is down 26,000 [-10% Reach] but then that would figure because my hits are down as well.
Oh and I just noticed that Google is also reading my text files from my stats counter, so that's another 20M of data it got off my server.

I guess I should also mention that over the last thirty days I've added maybe 30 pages and maybe that many pic files so that would account for a bit of the increase too. Anyway check out the attached chart, the lowest trend is bandwidth [normalized] ~ click to enlarge.