Disallow access to server stat files

So after 10 years on the web the spiders have found my server stat files, in this case AWSTATS. This is the program output I use to see data regarding site visitors; number of visits, page views, visit duration, key words and on and on. The program stores the data in large text [txt] files, but outputs the data to me in the form of charts and graphs. In fact the txt files are very large, multi-Mega Byte files, the graphs are some what smaller and more readable.

I would recommend that every web master add a 'Disallow' line to the robots.txt file to stop the web spiders from reading your stat files. In my case the line looks like this; Disallow: /awstats/.

The bottom curve is server bandwidth ~ the increase occurs as my stat files started showing up in my search results. Because the graph is set up to show number of visitors, the bandwidth is normalized. So the 100,000 horizontal bar which indicates 100,000 visits or page views... indicates 10GB of server bandwidth for the bottom curve. The 200,000 line indicates 20GB for bandwidth

I only noticed the server text files showing up in search result about a month ago, because normally I don't need to search my own engineering site ~ right, I wrote it. So I only added a 'block' to the robots file a few weeks ago; however I recommend that you block access now even if you don't have an issue. It only takes a few minutes to add and if you pay for bandwidth or blocked if you exceed bandwidth it may well be worth the time.

If you look back to 2006/2007 you can see that bandwidth tracked unique visits, but by 2008 the gap stated to widen. Two years before Google started to rank pages on down-load speed I had already started to make the web site more efficient.

Unfortunately now the bandwidth data is meaningless, because it only shows these large txt statistic files being downloaded. For example one 2.52MB txt file was downloaded 230 times last month, a 4.11MB file was downloaded 98 times. That's 328 visitors that used the search bar and received bogus results, are they going to come back for a second visit? Really its much worse, before I stopped counting, there were 1,206 people last month who thought that one of those text files was a valid search return.

Download Speed vs Site Performance

I check Google's data on the Engineering Site all the time, again today I rechecked the Site Performance. At least for the last few weeks the time it takes to down-load a page has decreased [always a good thing]. The attached graphic shows the amount of time it takes Google to down-load pages on the site. The numbers don't represent a single page but the average of some number of pages Googlebot tried to read. I don't really think the data is that accurate, because it never takes 4 seconds to download a page, but I read the data because that's what Google is looking at to determine site performance.

There is no way to tell why my site is coming in faster. The server could be working faster, Google pulled fewer pages or Google pulled small pages; who knows? Well I can check and see how many pages Google crawled per day and it's about the same between now and mid June. Google Crawl Stats indicate an average of 673 pages per day over the last few months.

However; I've been working on making the site faster for months, but not really getting anything to work. The basic problem is that any time I update a page and make the HTML text smaller [less code], I add more data making the page larger. So I may make the html code more efficient [decreasing download times], but I add more human readable text increasing the download time.

The current data [below] indicates the average time to download pages on the site takes 2.6 seconds. This is only important because Google rates site by download speed, so it's Search Engine Optimization [SEO].

Related Blog Posts;
Page Download Times [7-15-2010] 3.1 seconds to download
Speed Performance Overview [6-16-2010] 2.8 seconds to download
Web Site Performance [4-22-2010] 3.7 seconds to download
Google now ranks pages by speed [4-14-2010] No speed data.
Website Speed Performance [4-3-2010] 3.7 seconds to load.

Web Site Performance

I've got a few more weeks of data from Google's Site Performance. The data seems to indicate that the site speed is about the same it was in the last posting. However now Google indicates I'm 7% slower, so I have to assume that the other sites being used as a comparison are also getting faster? Because it still indicates 3.7 seconds to download a page.

Their text; "On average, pages in your site take 3.7 seconds to load (updated on Apr 20, 2010). This is slower than 62% of sites. These estimates are of medium accuracy (between 100 and 1000 data points). The chart below shows how your site's average page load time has changed over the last few months. For your reference, it also shows the 20th percentile value across all sites, separating slow and fast load times.".

So far 833 pages have received the newer smaller html code for Google's Search Bar, but I guess the 1k of text reduction per page doesn't seem to be helping any. Looks like this is a scrolling graph, as Nov has fallen off the end and replaced by newer data in Apr. The Page Speed suggestions do not seem to be updating, as Google still shows the pages and suggestions as before.

Web SIte Speed Enhancments

So to follow up with 3 of the last four postings regarding page downloading times and so on. I'll go ahead and detail a few of the things I've been doing to the web site to speed up down-load times. Or really to decrease page down-load times, depending on the page that received a change or not. I really can't make a single change that would effect the entire web site, each page is completely separate from another page.

So the new Google search bar is now on 684 pages. Each page that gets the new search code sees a reduction in html code or text of 1,480 characters [1,480 bytes] ~ I started replacing the search bar code a few days before I posted about it with Custom Search Bar. Just 1k Byte may not sound like a lot of data, but in fact it is when you consider how often these pages are down loaded, perhaps 15,000 page requests per day. It's a big saving on server bandwidth [over time] and Google sees the page as 1K smaller too, which was the point. People really pay for server bandwidth by the month so my reduction would be 400,000 pages x 1,480 Bytes, once all the pages get switched over. The second benefit is that the old search code from Google used an Engineering site logo which Google saw as another DNS look up that was a drain on the page loading time [logo stored off-site].  So this change saves the site 1 DNS look up and 1KB per page.

Using data from Feedburner [the blog feed] and Adsense [advertiser] I determined that there were not that many people reading the blog as a news feed. Plus the news feed was not generating any revenue, so I decided to remove the feedburner banners from the web site. Right, why publicize; the banners take up space, slow the page down and produce no income from the site. In addition the banners required 663 characters of html text and required an additional DNS look-up. The down side is that the banners were only on about 6 pages, so the savings is small, but those 6 pages should make the entire site appear faster [to a small degree].

Five gif files have been removed from the site, two were reinserted into this blog. The attached graphic shows monthly traffic to the web site for 2009. In addition to the page losing the graphic and seeing the size reduction this blog gets a link from the web site indicating the new location of the graphic. The 5 pages also no longer require another DNS look-up because the graphics were out on Google Picasa. Now the FAQ pages never received that many hits and the gif's were out on Picasa so my server sees no change. However Google will see the loss of a DNS look-up and the disappearance of five 80K Byte pic files.

In addition to removing those pic files I also reduced the size of another 14 gif files, saving between 20 and 30K Bytes per file. Yet another small change, but these files were local so the server will also see a reduction.

Any single change is small but the aggregate speed increase to the web site should help. I'll find out in a few weeks when Google up-dates the Site Performance report again. It's hard to tell, but this is Search Engine Optimization [SEO] because Google uses down-load time as part of its Page Rank algorithm.

Google now ranks pages by Speed

Seems hard to believe, but they do indicate that one of the 200 different criteria for Page Rank is page loading time. Recall that just a few posts ago I indicated that their own speed rating was complaining about their own Google products [Web Site Speed Performance] being used on my pages. Even back in 2007 I had complained about the (then new) W3C strict coding style with the Physical Page Size post, as even the smallest function required a large amount of coding.

Anyway most of the comments (including mine) to Google's blog posting were negative, and for good reason. How do you trade off page content with page loading speed. Many people mentioned Google Analytics code [tracking] or Adsense code [ads] as issues with loading time. However there were two comments I would like to bring over from two different posters [each from a different Google Blog];

"Doesn't this punish the small operator who has less control over their, usually shared, hosting? Or those in countries that have lesser infrastructure? At the same time, allowing bigger business to throw money at the speed problem and gain a better ranking?"

"With the recent court ruling with the FCC vs. Comcast speed might be tiered or throttled in the future. Is that a concern?"

Anyway it's already been said on those blogs. It's the Gmail Buzz debacle 1 month later, I just don't get it ~ as I remove Google product the rest of the night.

The attached graphic is daily stats for the engineering website for March 2010 [generated by AWSTATS]. Nice to look at, but the real reason it's here is because it was just deleted from my site ~ replaced with a link to this posting. It was a large graphic that required an additional DNS look-up [because it was located out on Google Picasa]. The next few posts will also contain one of these FAQ pics as they are moved off the site to increase the speed of those pages. The pics also get dumped from Picasa.

Web Site Speed Performance

I figured I would follow yesterdays posting regarding adding the new, smaller, script for the Google search bar with how Google sees my site when down-loading. Remember their web spider, GoogleBot, reads 500 of my pages every day. See a previous blog posting on Special crawl setting. So Google would know if my server was slow or not.

Here is what Google had to say; "Performance overview
On average, pages in your site take 3.7 seconds to load (updated on Mar 26, 2010). This is slower than 55% of sites. These estimates are of medium accuracy (between 100 and 1000 data points). The chart above shows how your site's average page load time has changed over the last few months. For your reference, it also shows the 20th percentile value across all sites, separating slow and fast load times."

Google Webmaster Tools gives a lot of page examples and what I could do to speed them up.
Their first suggestion is to 'Enable qzip compression' to reduce the page size. That's a nice idea but it makes working on the page a bit hard. Why don't I just save the 2k and continue to replace the Google search bar. I mean I am careful about up-loading large graphic files. In fact for pic files that can't be reduce, get uploaded to GoogleSites, and I only use a link to the file.

Their second suggestion is to 'Minimize DNS lookups', well guess what the DNS lookups are being used to access Google products. There are three common look-ups that they are referring to.
A logo used with the old Google search bar, which gets removed as the new search bar replaces it.
The Google Analytics code that I use as the site counter, provided by Google.
Finally, Google is complaining about pic files that I'm storing on Google Picasa that it has to down load. I just posted about Google Picasa off-line the other day too and how I used Picasa to save bandwidth.

So there are three of the four things Google says is slowing down my site, and there all Google products, does that make any sense. The fourth compression issue may not be an issue at all if the rest of the Google code on my page was a tad bit smaller. For example the new search bar code that is much smaller than it's been over the last five years. The adsense code also got smaller a few years ago, but could also be smaller.

Picasa Web Albums

So it would appear that Google's Picasa Web Albums is off-line, and I'm not really sure how long they've been down. The web site [Engineering Portal]  uses a lot of grapics and pic files, but not all of them are local to my server. I off-load some of them to Picasa to keep the server bandwidth down, which runs around 13G Bytes a month. Right, if someone else is serving the picture files then my bandwidth is not effected.

It's hard to say how many pic files reside on my server, as they could still reside in one of my directories but not used any more. So the count is an estimate but it would appear there are 687 picture files local to my server [give or a take]. Picasa on the other hand is serving another 919 graphic files [or not]. Because it appears that currently there are over 900 pic files not showing up on the web site.

So is it a good idea to up-load your files to another server? Well if your doing it to shows friends, sure. But what about if your trying to run a business? I guess I don't have an answer, but I am saving on my bandwidth. It's not saving me any money because my bandwidth limit is much higher than 13GB. What I am saving, or enhancing, is page load-time [I hope]. If the page is downloaded from my server and a pic file is downloaded from Picaca the visitor should see the page render faster. Or what if the files were on my server, what would be the bandwidth then?
Maybe a standard graphic file is 10 to 20K, and there are over 360,000 page views a month [over 380,000 page views this month]. That seems like a lot of down-loading [saved]......

So I see that the graphics are down in blogger as well [this blog]. I guess that makes sense because Google stores the blog graphics in Picasa too. I was going to attach a graphic showing server bandwidth vs page views, so that will have to wait. I'll add a link to this posing as a comment a bit later [SEO Techniques]. And it seems like just a few days ago the web site was off-line for a few hours too.

Oh, SEO stuff; the website link 'Engineering Portal' points to the normal site, I'm just using a different term to describe it ~ for the search engines. To try and insure the words are assigned to my home page , or related to...

Document Copyright Issues

I was doing a little research the other day looking up some data to support the chassis design section of the web site.

So I pull up this Handbook from the DOE and I'm speed reading, scrolling through the pages really quick. Stopping for a moment as I see some data on equipment rack grounding I like, other wise I keep scrolling.

I scroll past a table with a blue background, and I stop ~ all the tables on my web site once had blue back grounds. It's an Ampacity table for copper wire. Just above the table is data relating to derating wire bundles, just like my site.

Scrolling to the next page I see another table for chassis wire color coding, again copied from me. Even the text above the tables has been lifted. Even a disclaimer I used about not covering Military color coding is used.. The next 5 tables, with text, all from my site.

Now to be fair the data used in the tables covering color coding by voltage and TFE wire Ampacity were both derived from some other DOD standard. However there are two differences here; first any data derived from a military specification did not have a copyright, and second I didn't copy their table I retyped and reformatted the facts.

So what is the issue here?
a. First the US Government does not copyright their information. So if I copy data from a military standard I'm safe, it's free to use.
b. Second, I do copyright my data and it is not free to use. Also my site is not listed in the References section of the document, so I received no credit for the copied data.
c. Third, Once my data was put into that government handbook I lost my copyright because US Gov documents have no copyright ~ so now anyone can re-copy it.
d. Finally, Did my tax dollars pay someone to copy data from my site ~ it's a double hit. I paid someone to undermine my website.
e. Guess what, this posting on site visits (6/20/09) noted a 50% reduction in visits to the page that lost the data [Wire Insulation Color Coding]. All the text and tables that were copied originated from that page.

Notes;
1. For a number of years many tables on the web site had blue back grounds, but a few years ago all the tables were changed to a clear background. Changing the background color was one of a number of changes I made to reduce the bandwidth of the site. A clear background does not require any HTML code, while a colored background requires bgcolor=xxxx (what ever).
2. In some cases the department seal used by some agencies do have restrictions on use. I'm not sure if I can display the DOE seal or not, I know NASA does not like it.
3. The chassis design section covers a few dozen pages and addresses the issues that may come up while designing an equipment chassis, with Equipment Rack data..
4. I will be using data from this handbook, I have already posted a graphic for an equipment rack ~ again there is not copyright covering this DOE Handbook.

Web Site StatCounter

Here is the latest webStats for interfacebus.com. You'll note that there has been a bit of an improvement over the last few months.

Visits are up over the last year, that includes Unique visits and Total Visits. Normally the first of the year shows an increase in site visits, or the increase could be due to work done to the web site during the end of last year.

The increase in Page Visits tells me that the site Bounce Rate is decreasing. Which means that people are checking out more pages per visits ~ which is always a good thing.

However; over the last 3 and a half years these lines are flat ~ after 3 years of work.........

Web Page Code Optimizing and SEO stuff

No new page addresses in the last few days, so I'm enhancing some of the existing pages.

Google came out with some enhanced script for their Analytics software, so I've been updating that code on the web pages. The code is a tad larger than their old code but it does more (I think). Google Analytics is used to track site visitors and provides a lot of data.

While updating that code, I also update the java script code that displaces the ads on each page. The new java scrip is only half as big as the previous code. So I save several hundreds of bytes per page with each new up-load [first post on topic; Page Optimization].

SEO Up-dates:
Images ~ I've also added several new graphic files in the last few days, mostly on the new pages, and this is why; Hits from Image Referrals.
Links ~ I'm up-dating the link text for page to page links (the underlined blue text), to insure that it's descriptive. Don't link to a new page with 'more info', use a descriptive link.
Bounce Rate ~ Trying to decrease the bounce rate as a page gets updated. When the page has a high Bounce Rate [click away to another web site], I spend the extra time to insure all possible page to page links are there, and keeping people on the site.
Back-Links ~ Increase the external links that point to the main Engineering web site [this links counts]. Note that the links associates the term 'Engineering' and 'web' to interfacebus.com. There are two ways to go here; always associate a single term with your site/page, or associate the page with a different term each time [with in a topic].
Meta-Tags ~ Back in 12/14/07 Google Sitemaps indicated that 208 pages on the site had Meta Tags that were considered short [need more descriptive text]. The current report only indicates 70 pages with a short meta description. It may be less than that, because you can't determine the date of the report or which pages were last checked.

Web Site Optimization vs Bandwidth

I up-loaded the xml sitemap for interfacebus up to my site on Google Pages. The xml site map is used by search engines to determine the page addresses of all pages located on a web site. Search engines find your site map via a command within your robots.txt file. Up until now the robots.txt file indicated the site map was out on my server. The sitemap started on my server because Google Sitemaps does not give you the option of having the map in another location.

However, all other search engines find the sitemap via the robots text file. So Google will still check the site map, and hit my bandwidth, but now all the other search engines will go out to Google pages to access the sitemap.

The xml site map is 284k bytes in size and was viewed [down-loaded] 43 times last month. That's over 12 MBytes of server bandwidth. Yes I'm still trying to reduce bandwidth; currently running at 58.35kB/visit.

The html version of the sitemap [used by people] has been viewed 631 times this year. At the bottom of the sitemap is a list of the html pages located on 'Google Pages'; however the xml file will not be listed. The html viewable sitemap is also out on the Google server, saving server bandwidth..

The attached graphic shows the search trend for the term "miniPCI". I checked after my Analytics report indicated only three hits to the miniPCI page on the site [that's 3 hits for the year]. However it looks like I viewed data for a 404 page. The active MiniPCI pages has seen 3,377 page views. The page covering the MiniPCI 100-pin Signal Assignments page has received 333 page views, while the page covering the 124-pin MiniPCI card has received 3,497 page views.

Search engine optimization

Search engine optimization [SEO] is the process of configuring or setting up your web site so that it is search engine friendly. It really accounts for much more than that by insuring that a web page ranks high in a search result via key word placement. "Usually, the earlier a site is presented in the search results or the higher it "ranks", the more searchers will visit that site." I practice SEO; I make sure the title or page address relates to the topic, insure the first paragraph describes the topic and so on.

Any way, I just ran the program Xenu to validate the links on the site. The robot found only four bad links, and a ton of redirects. I don't think many of these companies practice SEO. They seen to change web names, redirect to a new address for awhile and than go back to the old address.... I really don't get it. I don't even list a few companies any longer because I continue to see dead pages as they keep changing their page addresses...

I should be up-loading a new sitemap tomorrow once I get a chance to delete some of the redundant listings in the report. I also ran Gsitecrawler and generated an XML sitemap for the search engines, Google is still checking it.

On a side note, the web site seems to be going well. Download speed is being increased by deleting HTML comments, or breaking up large pages into smaller pages ~ and making the pages more focused on one particular topic. Adding a search bar to all the pages, so the site may be searched from any page. Changing the search bar so that it defaults to a website search and not the web [some pages had search bars that searched the web, while some defaulted to searching the site which could be a little confusing. I also continue to add more pages, better than 80 pages in the last four months. Google now indicates 1606 URLs submitted.

Site / Server bandwidth remains at 57kB/visits, but I think the savings from reducing the html comments is being off-set by adding the search bar to the hundred's of pages that did not have one.

Not really web site related, but I've been trading out the older google referral links in favor of their new referral code. The newest code selects one of a hundred ads, while their old code would only run google related products ~ Adwords, firefox and so on. Plus the html code is smaller in most cases as one referral link may replace three other older google referrals.

Jan 2008 was the best month ever in terms of visits, and from the site history this month should be even better. March is higher than Jan, while Feb is lower January. New content is always being added, and html coding mistakes are fixed as they are found. ~ All is well

Optimizing Page Code

Following yesterdays blog about reducing monthly bandwidth by 259M Bytes by implementing the new adsense code, I've added the new code to yet another 9 pages. These would be the next nine best performing pages after the first four or so listed yesterday. The pages include: PCIe, Wire Color Codes, IDE description, SATA description, PCI Bus, Voltage Thresholds, Interface Buses, and Firewire.

There was one other page that was updated based on how it was placing, but because it was an old page address I do not list it here.

Although these pages do get less pages views then the previous 4 that were updated, when combined they get many more. So I expect better than a 259M Byte reduction per month in bandwidth. The addition of the two up-dates should produce a reduction in server bandwidth of better than 600MB/ month. Normal Bandwidth per day is around 460 MB, so the monthly bandwidth was reduced by more than one day.

This is yesterdays blog page;
http://interfacebus.blogspot.com/2008/01/page-optimization.html

Physical Page sizes

I can't remember if I've blogged about this or not, but I really regret updating my pages to XML. It increased the size of all my pages, increasing the download time and really hitting my bandwidth.

The so-called 'strict' coding approach is "BS", all those spaces that were added. On many pages the down-load time is doubled if your on a modem. Who came up with this idea?

Any way, the page still indicates it complies:
DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd

But, as I edit a page, I'm deleting some of those spaces ~ 1 byte per space. Maybe save a few 100 bytes a page [not removing all spaces, just a certain type]. What 130,000 visits x 2 page views x 300 bytes = 78,000,000 [78MB/month].

No wonder the internet is slow ~ it was conceived to be so

Web Site Status

The server numbers are in for November, not to bad considering the holiday. There were 202,078 visits, viewing 337,154 pages. The attached chart shows unique visitors [143,950 for Nov.]. Server Bandwidth was 12.71GB.

A unique visitor is counted only once regardless of how many times they may hit the web site during the month.

The posting on Declining PC Interface Buses provides one reason why the site seems to have plateaued ~ as a large number of PC buses are no longer being used.

Web Site Maintenance

The server for interfacebus.com provides two statistics generators or counters; one is AWSTATS, and the other is Webalizer.

Webalizer has a chart that shows the top 30 files being downloaded - or taking up the most bandwidth. A few weeks ago I moved a jpeg off this server to another server I use [server bandwidth post], saving 700MB/month. That particular pic no longer shows up in the top 30 list, reducing the strain on my server.

Now I'm looking at some of the other files in that same chart. I optimized a few css files and few html files. The css files are relatively small, but get download regardless of the page being displayed. One html page in particular for example: AWG Wire Chart was 42,116 bytes. Now that page has been optimized down to 27,388 bytes ~ by removing non-text related spaces in the HTML code. That's a saving of 14,728 bytes per download. Not including web spiders, that page was viewed 13,211 times over the last 30 days ~ That's a reduction of 194,561,608 bytes!

The CanBus page was reduced 5k bytes for a savings of 49,385,000 bytes per month!
The DVI Bus page was reduced by 4k for a reduction of 311,520,000 bytes per month!
The USB Interface was reduced by 3k for a reduction of 229,560,000 bytes per month!
Final savings in server bandwidth = 785,026,608 bytes per months.
Add that to the jpeg moving off the server and you end up with a reduction of bytes served by 1.5GB of data.... Plus faster down loads.

I assume the extra spaces were brought in when the pages was upgraded to XHTML. Anyway, next month I should see a new group of pages in the top 30 list from the page counter.