Cubic

How to calculate Password Strength (Part II)...

2012-01-21T18:57:00.003-05:00

Previously, on Cubic: The main character introduced a broad analysis of a new algorithm for calculating the entropy of passwords so that a threshold may be applied and weak passwords rejected. Will our hero's new algorithm pass more rigorous testing or will his arch nemesis Statistics Boy defeat it? Let's find out!

Since my last publication, I've been busy doing some other things. But this week I got back to working with this algorithm to see how good it actually is. My primary goals with my tests were to figure out how well it performs against real-world data and to determine a baseline entropy threshold for the algorithm that rejects most bad passwords. And what better real-world data is there than to use databases of passwords that were stolen from hacked websites?

I ended up testing against two types of information. The first type were hacking dictionaries. These are specially formulated files designed to defeat commonly selected weak passwords. The latter type were actual password databases that someone else extracted from various sources and someone else published to the Internet.

My overall analysis so far shows that the algorithm works very well and eliminates most bad passwords at the 18 bits of entropy level. This is the relative conclusion I've come to from analyzing millions of passwords and looking at the resulting output. It also surprised me by doing remarkably well on non-English passwords but I only had one database to work with for those.

Of course, you probably now want to see the relative levels of strength. Here is example output for the top 10.4 million passwords from the massive 32.6 million password RockYou database leak:

    8 => 2531803,
    9 => 2501749,
    10 => 1561832,
    11 => 1548202,
    12 => 918072,
    13 => 825913,
    14 => 315297,
    15 => 296425,
    16 => 113774,
    17 => 102698,
    18 => 35126,
    19 => 25454,
    20 => 12288,
    21 => 4274,
    22 => 4183,
    23 => 1870,
    24 => 958,
    25 => 796,
    26 => 366,
    27 => 227,
    28 => 227,
    29 => 183,
    30 => 138,
    31 => 138,
    32 => 138,
    33 => 138,
    34 => 138,
    35 => 138,
    36 => 138,
    37 => 138,
    38 => 85,
    39 => 85,
    40 => 85,
    41 => 85,
    42 => 85,
    43 => 85,
    44 => 85,
    45 => 85,
    46 => 85,
    47 => 85,
    48 => 85,
    49 => 85,
    50 => 85,

How should you read this? Out of the top 10.4 million passwords, only 35,126 (0.3%) would have made it passed the algorithm at the 18 bits of entropy level. My gut instinct and preliminary testing before I started seriously testing showed 18 bits to be the minimum acceptable level for this algorithm. I'm leaning toward the following rules:

18 bits of entropy = minimum for ANY website.
25 bits of entropy = minimum for a general purpose web service used relatively widely (e.g. Hotmail).
30 bits of entropy = minimum for a web service with business critical applications (e.g. SAAS).
40 bits of entropy = minimum for a bank or other financial service.

I was actually already leaning toward this as my recommended "rules of thumb" before running these tests. These more serious statistical tests merely validate my gut instinct and preliminary testing.

The 85 passwords exceeding the minimum for the last rule are actually an outlier in the data and can be counted as just one password. So out of the 10.4 million passwords that were analyzed, only 712 (796 - 84 = 712) were actually good enough to be used for RockYou at the 25 bits of entropy level. That is, the minimum at where they should have been running their service. What this tells me is that my algorithm rejects over 99% of all bad passwords from the start. Which is actually better than I was expecting. The other data sets show similar results.

At this point, I expect some developers are chomping at the bit to see this algorithm. The hard numbers above prove I've got something legit but you are still going to have to wait just a little longer...

What is going to happen? Will developers everywhere finally gain access to a seemingly miraculous set of functions? Will we be teased forever? Join us next time when the adventures of Cubic continue!

"Take that, Statistics Boy!"
"Oh noes! I've been defeated!"

The Ultimate Chair (Partial Resolution)

2012-01-21T12:09:00.001-05:00

A while back, I wrote a series of posts of creating a chair that would allow me to sit outside and soak up some rays. Programmers are white and nerdy because they sit inside. There are several benefits you can get from going outside:

Fresh air
The sounds of nature
Sunlight
Knowing what time of day it is
Not looking like a pasty-white programmer/office worker

Anyway, this post was an itch I've been meaning to scratch for a while to provide some closure and someone finally commented on it, so here goes...

I used the chair for a few months. (Feel free to take that sentence out of context.) And it sort of worked. It got me outside but I had several problems that I could never resolve to my satisfaction:

1) The chair itself wasn't comfortable to sit in for an extended period of time. I hate anything that provides so-called "back support". As an intelligent alien lifeform on Earth (that's a joke, BTW), back "support" is more like back "torture" to me.

2) Glare from the sun made it impossible to sit outside in direct sunlight. I know you aren't supposed to sit in the sun for long periods of time, but 15 to 30 minutes gives you the Vitamin D you need to stay healthy. But I wasn't able to read the laptop screen due to glare. I researched the issue back then and came up empty-handed but walked away from the experiment around the time e-ink hit the market and decided to watch from the sidelines as e-ink displays might supposedly breathe new life into this project. So far, they haven't.

3) It took a while to set up and tear down. Whereas my computer inside was all ready to go. I couldn't leave it outside because of possible weather issues. Nobody likes waking up to soggy chairs and wet digital equipment. That, and electronics have this habit of not working when wet. Go figure.

The digital egg timer eventually died and some of the custom parts of the setup actually melted one summer. That latter aspect was an interesting mess to clean up: Goo everywhere.

Overall, it was worth the effort for a DIY project. The ability to get outside to write software is still in me. I may try again if e-ink ever gets good enough that laptop/tablet manufacturers start implementing them. I'll probably just use a regular outdoor table and chair though. Comfortable chairs, outdoor or otherwise, are hard to come by. I like chairs that I can lean back into and rest my head on when I just want to relax.

However, I still use one component extensively: The ultra-sturdy mousepad. That thing is built like a rock, works on any surface, and doesn't go anywhere. My only complaint is its thickness. 1/8" Plexiglass would probably be plenty and, if anyone wanted to be really cheap, a slice of cardboard or thin wood might also work. The only issue with alternate materials is getting enough grip to take place between the surfaces and the contact cement so that they stay put or finding a different glue to use. There is room for improvement here and I'd like to think that there is even a market for it.

Ran a few Google searches on e-ink again. Qualcomm's got something called 'Mirasol' that is prototype-ish looking and claims to solve the problems of e-ink - namely color and performance. Nearly impossible to find videos of it in action though and was just debuted at CES in product format (Kyobo reader), so it'll be another year before it gets to be solid enough tech. to be worth trying again.

So, as it stands, I'm still kind of waiting for the technology to improve a bit before trying again.

Cross-platform multilingual support in PHP for the lazy programmer inside you

2011-12-27T10:38:00.000-05:00

If you are like me, you dread adding multilingual support to applications. It isn't that I do not like the various, rich cultures of our world, but rather that it is such a pain in the neck to implement multilingual support into an existing application from a programming perspective.

One of the problems stems from the fact that nearly all programming languages are written in English and limited to the basic ASCII character set and targeted at English-speaking people. Sure, I know of one programming language in Polish and a few others in other spoken languages but non-English programming languages are few and far between. It really has nothing to do with America vs. whoever but more to do with settling on something we can use to get work done on a computer and English happens to fit nicely into a single byte (or less) and seems to be one of a handful of what I call "common trade languages" - that is, if you want to conduct business across international borders, it helps a lot to know some English. Regardless of the reason, programming languages are written in English and likely will continue to be for years to come.

Of course, for native English-speaking programmers, multilingual support is perceived as a difficulty - suddenly we have to think very differently in ways we aren't accustomed to thinking. And, if this is your first time writing a multilingual app, you might search around to see what the "standard" is. There isn't one and people are all over the place on what the "best" method is. If you write a lot of PHP like me, 'gettext' crops up BUT it has too many issues including not being thread-safe and isn't necessarily available on the target web host.

What most programmers are looking for is a good strategy. What I'm going to share with you is my "lazy man" approach that works on all PHP installations and with how I prefer to develop software. If you need working PHP code, you'll be able to find it in the next release of Admin Pack.

First, I implement the application how I normally would: In English with all strings inline. This doesn't work so well for some languages, such as C/C++ (not impossible though), but it works great for PHP. Also, be sure the application uses some form of Unicode. UTF-8 is pretty much universally supported.

Next, when the situation for multilingual support arises, I implement a set of functions that introduce a "language stack" and load in languages in a specific order. This is something of my own creation that allows progressive fallback to my built-in English strings. English is at the very bottom of the stack at the "system" layer, a "default" language is at the next layer, and then a "user-specific" language is at the top. Now, let's say a function called TranslateStr() exists. TranslateStr() first checks to see if a string matches in the "user-specific" language mapping. If it doesn't exist, it checks the "default" language mapping. If it still doesn't exist, it falls back to the English string. Of course, this approach can create some "interesting" results where Chinese, French, and English all are displayed in the same user interface, but that's a translation issue, not a programming issue. As a lazy programmer, you shouldn't care. The key here is that something gets displayed even if it is in the wrong language. A lot of multilingual systems don't have progressive fallback despite being really easy to implement. The TranslateStr() function is then applied liberally throughout the application wherever English strings appear. The function can be equipped with a "this string doesn't have a translation at all" notification method so that translators can be made aware of strings that need translation work to be done.

Full static strings are easy one-to-one translations. Dynamically-constructed strings are much harder to translate. In PHP, functions exist like sprintf() which allow a string to be built differently based on the format specifiers in the first argument (e.g. '%1$s') - the lazy English strings only need '%s' though. This sort of functionality makes some string translations easier. Multilingual support gets messy when you discover that currency, numbers, and dates/times are displayed differently in different countries and that some languages are read in different directions. If you want to be lazy, just do your best to allow a translator to create a "mostly correct" translation. The human brain will adapt to and implicitly cover over most mistakes with only mild irritation. It helps a lot if the entire application is designed for "possible future multilingual support" so that it only takes a day or two to deploy a complete multilingual solution.

At this point, you are probably wondering how to create the actual translations themselves. If you are building a PHP-based web application, multilingual support is done lazily with IANA language codes (e.g. 'en-us') - which can be used with HTTP_ACCEPT_LANGUAGE parsing - and storing the translations into PHP files. But you don't want translators messing around with raw PHP files - trust me, they don't understand software development and you'll spend more time fixing the problems they create than anything else. A decent approach is to capture the output of var_export() using ob_start(), ob_get_contents(), and ob_end_clean() and create a web-based editor where they can manage the translation files that way. Be sure to protect the translation editor - the last thing you need is the German translation of your web application spouting stuff about certain "enlargement medicine" to your users because some script kiddie found your insecure translation editor. I'll probably figure out how to make one of these for general-purpose use, so you can sit back and wait for it (remember - when it comes to multilingual stuff, you want to be as lazy as possible).

Speaking of security, multilingual support has been the source of countless security headaches and several major security exploits have been the result of not thinking through possibilities. Make doubly-sure you don't introduce exploitable code as you deploy multilingual support into your application. The lazy approach allows you to carefully inspect the application as you go along, so you might end up fixing other vulnerabilities too.

Some programmers may observe a slight decrease in performance and increased memory usage when the multilingual support routines are added. These are unfortunate side-effects of multilingual support. But keep in mind that multilingual support is sometimes a necessary evil to keep users happy. You'll be able to get away with single-language-only applications more often than not.

And that is pretty much it. Enjoy being a lazy programmer!

How to find "useless" MySQL indexes...

2011-12-11T00:48:00.001-05:00

I was looking for some information on some high-performance MySQL questions lurking around in the back of my mind and found this very useful slideshow:

How to Kill Mysql Performance

On slide 40, there is a fairly complex and nearly unreadable (without going full screen) MySQL query that finds "useless" MySQL indexes by analyzing their cardinality. Since it is not able to be copy-and-pasted, I figure I'll save someone the trouble. It has been slightly modified for average data sets and to fix a case-sensitive bug:

SELECT t.TABLE_SCHEMA, t.TABLE_NAME, s.INDEX_NAME, s.COLUMN_NAME, s.SEQ_IN_INDEX, (SELECT MAX(SEQ_IN_INDEX) FROM information_schema.STATISTICS AS s2 WHERE s.TABLE_SCHEMA = s2.TABLE_SCHEMA AND s.TABLE_NAME = s2.TABLE_NAME AND s.INDEX_NAME = s2.INDEX_NAME) AS `COLS_IN_INDEX`, s.CARDINALITY, t.TABLE_ROWS AS `ROWS`, ROUND(((s.CARDINALITY / IFNULL(t.TABLE_ROWS, 0.01)) * 100), 2) AS `SEL %`

FROM information_schema.STATISTICS AS s INNER JOIN information_schema.TABLES AS t ON (s.TABLE_SCHEMA = t.TABLE_SCHEMA AND s.TABLE_NAME = t.TABLE_NAME)

WHERE t.TABLE_SCHEMA <> 'mysql' AND t.TABLE_ROWS > 100 AND s.CARDINALITY IS NOT NULL AND (s.CARDINALITY / IFNULL(t.TABLE_ROWS, 0.01)) < 1.00

ORDER BY `SEL %`, TABLE_SCHEMA, TABLE_NAME

LIMIT 25;

The rest of the slideshow is pretty good too.

The second to last slide on using 'auto_increment' is a key performance improvement that I've always suspected exists but I have yet to run into a corporate drone, I mean, DBA who agrees with me. I've always held that integer lookups are many, many times faster than doing multiple string-based lookups when joining multiple tables together while the DBAs I've run into won't budge from their silly little "3rd normal form" to inject a "redundant" auto-increment field (aka Surrogate Primary Key). The slideshow is straight from an expert on MySQL performance who worked at/on MySQL AB - so take that DBAs!

Then again, I don't know many DBAs.

How to calculate Password Strength...

2011-11-03T02:29:00.000-04:00

When I visit websites that want me to create an account before doing something, I typically enter in bogus information and occasionally I see a "password meter" that determines that I've entered a "weak" password. At least it is considered "weak" by some systems and "average/strong" in others. Being the curious sort of person, I've been trying to come up with a good, consistent strategy for calculating password strength and then something useful to do with it. I assume most developers only want to write password strength code one time, do something useful with it, and then move onto the next task.

What constitutes a strong password? An excellent question and something the industry seems to have difficulties figuring out at the moment. NIST, the National Institute of Standards and Technology, has a few words to say on the topic. Basically, password strength boils down to the number of bits of entropy that a password has.

So the next question is: How does one calculate the number of bits of entropy of a password? NIST has proposed the following rules:

The first byte counts as 4 bits.
The next 7 bytes count as 2 bits each.
The next 12 bytes count as 1.5 bits each.
Anything beyond that counts as 1 bit each.
Mixed case + non-alphanumeric = up to 6 extra bits.

After thinking about this for a while and crunching some numbers, the NIST proposal seemed decent enough. Plus, implementing the algorithm in software is easy. Unlike a lot of programmers, I don't take algorithms like this at face value and just implement them blindly. I like to test their claims first and sometimes introduce my own ideas.

During my research phase, I also ran across this xkcd web comic:

While I don't generally care for xkcd comics, this one actually made me think. Is "correcthorsebatterystaple" actually 44 bits of entropy? I immediately said to myself "no" but, being a sucker for self-inflicted punishment, I went to check.

Modern security is best done by not "handing out" bits of entropy. Basically, you only want to say that something has "at most 'x' bits of entropy" if that something passes a number of tests.

My initial calculations of "correcthorsebatterystaple" against the NIST rules result in a maximum of 41 bits of entropy. So, xkcd handed out three undeserved bits. Three bits doesn't sound like a lot, but it is the difference between 550 years and 69 years. Also, this is a phrase using words from the dictionary and the letters of the phrase have a lot of repetition, so I modified the NIST algorithm to only count a multiplier of 75% of each repeat character (e.g. the first 'r' counts for 100% of the bits, the next 'r' counts only for 75% of the bits, etc). This drops the number of bits of entropy to 34.2. My own algorithm runs some more tests but the result still comes out to 34 bits. This is 10 bits fewer than the xkcd comic's claim for that strategy to selecting a password. Or, more simply put, roughly half a year at 1000 guesses/sec. Not 550 years.

For the "Tr0ub4dor&3" part of the comic, my same base algorithm gives me 22.1 bits, not 28 bits. Or roughly 1.2 hours at 1000 guesses/sec. However, the extra tests in my algorithm actually calculate it as having about 14 bits of entropy.

Of course, the actual time it takes to guess the password is going to be somewhere between my theoretical time and xkcd's theoretical time. So, while on a surface level, xkcd seemed way off, the approach to selecting a good password with four random words isn't actually half bad and is a considerable improvement over today's passwords that users select. And we are all for anything that improves the passwords that users select...right?

If the computer generates dictionary-based passwords for the user, then that will eliminate the human equation. A human, following xkcd's rules, would likely select words of things that they can see, hear, and touch in their immediate vicinity - making it easier for a hacker to build a small attack dictionary. A quick glance around me and I get "computerkeyboardvideogamenetflixmoviewikipedia" - a great password from the algorithm's perspective except for the fact that I'm sitting at a computer with a keyboard, I love playing video games, I have a Netflix DVD movie in front of me, and I visited Wikipedia recently. So, even though I'm sure it has terrific entropy, it is a terrible password. Hence the need for a computer to generate dictionary-based passwords.

One thing I've noted with so-called "password strength meters" is that they are generally useless. The server-side doesn't enforce password strength requirements. Part of the problem is that no one seems to have come up with a solid algorithm (other than NIST) and no one has actually said, "here is how you should use this password strength meter..." This combination is useless to the average server-side programmer, so the only thing I've seen done is display the meter to the user and hope they make a strong password. Ha! As if that will ever happen.

What should actually happen: Calculate the number of bits of entropy in a password using a good algorithm and then apply a minimum threshold. (And get rid of the silly password meter.) If the password exceeds the threshold, then the password is strong enough, otherwise the user has to try again.

This approach allows a web forum owner to choose a minimum threshold of 18 bits of entropy and an online bank to choose a minimum threshold of 40 bits of entropy (or more). Each industry requires different password strengths. A password strength meter, if you really need one, could then become useful for prevalidation against the threshold. After all, the user only cares to know the answer to the question, "Will my password be strong enough so I won't have to resubmit this form?"

I'm still nitpicking at my algorithm, which is one of the reasons why it isn't being published (yet). So, don't get your knickers in a twist over my "hand-waving" claiming to have a solution and not playing nice in the kiddie pool and sharing said solution. The algorithm will be released in due time.

Another example: Google recently published an online safety guide on their homepage. This guide contains a YouTube video and other similar textual instructions on creating so-called strong passwords. The YouTube video shows "Garden1ngF@n" as a "strong password". For that password, my base algorithm gives 22.8 bits of entropy, but the extra tests portion of my algorithm calculate it as having around 16 bits of entropy. Yikes. Seems really broken to me.

So it looks like I'm onto something here. My algorithm is blasting apart even Google's recommendations for strong passwords. Woot!

Some thoughts about programming for the "Mobile Web"...

2011-08-20T11:41:00.002-04:00

I recently implemented a new feature called Cache Profiles into Barebones CMS. I'm a bit stuck on the documentation because I've had to stop and figure out how I'm going to create a mobile-friendly version of the website. I eat my own dog food after all. Om nom nom.

The concept of a "mobile-friendly website" is foreign to some people. A lot of people think, "Our main website displays the exact same on the desktop, iOS, Android, etc. and therefore it is mobile!" That is NOT a mobile-friendly website. If a user has to use pinch-to-zoom or scroll horizontally at all to read the content, you have a desktop-only website. While desktop websites do display on most modern mobile devices, they are NOT mobile-friendly and users will hate you.

Actually, that is a pretty good definition of a mobile-friendly website:

"A mobile-friendly website is one that displays on small screens such as smartphones, scrolls only in the vertical direction, and is clearly readable without requiring pinch-to-zoom or similar features."

Anyway, back to the problem at hand. I have made mobile-friendly websites before, so this is old-hat for me. My problem is coming up with a strategy that is universal and makes web developer's lives simpler. With the new "tablet" devices running "mobile" OSes, it is no longer a matter of making just two designs for a single website. We've suddenly got a lot of web developers stressing out about designing for desktop, phone, AND tablet. Which is causing some developers to ask some important questions, "What's next? How many different designs will I have to make to satisfy all these newfangled devices?"

Which brings me back to Barebones CMS. The work I've done thus far on mobile web development involves, for all intents and purposes, user-agent detection. However, if I stuff user-agent detection into the caching subsystem of Barebones CMS, the performance of the cache will eventually suffer. User-agent detection has always been and will always be a very bad idea. There is a project called WURFL, which detects any mobile handset and returns the feature set it has, which is pretty much as extreme as you can get. The download is, unfortunately, several megabytes and it isn't exactly Speedy Gonzales. In the last month alone, the download size of WURFL increased by 900KB. And that is compressed data. This situation is only going to get worse. There has to be a better way.

My initial thoughts are to detect if a specific cookie is present on the server. If not, serve up an alternate page that runs a series of tests, sets a couple cookies, and then redirects the user back to the content. The test page would mostly be Javascript that would run three primary tests: Determine actual browser screen size, connection speed, and "touch" capability. Maybe detect Flash support or whatever other anonymous features might be useful to know from the server's perspective so the correct content gets served up. If Javascript is not available, 'noscript' tags could load an image that sets the cookie - maybe fallback to user-agent detection - or just tell the user to enable Javascript instead of being one of those few people that disable it. The 'noscript' response really depends on how much the site depends on Javascript. If cookies are not available, have the page itself meta-refresh to an alternate URL after 'x' seconds pass. The page should have a link to do the same thing. The important thing here is to not get stuck in an infinite loop. Attempt to detect - on failure, display the default layout and content (usually the desktop version).

The basic idea is to attempt to determine which layout to display regardless of device. Let's say I'm on a desktop but on a super-slow dialup connection (yes, those still exist). Well, instead of being a jerk and serving up the painfully slow desktop version, how about serving up the lighter-weight mobile edition instead? With user-agent detection alone, you can't do it, but with my basic idea, you theoretically could. This is why using user-agent detection just won't work.

Edit: After thinking about it some more, things like Googlebot and web scrapers would make my idea a nightmare to implement. Googlebot would get SO confused. Ack. Maybe serve up the default layout as-is, add the Javascript "solution" into the header of the default layout, and serve up the mobile version to Javascript enabled devices by reloading the page. Server-side user-agent detection is soooo hacky but maybe use something there too to avoid loading content twice.

Edit #2: And after thinking about this a LOT more, I've got what I'm going to do worked out. First, all browsers go to the main content. If Javascript and cookies are enabled, set some global that I can use later. Also check to see if a specific cookie is set. If it isn't, determine if the user's screen width is narrower than some set limit. If it is smaller, set the cookie to mobile and reload the page. Offer all users with Javascript and cookies enabled the option to switch between mobile and regular views. The server checks for the specific cookie and then, if it exists, loads the appropriate cache profile. Those with Javascript and/or cookies disabled get served up the default cache profile without the option to switch to the mobile site. This approach makes it possible to serve content to mobile devices and stay relatively future-proof (i.e. without browser sniffing). The downside is that there are a couple of extra requests to the server for the initial mobile connection, but the user won't likely notice. I personally like this approach because it will allow me to test it easily with a plain-ol' web browser without having to fake a user-agent string.

Alright. I'm done with my thoughts.

Multiselect list boxes are not intuitive. Do not use.

2011-06-04T14:09:00.000-04:00

I've come to the inescapable conclusion that multiselect list boxes are not an intuitive UI element and are actually quite frustrating to use for most users. A set of two-state checkboxes is much more intuitive. In fact, I've found that anywhere that multiselect list boxes can be used, two-state checkboxes can easily replace them for a much more intuitive solution.

Here's an example of a multiselect list box:

Set the average user in front of a computer screen and ask him to deselect all selected items. The secret is to hold the 'Ctrl' key while clicking, but how often do people actually execute a 'Ctrl + click'? Not very often. Most users, in fact, aren't even aware of special key + click combos. These things are put into applications for power users AFTER the basic functionality is implemented. All functionality of a UI element should be available through a left click or a right click. This is User Interface Design 101. The multiselect list box violates this principle.

As a result, I constantly hear "How do I deselect all items...?" and "How do I select multiple items...?" in regards to AdminPack implementations and multiselect list boxes. For this reason, my next release of AdminPack will do away with multiselect list boxes and just output checkboxes. The net result will be the same and headaches will be reduced for both the end-user and myself. I also was just generally annoyed with the visual look as well. This particular UI element is just ugly with a vertical scrollbar that tends to show up more often than not.

I was kind of curious to see if anyone else had also made a similar decision about this UI element. So I started scouring Google and there is general consensus that this is a lousy UI element. However, using checkboxes can apparently add clutter if there are many options. I can see how that could be a problem but it is less irritating than a multiselect list box. During my searches, I ran into the jQuery UI Multiselect Widget which is a pretty impressive solution for the lazy since checkboxes can be rather annoying to implement. My only major gripe with the widget is that, while it is closed, it only displays "2 items selected" or something similar by default, so it is hard to see, at a glance, what exactly is selected. But that, like most jQuery solutions, can be configured (selectedList option). But, frankly, that's a rather small price to pay if someone needs to display 100 items. There's also a plugin for the widget that makes it easy to find the item you are looking for. That widget will probably find its way into AdminPack Extras.

Since some pretty good alternatives exist to plain multiselect list boxes, it becomes possible to simply never use them. And your users will thank you for it.

Truly free SSL certificates are here!

2011-02-25T09:51:00.002-05:00

I and many other people have been waiting for a decade for this, but truly free SSL certificates with a root certificate installed in every browser is finally available. It used to be that to get a signed cert, you had to shell out tons of money. That was and is a ripoff. SSL certs cost ISPs nothing to produce and are pure profit. Even the EV validated certs (green bar) are a huge ripoff - sure the setup fee might make some sense where they do real checking, but, after that, the renewal process is entirely automated. Some places want $400 per certificate per year. This is one of the hosting/reseller industry's best-kept secrets.

I've been keeping a very close eye on the free SSL certificate market for a while now. Every couple of months for the past decade, I've run a search query like "free SSL cert" and looked carefully at the results.

The first organization that popped up on my radar was CACert.org. It was exciting when I first saw this because it merely confirmed what I knew all along was that SSL certificates cost nothing to produce. The only problem facing CACert was web browser and OS integration. After a few years of waiting for them to do something - anything - it became apparent that they were collapsing in on themselves. They seem to have a strong following in Europe, but that's it. They don't seem to be interested any more in getting included in the root certificate store of every major browser and OS. Which is sad, because they seemed like they could potentially have pulled off something fantastic - a non-profit organization with the potential to even produce EV validated certs for free. They could have crushed the myth that SSL certs cost hundreds of dollars to produce.

SSL certificates are chains of certificates that trace to a root certificate. Every browser and OS has a list of root certificates that it can validate against. This is called the root certificate store.

The biggest hurdle is being included in the root certificate store of every major browser and OS. This allows the most popular SSL-enabled applications to trace the certificate back to a valid root. A SSL certificate also determines what the certificate was authorized for. Not all SSL certs are created equally. Some can be used for e-mail, some for web servers, some for code signing, etc.

Anyway, most people are usually just interested in setting up a web host with SSL support. A couple years ago, a new startup called StartSSL appeared on the scene claiming free SSL certificates. What they were doing was interesting but they lacked the usual browser and OS support. Then something happened. They started getting into the root certificate stores of browsers and OSes. About two years ago, they were in every major browser and OS except Opera. The list was impressive, but, without Opera support, it didn't matter for web developers.

Sometime in the past couple of months, it looks like Opera finally got up off their butts and approved them. Check out the list:

http://www.startssl.com/?app=40

As an interesting consequence of this, every domain/reseller provider out there has had to lower their SSL certificate prices to more reasonable levels to compete with this new threat. Which, again, only confirms that those high-priced SSL cert products are nothing but pure profit.

So there you have it. Free SSL certificates finally exist. Over the next couple of years, we will likely see other vendors doing the same thing to remain competitive. Some products will likely continue to cost money but people are going to learn to shop for the best deal. This will drive prices down everywhere to reasonable levels.

Google Instant bugs - and no good way to report them...

2011-01-22T10:52:00.002-05:00

Ever since "Google Instant" was released by Google, the search box occasionally just disappears in Firefox:

This seems to happen right around the moment when Google Instant directs Firefox to a new URL and I happen to be pressing the back button on my Logitech G500 Mouse.

Which is, by the way, a very nice mouse for gaming and general-purpose use for us right-handed folks. Left-handed folks are, unfortunately, so neglected by computer equipment manufacturers.

Anyway, I'm not sure why this bug happens.

There is also another annoying bug with Google Instant that happens when I'm typing quickly. It switches from the main page to the Google Instant page. The search box is there with my search but complete search results refuse to show up and clicking the search button does nothing. Starting over and running the exact same search causes search results to show up.

Google is one of those companies that doesn't publish their e-mail addresses anywhere. I do have a few direct e-mail contacts but none of them are exactly appropriate for reporting bugs with their core product (search).

Two ways to make your computer faster that no one thinks of.

2011-01-15T11:13:00.001-05:00

I've been using Windows for a really long time. As have many other geeks/nerds. I've seen my share of "Registry Cleaners", speedup tools, and tips from other people. All these things don't work. They really don't. You might get a temporary "boost" in certain areas, but a month later you'll be experiencing enough seemingly unrelated system problems that you'll end up reinstalling Windows.

People always tell me their computers are slow and want to know how to speed it up. Today, I'm going to show you two ways to truly speed up a Windows-based computer and keep it lightning fast.

First, fire up that slow computer and launch Task Manager. Go to the "Performance" tab, and look at the amount of memory being used. If the amount of memory being used exceeds 80% of the amount of available physical RAM, that means that portions of the OS and other programs are being moved to what is known as "swap space". Swap space is where the hard drive is being used to temporarily store bits of other programs to the hard drive so the current program being run by the OS can execute. Since the OS switches "rapidly" between programs to give each one a chance to execute some code, it is likely that the OS will start falling behind very quickly and the entire computer grinds to a halt as each program starts using swap space.

The hard drive swap space is there as a buffer so that the OS doesn't crash. It isn't meant to be used for extended periods of time. In fact, always running on swap will lower the life expectancy of the section of the hard drive where the clusters for swap space are being stored, causing premature failure of the drive.

The solution to this problem is to eliminate running programs you don't need running. This is actually a lot harder to do than it sounds and reinstalling the OS is the sometimes the fastest way to accomplish that task. Google is your friend here but it isn't easy to decide what to remove and what to keep even after searching. I'd love to recommend uninstalling anti-virus software because it is all unnecessarily bloated software but, unfortunately, most people need protection from themselves.

The second thing that can drastically speed up a system is eliminating useless entries from the "System path". This hearkens back to the bad old DOS days from which Windows was born. The system path is buried on all Windows OSes as an "Advanced" feature. On my Windows 7 install, it is located in: Start, Control Panel, System, Advanced System Settings, Environment Variables, System Variables, "Path". The most basic system path on Windows 7 is:

%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem

When Windows looks for a file, it will check first in the current working directory and then every entry along the configured user and system paths before giving up. Every addition to the path increases lookup time exponentially. If you run any sort of monitoring program (e.g. SysInternals Process Monitor), it becomes apparent that Windows performs extraneous lookups quite frequently and fails to find anything more often than not. Fewer paths = faster system. However, be careful when removing paths as any application that depends on the path may suddenly stop functioning. Again, the only way to easily remove a path is to just reinstall Windows.

It is fairly common knowledge that a fresh Windows install is fast. People blame the Windows Registry for bloat, so someone came up with a "Registry Cleaner". These have incredible risk of damaging your system in unknown ways and most don't work as advertised anyway. I never use them and never have problems.

If you are contemplating reinstalling Windows, also consider installing what are known as "Portable Applications" or "Portable apps". Portable apps are designed to be run from a USB thumbdrive so you can take them anywhere but they can be "installed" locally too. More importantly, though, is that they don't affect a Windows installation in any way unless you connect it up yourself. A portable application doesn't touch the Windows Registry or put files on the system. You could have hundreds of portable apps "installed" and the performance of Windows will remain the same. Plus, you have the option of taking your applications with you wherever you go. In addition, by using portable applications, every portable application used is one less application that has to be installed when Windows is reinstalled in the future and the data also travels easily between computers. It is a win-win scenario to use portable apps.

Dedicated Hosting just got affordable...

2011-01-14T10:38:00.000-05:00

It used to be, in order to have your own dedicated host, you had to plop down hundreds of dollars a month. Today, I noticed that 1and1 has a fantastic deal on dedicated hosting for $60/month on a permanent basis - this is not one of those 6 month deals. Cloud hosting and virtual hosting are, in my opinion, no longer competitive with this deal.

I use dedicated hosting myself and absolutely love it. Most people opt for shared hosting, which introduces them to a world of hurt. If you have multiple websites, shared hosting gets expensive really fast. Plus shared hosting has significant downsides:

- No control of the server. You are at the mercy of the hosting provider and their settings.
- Web hosts cram hundreds, possibly thousands of websites onto a single box. Your website is not alone.
- Greater risk at getting hacked because someone else on the same box got hacked.
- You risk getting kicked off the hosting provider due to "too much CPU, RAM, or other resource usage".

A dedicated host is great because you control the server, what goes on the server, and can chug CPU, RAM, and hard drive all you want. It is your box.

The key aspect of ANY server is bandwidth. Bandwidth is usually most web host's trigger mechanism for "there is a problem" and you get booted. 1and1 offers the most bandwidth of all the various dedicated hosts out there (1TB per month) before that trigger mechanism - although, instead of getting the boot, they just charge per GB past that point. 2GB RAM is a bit skimpy though to run, say, the resource hog known as WordPress. However, that's plenty of RAM for Barebones CMS.

How to get unblocked from Hotmail/Live

2011-01-06T10:11:00.000-05:00

For the past couple of weeks, I've been trying to get my domains unblocked from Hotmail/Live. I set up a new domain for Barebones CMS (barebonescms.com) and put forums on the site and forgot to add a SPF record for the domain. A couple weeks ago, Hotmail/Live servers decided that e-mail from barebonescms.com was invalid and therefore refused delivery. Due to the holidays being crazy, I was unable to get around to dealing with the problem.

Then I discovered that Hotmail/Live was actually blocking ALL e-mail from my e-mail server regardless of domain. This sent me on a search to see how I could get unblocked. Of course, the first step was to fix my DNS records to add a SPF record. SPF-aware mail servers should really be assuming a default of:

v=spf1 mx -all

Or:

v=spf1 a mx -all

That way, most of us don't have to fiddle with silly things like this.

The next two steps to getting off the Hotmail/Live block list is to go here:

https://support.msn.com/

And request to join both the "SenderID" and "Junk Mail Reporting Partner" programs. Those are the minimum required to get off their block list. Of course, because of Internet delays revolving around e-mail and DNS, it can take 24+ hours before any changes are to be made to their servers and then another 48+ hours to take effect.

During this process, I ran into a couple nifty little tools that everyone who runs an e-mail server should be using:

https://www.senderscore.org/

Sender Score is a pretty accurate measure of how much e-mail volume a specific IP address is churning out. The score is calculated based on anonymous ISP data. Hotmail/Live appears to use this information to determine the reputation of a specific IP address, which is likely used in their spam filter to determine whether to accept the message, drop it in the spam folder, or simply reject the message altogether. The graph they display is really nice for determining trends for a specific IP but is otherwise meaningless.

The other nifty tool I ran across is Microsoft Postmaster Smart Network Data Services. This tool is a little more difficult to set up but you can see issues regarding the specific IP address associated with a mail server. For example, if someone reports an e-mail as spam or Hotmail/Live detects a virus coming into their servers from a specific IP, that information is made visible using that tool.

All of this takes time to complete. By the end of this week, though, my own situation should be cleared up. Hopefully this helps someone else. The moral of this story is: Don't forget to set up SPF records on new domain names.

KB976902 - "Black Hole" update

2010-10-27T01:42:00.015-04:00

Update February 27, 2011 - Windows 7 Service Pack 1 appeared in my update queue with the checkbox unchecked. Hmm. To update or not to update?

Update January 14, 2011 - This appeared again in my Windows Updates. I figure installing it is okay now that Microsoft appears to have their ducks in a row. After the first fiasco, they probably took their time to release it correctly. This is the precursor to installing Windows 7 SP1. From Microsoft's website: "Windows 7 SP1 Release-to-Manufacturing (RTM) will be available in the first half of calendar year 2011. When released, it will be made available as an integrated release." This updates the installer in Windows 7 so it can upgrade to SP1 later. SP1 isn't available yet.

What follows is the original post.

Today, a mysterious Windows Update was released to all Windows 7 users. I'm going to preface this by saying what some Microsoft MVPs are saying: DO NOT INSTALL!

The update says, "Install this update to enable future updates to install successfully on all editions of Windows 7 or Windows Server 2008 R2. This update may be required before selected future updates can be installed. After you install this item, it cannot be removed."

This update appears to only be known as the "Black Hole update". No one knows precisely what this does but it appears to be a precursor requirement to installing Windows 7 SP1 - why it can't included with SP1 itself is unknown. There is no knowledgebase article (HIGHLY UNUSUAL!) despite having a KB article number.

---

Update (Oct 27, 2010): Korablikovas (see comments) pointed out that this update is not localized. In my extensive experience, Microsoft localizes just about everything that they publish. Which makes this update even more of an oddity and quite disconcerting. No official word yet from Microsoft on this issue (although someone decided to mark a reply an answer in the "semi-official" Microsoft forum post). Still no KB article. It appears that this update creates a System Restore point prior to installation, so if you accidentally installed the update and want to roll back the installation, you can use System Restore to do so.

Update #2 (Oct 27, 2010): Reports of this update disappearing from the "hidden" updates queue are cropping up. I can confirm that this update has vanished from my own queue and can also confirm that it wasn't installed. Looks like Microsoft redacted this update from any computer that hadn't applied it yet. As I somewhat suspected as I learned more from various sources, this was possibly just an accidental release of a piece of the RC build of Service Pack 1 - some sort of precursor component upgrade related to Windows Installer/MSI stuff. There's some people freaking out over this being a WGA update - apologies if anyone was lead to believe that from this post. And then there are the idiots who believed the TechArp April Fool's joke. It does bother me a bit that Microsoft can remotely redact updates but I've got mixed feelings on this particular update regarding redaction. I'd rather have remote redaction than have it still in everyone's queue who hasn't installed it. The problem Microsoft has to deal with now is those people who installed the update prematurely to roll them all back to a consistent state with the rest of us. That's probably a more difficult problem and Microsoft will likely take the approach of leaving well-enough alone until Service Pack 1 officially releases. I do recommend that Microsoft change the language of this update to not be so mysterious/suspicious and to get an actual KB article for the update put together ASAP. However, the potential disaster this could have been seems to have been averted for the time-being.

Update #3 (Oct 29, 2010): There is now a KB Article that goes along with this update. I didn't actually come up with the name "Black Hole", I just saw other people using it to describe the update because when a search was made for a KB article of the same number on the support site, no search results occurred. That is, the link and search went nowhere - a black hole. And, since it was redacted from the queues of those who did NOT install it, we can assume that Microsoft made a mistake by releasing it early. Of course, there is no word yet on what Microsoft plans to do for those people who installed the update prematurely.

What follows is original content for historical purposes.

---

The best information I've seen about this update is here, which says:

"In preparation for supporting a new Windows image, this update is updating the servicing stack to 6.1.7601.17105 including Component Based Servicing, Component Management Infrastructure, Package Manager, and Windows Management Instrumentation. This is to support Windows 7 and Windows 2008 R2 SP1."

I ran a quick Google search for portions of that text and simply came up with that blog entry again. So I have no idea how that person got that information (perhaps an internal TechNet memo). It looks like something Microsoft might write - just generic enough to be basically meaningless except to the few people who are nerdy enough to know what it likely means. I recognize a few terms but it is rather vague on the details.

Before people go crazy, I should point out that someone on that page linked to an "article" that will scare people who don't pay careful attention. It took me a while to figure out that it was an April Fool's joke. Bloggers and other media sources need to be VERY careful when republishing on this one.

The description indicates that it is an update to the installer components and therefore could merely be a mistake on Microsoft's part for not publishing a KB article. However, it could be a lot more sinister than that as well. As we already know, Microsoft is evil. Google, of course, does no evil.

Due to the speculation and the weird message in Windows Update, hold off on installing it until we know for sure what it is that is being foisted off on us. If anyone from Microsoft reads this - you need to work on being more transparent about your updates and seriously improve your public relations image. The way this update is worded and the mysteriousness surrounding it makes it look like another WGA-style fiasco in the works - and trying to sneak it by hoping it goes unnoticed. Guess what? I noticed it right away. Your pants are down...again.

Microsoft is baffled as to why people prefer Google. Straighten up, fly right, and stop treating your customers like we're unintelligent and incapable of running Google searches. Keep in mind that I'm a software developer. Here - have a reminder of who you actually cater to:

Developers! Developers! Developers! Developers!
...
Deodorant! Deodorant! Deodorant! Deodorant!

Easy Invoice Numbers

2010-10-19T12:39:00.005-04:00

If you are building or using an e-commerce system and want to look all professional, you need invoice numbers. Invoice numbers are required, at the very least, for auditing purposes by most businesses. However, nothing says "not professional" quite like "dressing in a shirt that is flashier than a light pinstripe" or invoice numbers based solely on a MySQL 'auto_increment' field starting at '1' or, worse, a false arbitrary starting value.

Searching Google for an industry-standard practice of creating an invoice number turns up pathetic results. So this blog entry aims to correct this severe oversight of the Internet and bring it down to the level of the average programmer.

Businesses that are large enough have dedicated finance departments. These people like things to be EXTREMELY organized. If you are developing an application that is going to bring in money (e.g. an e-commerce solution), it needs to generate sequential orders that can be verified later should the organization be required to perform an audit. Audits are annoying. If you start numbering invoices/orders at 500, the auditor is going to want to know where invoices 1 through 499 are. So, you really should start numbering orders at '1'. However, from a customer's perspective, being the first person to order anything is viewed as rather "tacky" for the invoice. So, what should be done about this?

Invoice numbers should represent the order in some fashion that makes sense and makes looking up the order later on easy to do. It should also include some letters with the numbers. What follows is my own blend of parameters that creates a rather good-looking invoice number in most cases, makes it easy for programmers to extract the order number, AND meets the auditing restrictions imposed by most businesses:

CompanyAbbreviationYYMMDD-OrderNum

So, a sample invoice number could be:

CS101119-1

The example is the company "CubicleSoft" abbreviated to an acronym, the two-digit year, two-digit month, and two-digit day of when the invoice was created (November 19, 2010), a hyphen (-), and then the order number (Order 1). It could be the very first order ever in the system but the customer won't notice - they will likely just think it was the first order of that particular day. For programming purposes, it is easy to locate the hyphen, convert whatever comes after that to an integer, and then use the value to look up the order in the system. In fact, the only thing that really matters with the invoice is the number after the hyphen. The stuff before the hyphen can just be used for verification purposes.

The next issue is that finance departments require the ability to map an invoice number to the money. That is, they need to be able to exactly match the order to each and every penny. An e-commerce solution depends on a third-party to process the payment. This creates a disconnect between the order and the payment that must be resolved. The invoice number on the payment end of things must match the order entry system or auditors get unhappy. Therefore, be sure to send the payment processor the fancy invoice number so when someone later on goes to do a report, they can easily get that information, which makes that person's life easier and simpler.

In conclusion, I really like that video. :)

Forget flock() and System V Semaphores - use WebMutex instead

2010-10-11T09:57:00.005-04:00

While developing my latest PHP-based project, WebCron, I ran into an issue that has bothered me for a while - atomicity. An atomic operation is one where only one thread of one process is allowed to execute some piece of code. Actually, even under C/C++, I've been bothered by this issue.

Unlike Linux, Windows really has the most friendly approach to creating an environment where atomic operations may thrive. Named mutexes is one area where Windows really, truly shines above all the OSes out there. Try porting CreateMutex() to another OS and you'll inevitably have some real head-scratching sessions when you try to do a cross-process, named mutex. So-called 'mutexes' under *NIX OSes are usually 'pthread'-based, which are really more in line with Windows "critical sections" than "mutexes". A programmer coming from the Windows world is going to be utterly confused because they've been pampered by Microsoft and no one in the Linux community sees any benefit of adding REAL mutexes to the kernel.

The problem lies in the fact that the Windows OS itself manages the mutexes. When you want to lock a mutex, a jump into kernel mode is made and the kernel manages the locking process. If a process or thread goes away while holding onto a locked (signaled) mutex, the Windows kernel potentially eventually cleans up the mess left behind. And here is the key point: There is no equal process on any other OS and definitely nothing that is cross-platform (natively). Or at least no single approach that has been "standardized".

Let's look at the options available for creating something close to a named mutex. I'll be specifically focusing on what is available for PHP:

flock() - This function is the first recommendation by the PHP developers as their "cross-platform" "solution". If you read the documentation and the comments carefully, it quickly becomes apparent that it does NOT behave the same under all OSes, especially Windows. Here are the core problems: The non-blocking option is ignored (i.e. doesn't work) on Windows, the return values are not consistent across OSes, it has issues with multi-threaded web servers, and it requires a file to already exist (in advance) on the system to be used as the "lock file".

System V Semaphores - These functions aren't available for Windows, requires an integer instead of a name to identify the semaphore (seems hacky to me), and the functions are not compiled into PHP by default.

For WebCron, what I needed was a mutex object that provided a cross-platform, cross-process, cross-thread, non-blocking, expire-capable, multi-lock capable, named mutex. When developing a product that will be used on who knows what OS, it has to at least support "The Big 3": Linux, Windows, Mac. Clearly the above functions weren't going to cut it for my needs. I needed something else.

I scoured the PHP documentation for a while. I knew the key was that the OS must manage the creation of the mutex, not the application. I knew that if the application managed the mutex, there will be race condition issues and other nastiness involved.

Then I found the solution I was looking for: fopen().

Every OS manages the file system and all access to files. However, the 'w' and 'r', 'a', and 'c' options of PHP automatically create a file if it doesn't exist. And then I found...'x'. It creates a file if it doesn't exist and will return false if it does. The '@' prefix suppresses errors and warnings. Since the OS manages the creation of the file and since file system access is universal, I had found my cross-platform solution (i.e. silver bullet). And WebMutex was born.

The first hurdle was deciding how to handle the lock files. I ended up opting for using an absolute path and filename and then appending '.lock'. I call @fopen($filename . ".lock", "xb") and check the result. If it is false (i.e. some other process beat this process to it), then I don't have the lock, sleep for a random amount of time, and try again. Releasing the lock was a matter of deleting the file. If Lock() is called multiple times, I just increment an internal counter instead of touching the file system again.

My next problem was that PHP scripts can only run for 'x' seconds before PHP forces the script to exit. PHP doesn't run cleanup code when it forcefully stops execution. So I had to come up with a way to delete the lock file. I eventually settled on creating another file using the fopen() 'x' option with '.stale' appended to the base filename. That way there aren't multiple processes deleting the lock file - potentially introducing a race condition that allows multiple locks to be obtained. Of course, it becomes possible to end up with both '.stale' and '.lock' files. However, that should be rather rare and requires manual intervention anyway.

The solution I came up with to the second problem isn't without its own problems though. If the file system is shared and there are two processes running on multiple machines and the timestamps of those machines aren't perfectly in sync, the lock file might get deleted prematurely.

The downside to this whole approach is performance. A file-system based solution isn't going to be nearly as fast as a native kernel solution. However, named mutexes aren't speed-demons to begin with. I'm after something that works not something that performs spectacularly.

However, the end result is actually a lot simpler than I originally thought it would be. It meets all my needs and addresses my concerns. And, perhaps more interestingly, this solution is easily portable to other languages too. The only requirement is being able to open a file for creation and return a failure condition if it already exists.

The source code to WebMutex is under the same license as WebCron (MIT or LGPL, your choice). You'll have to download WebCron to get at the source code for WebMutex.

Fixing slow Apache on localhost under Windows 7

2010-07-14T09:50:00.007-04:00

A couple days ago, I documented my recent experience with my wireless network and how I got hacked. I briefly mentioned that I'm installing Windows 7 this time around.

Yesterday, I ran into an issue with 32-bit Apache 2.2.15 running on Windows 7 Ultimate 64-bit. This issue appears to only affect 64-bit Windows 7 and 64-bit Windows Vista web developers attempting to run 32-bit Apache on the system.

NOTE: I didn't test 64-bit Apache because that is experimental and 64-bit PHP is even more experimental. However, I doubt the results would be too different.

The issue is slow response times (anywhere from 1 to 3 seconds per request) when connecting to 'http://localhost/'. Connecting to 'http://127.0.0.1/' and 'http://[NetBIOScomputernamegoeshere]/' have fast response times. I've seen various fixes around that seem to boil down to these three:

- Disable the Windows Firewall.
- Disable IPv6 support.
- Edit the 'hosts' file.

It looks like there is a bug in IPv6 DNS mapping for localhost under Windows that causes the slowdown when used in conjunction with the Windows Firewall. If Microsoft employees are reading this - THIS IS A BUG...FIX IT! Apache doesn't seem to be the issue but it could be that 'localhost' via the firewall first tries an IPv6 address first, fails after a second or two, and then falls back to IPv4 at which point a connection to 32-bit Apache is established. Apache isn't the issue here - the Windows Firewall combined with IPv6 is. Anything 'localhost' only listening on IPv4 will have slow responses.

What worked for me was to use the IPv4 version of 'localhost' in my hosts file. The 'hosts' file is usually located in the 'C:\Windows\System32\drivers\etc' directory on most systems. Editing the 'hosts' file was a pain in the neck. You can't simply right click or double-click to open the file (BUG! ALSO NEEDS TO BE FIXED!). You first have to fire up Notepad or your favorite editor using "Run As Administrator" (right-click) and then hunt for the file using the "File -> Open" dialog. Windows prevents editing that file normally (a good thing) but then get in the way of actually editing it (a bad thing).

Anyway, under Windows 7, the following two lines are commented out:

# 127.0.0.1 localhost
# ::1 localhost

Uncomment the first line but leave the second commented:

127.0.0.1 localhost
# ::1 localhost

Save the file. Try loading pages in Apache again. Should be back to normal speed.

Why this works?

The 'hosts' file supersedes all DNS mappings. So this "fix" merely forces the IPv4 mapping to occur when 'localhost' is requested. Uncomment the second line and Apache will slow down again.

Disabling the Windows Firewall supposedly fixes the issue but I'm pretty sure no one wants to run without a firewall. Disabling IPv6 also apparently fixes the issue but IPv6 is supposedly the future - IPv4 is what the OS falls back on without IPv6. The 'hosts' file fix is really the most desirable until Microsoft can figure out why the Windows Firewall is broken.

Hopefully this helps someone else and helps provide insight into the actual problem.

My wireless network got hacked. Unremovable rootkit? New botnet tactic?

2010-07-12T22:13:00.007-04:00

I take security VERY seriously and violations of that security even more seriously. Hacking my personal networking infrastructure is near impossible. Or so I thought.

First, some background. I run...well, I used to run a wireless network access point. Yes, it is one of those consumer-grade, wireless network setups because I'm a cheapskate. The brand doesn't matter. Here's the critical bit of information: I ran the wireless access point with WPA-PSK (TKIP) using a completely random key of about 40 characters in length and a different SSID from the default plus MAC address filtering. Standard WPA-PSK cracking tools were theoretically never going to gain access to my personal network. I also had the benefit of being surrounded by quite a few open wireless networks. I figured those networks would be picked over mine. Plus, I don't usually turn on my laptops very often - so passive traffic sniffing would make it a nightmare to gain access. Basically, the worst possible set of conditions for anyone hacking my network - psychologically undesirable, physically undesirable, hard to crack, and I monitor my network and computers like a hawk.

I hereby declare my original article on setting up a "secure" wireless network as Dead On Arrival (DOA). It has become rather painfully clear to me that WPA-PSK is completely broken - somehow. Searching Google turns up nothing other than some basic articles utilizing mostly dictionary attacks. Someone, somewhere has figured out how to hack any secured WPA-PSK wireless access point. The only protocol left is WPA2-PSK. And I doubt it too will remain secure if it hasn't already been equally broken. Consider all wireless access points to be huge vulnerabilities in your network.

Thankfully I was sitting in front of my computer when it happened. Then again, I rarely step away from my computer when it is turned on. I noticed that I lost keyboard and application focus three times over the course of two minutes. While I was editing a Powerpoint presentation. I was curious after the first time, concerned the second time, and went to fire up Task Manager after the third time. Right before I fired up Task Manager, Internet Explorer started on its own. I don't run IE these days for various reasons - mostly because Firebug exists.

At that point, I knew I had been hacked. I glanced over my active applications I had open to memorize what was active and immediately shut down the computer. My data is critical to me and therefore must remain secure and out of the hands of would-be thieves. If the computer is off, the most the enemy would get is a few megabytes. However, the person seemed more interested in dumping additional software on my computer than seeking out my data. Probably to solidify their position on the box.

I happen to keep around a few Rescue and Live CDs for just-in-case scenarios like this. I ran the Rescue disks (F-Secure and BitDefender). As of July 2010, F-Secure has the edge in terms of ease-of-use. BitDefender seems to be lacking on the networking front - it couldn't find my Ethernet cards, so I couldn't get the latest definitions.

I also have a netbook with Ubuntu on it, which I fired up right away and started doing research while the rescue CDs did their thing to find malware on the system. I tried to figure out what I might have done in the previous 24 hours to allow for this to happen. I determined that it was impossible for it to have been me unless the MySQL developer's website was somehow compromised. Then I started thinking of the absurd...and my first thought was my wireless network. I immediately logged into the router, flew to the active wireless network connection list and, lo-and-behold, not five minutes before, someone had gotten onto my wireless network coincidentally around the time my Windows computer had been violated. I foolishly forgot to take a screenshot because my knee-jerk reaction was to shut down the wireless radio. A reasonably wise decision on some level. A screenshot would definitely help the credibility of this blog post. I did disconnect the Ethernet cables except the one to the Ubuntu netbook and turned the wireless radio back on for a couple minutes to see if I would get any takers but eventually decided the risk wasn't worth it and shut the radio back down for good. (I also needed to go to the bathroom at that point.) It is also important to note that I'm pretty sure they only had access to my network for a few minutes. I'd probably leave too and not return if I noticed a Linux box come online shortly after I deployed a rootkit and the wireless radio mysteriously vanish shortly after that.

The F-Secure Rescue CD picked up on a backdoor rootkit and a piece of spyware and supposedly eliminated them. BitDefender picked up on my own personally developed software - VerifyMyPC - because it has both registry and file system scanning capabilities. False positive there. Again, as of July 2010 - F-Secure has the lead in building Rescue CDs with solid detection. But apparently that isn't good enough.

I spent the next couple days simply scanning the system with software I trusted - including installing AdAware, which now appears to have a real-time component. Bleh. The rootkit apparently hid itself quite well during the time AdAware was running. I eventually uninstalled it just to see if I was in the clear. It took a couple reboots but the rootkit and all that came with it returned with a vengeance. I shut down the computer again and began my plans for reinstalling Windows.

I knew I was pretty much hosed at that point. Reinstalling an OS is the ONLY way to know that you have truly gotten rid of a rootkit.

By the way, Live CDs of Linux are awesome. I highly recommend burning a copy of the Ubuntu Live CD. It works VERY well for recovering from failed Windows installations. Or environments with rootkits installed. Plus it seems to have every device driver on the planet included, which means networking and pretty much every USB thumbdrive and external hard drive will work right away - so backing up data prior to a reinstall is really easy.

Fortunately, I keep very good backups. Microsoft SyncToy saved my butt. I also eventually started keeping all my core data on a separate hard drive for the just-in-case scenario I need to reinstall Windows. Doing this helps to lessen the painful process.

On the upside, I've been meaning to install Windows 7. So this merely forced the issue of upgrading. Also, during this reinstall, I've made the decision to use as many "portable applications" as possible. I really miss one aspect of the old DOS days where each application was centrally contained in its own directory. Any application that did anything outside of the directory it was installed to was deemed to be a bad application. Windows came along and messed up my beautifully organized world of self-contained applications. With the advent of the USB thumbdrive, some good people have taken it upon themselves to recreate a world of self-contained, well-behaved applications again under Windows. By using portable applications, I have a pretty good chance of easily installing newer versions of Windows in the future.

There are a number of applications I use extensively that I hate reinstalling: Firefox, Thunderbird, Apache, PHP, MySQL, and Visual Studio. Standalone Visual Studio is a pipe-dream. But the others are definitely doable. Firefox and Thunderbird both have "portable" versions. I use both of those more than anything else these days. And the portable versions seem to operate rather well thus far.

At any rate, I'm left wondering what exactly the purpose of dropping a rootkit onto my computer was. I'm also pretty sure that my neighbors aren't intelligent enough to use the hardware and software tools required to hack a "secure" wireless network. But you never know for certain. On top of that, there are several insecure wireless networks in range of my own network. Low-hanging fruit and all that aside, someone put forth effort to gain access to my "secured" wireless network.

Given that my neighbors aren't likely to have done thins, it leaves me with two options - rogue government operation or botnet. Let's shave paranoia with Occam's Razor. Which brings me to my question: Is this the next generation of botnet tactics? Growing botnets by seeking out both insecure and secure wireless networks? I'm still not sure how my completely random WPA-PSK key was obtained but, with the processing power of a botnet at one's disposal, I can conceive that it would be possible to hack such networks. Once inside, seek out Windows boxes, expand network, wash, rinse, repeat. Each new computer on the botnet means that much more processing power, thus shortening access times to keys.

This definitely has the smell of a botnet. Botnet operators are predictable creatures of habit. When they gain access to a new machine or network, they immediately deploy a rootkit on the system (check), deploy spyware/additional tools (check), then start seeing what is on the system that is of value (probably check?), and start transferring things of value (maybe 1MB at most - if that - pretty sure they never got that far). The way the rootkit came back and something started talking again to a remote system really says "botnet" to me. In a sexy, nerdy/geeky sort of way. Interestingly, none of the anti-virus and anti-malware products truly cleaned the system up. Even the Rescue CDs missed whatever was installed, which means unknown, undetectable malware. Even if I had real-time anti-virus software installed, there was a good chance it wouldn't have detected the installation of the malware.

Prior to the installation of Windows, I wiped my BIOS and reinstalled it - I keep a copy of my current system BIOS around as well just-in-case - and also wiped the CMOS settings. It depends on the rootkit, but they occasionally inject themselves into the BIOS. If anything survived all of that, I'll be impressed but slightly annoyed.

My only conclusions are thus: All wireless networks are insecure and running Windows on a wireless access point is just asking for a rootkit and botnet to get installed.

In a future blog post, I will share a solution to this problem that will allow Windows users to safely operate on a wireless access point. Assuming my idea works. Until then, turn off your wireless radio and use Ethernet cables. Data transfer is significantly faster over Ethernet anyway.

A call to open source developers: Let's eliminate ICANN.

2010-06-22T11:13:00.010-04:00

In the field of Internet development - ICANN and Network Solutions/Verisign are eyesores. There is a very unhealthy relationship between the two organizations and ICANN holds a monopoly on the Internet as a whole by holding the domain name infrastructure hostage.

On July 1, 2010, a price hike for .COM and .NET domain names will take place (VeriSign is the sole registrar for those TLDs). That means it will cost more to purchase and maintain those type of domain names.

The core problem is the Domain Name System (DNS) as a whole. It was designed in the dark ages of the Internet by a bunch of nerds to map a name to an IP address. It was wholly owned by InterNIC, now known as ICANN through various transactions - or at least that is the best I can explain it in a single sentence. The original Internet (ARPANET) was designed to supposedly be robust in the event of nuclear war and people like it for its supposed anonymity. Basically, the Internet was a United States Department of Defense project and it was robust enough at one time. Now the Internet is mostly commercialized and that means consolidation and therefore no longer capable of withstanding nuclear attack - or so I read somewhere. I digress. However, the United States, in essence, still owns the entire Internet because ICANN has an unhealthy relationship with the Internet as a whole - it is a U.S. "non-profit" organization that works in tandem with the U.S. Department of Commerce, which, in turn, reports directly to the President of the United States.

In other news, if current legislation passes, it will become possible for the President to virtually turn off the Internet for entire countries because everyone relies on DNS to map domain names to IP addresses. If the legislation passes, the President could turn off an entire country if there are enough "loopholes": Tell ICANN to delete/suspend root server DNS entries of every business and government website in the target country. ICANN could potentially have no recourse and therefore it would be "goodbye [country name goes here], nice knowing you". Not that it would ever happen. If it did happen, there would be a massive backlash, someone would deface the whitehouse.gov website, and a lot of backpedaling would take place.

Anyway...back to the actual topic.

In essence, purchasing a domain name is simply renting a sequence of human-readable letters, numbers, hyphens, and '.'s. You can never truly "own" the domain. But, more importantly, DNS is a creaky mess and rather poorly designed. Very few people truly understand how DNS operates, myself included, but this much I do know: There should not be one single organization dictating who owns what domain. But it really is much simpler than that: There should not even be a domain name system at all.

The purpose of DNS was to apply hierarchical, human-readable labels to an IP address. While it worked fairly well for a while, it has become a disaster. There are a whole slew of "record types" (A, CNAME, MX, SPF, DomainKeys, etc.) that are more confusing than useful. And .com, .net, .org, .co.uk, .xyz, .yourmom, www., etc. are increasingly meaningless and confuse most users. And, with ICANN's mandated DNSSEC extensions (which includes more "record types") being rolled out next month as well, there will potentially be a lot of broken infrastructure.

Here is what I want to see happen: Throw out DNS in favor of a cloud-based approach. Surely some of the technology surrounding the latest cloud-based computing initiatives can be applied to the basic underpinnings of the Internet. It would free the Internet from the tyranny of ICANN and every domain name registrar on the planet in the process. Registrars are expensive and greedy!

One of the things I also want to see go away is all the '.' nonsense. We need to stop thinking in terms of '.com'. It is only required because ICANN says so and they are constantly putting out new TLD extensions, which means defending a brand via the domain name system alone is nearly impossible unless you have millions of dollars burning a hole in your pocket. Most businesses know exactly what I mean.

Also, WHOIS needs to vanish. Under ICANN, correct WHOIS information is a requirement. Most people (mostly individuals) who register a domain do not realize that their personal information is being published to a publicly searchable, indexable database. Name, address, phone number, e-mail address. They might as well publish the person's social security number, a few bank accounts, and several credit cards in the process. Services have cropped up to replace public information and "privatize" it with other information. However, under ICANN rules, doing this effectively makes those companies have ownership of the domain! Plus you have to pay them extra beyond the cost of the domain. So you are paying someone else to own your domain for you so your information that should have been private in the first place is actually private...which doesn't make ANY sense. Public information in WHOIS is used by spammers, telemarketers, creepy stalkers, former employers, and competitors to harass and spy on both individuals and businesses. And who knows what governments do with the information! This invasion of privacy is completely unacceptable. On top of that, incorrect WHOIS information is grounds for letting someone else take your domain name. Wikipedia tries to give WHOIS a positive spin by saying that law enforcement benefits from this invasion of privacy. Great - so what is MY benefit?

I said before that the DNS system is old and crusty. So old in fact that it can't handle Unicode natively. Internationalization of domain names is done via a horrible hack known as Punycode. The hack takes the limited 37 allowed characters (a-z, 0-9, and the hyphen) in a domain name label and maps them to Unicode characters. On top of this, each domain name label is limited to 63 characters and the full domain name can't be longer than 253 characters. When you start talking international Unicode mappings, the 63 character limit starts to look more like 15 characters.

There needs to be a completely new system built that doesn't rely on ICANN, Network Solutions/Verisign or any other registrar, eliminates WHOIS, takes into account the fact that the Internet is international in nature, and that search engines are the primary means for finding most things these days. This is a task that some smart and creative open source software developers can take on.

Also, there needs to be a way for search engines to hook into this system to get a live feed of data. That would be a huge improvement over the scraping and crawling nature of the web. Google has made great strides in this regard with Sitemaps but I'd rather see this information being pushed to search engines instead of pulled. Pulling information is slow and error-prone. Pushed information can be formalized and result in live and/or near-immediate updates.

Also, this system needs to take into account internal networks, VPNs, and NATs. Maybe improve on the concepts somehow in a sort of Tor network/SSH tunnel way of doing things.

Being able to also declare temporary (optionally secured) resources on a network would be a great improvement over DNS. For example, if you want to send a 100MB file to someone - how do you do that now? E-mail? FTP?

As to the actual programming and implementation of this system, a Distributed Hash Table (DHT) approach would be a pretty good starting point. Obviously, since DNS has been around for a really long time, it will have the initial edge in terms of efficiency. However, if anything, Google has proven that cloud-based architectures are incredibly efficient and perhaps more than DNS is or ever will be. So a DHT or something similar has a pretty good chance of working rather well. Computers are staying mostly on these days, so there would be a massive network of nodes.

By the way, if this post doesn't seem well-thought-out, it is just a jumble of ideas that I want to get out there. Hopefully someone can take these ideas and create a legitimate open source product out of them. Everyone who owns a domain thinks that domains are expensive and that DNS is very confusing. DNS, ICANN, registrars, and the whole mess are in need of obsolescence. We need something better. Thankfully, search engines are already working toward making that possible.

Oh - and while you are at it creating this replacement system, feel free to obsolete the IETF in the process. They make spammers possible by ignoring the problem that SMTP represents. Redesigning the entire Internet infrastructure to handle future needs seems like something that the open source community could handle quite admirably and do better than formal organizations seem to be able to do.

What I've been doing for the past year...making a CMS.

2010-06-21T23:27:00.006-04:00

Over the past year, I have, ever so slowly, been dropping off the radar of my usual stomping grounds. Basically, I've spent the past year positioning myself in the industry to stay relevant and gear up for the next decade of software development.

Where is the software industry headed? Simply put, we are headed to a very mobile realm. What exactly that will look like is anyone's guess. What fascinates and intrigues me is always-on Internet in the palm of my hand. THAT, to me, is mobile. The problem we currently face in the mobile arena is that no one is making a device I want to program for. I willingly program in two languages: C/C++ and PHP. Since no one makes a multitouch device that I can write plain ol' C/C++ for (yet), I'm left with PHP.

To that end, I've started down a rather interesting path: This past year, I wrote my own Content Management System (CMS). For the past few years I've experimented with all sorts of lame-brained product ideas (some more successful than others). This new CMS, however, is the game-changer I've been hunting for. I'm at the top of my game when I get to do things that are new and exciting that I'm actually interested in.

Without further ado, here is the website for a new open source product called Barebones CMS:

http://barebonecms.com/

Barebones CMS was born from talking and listening to people in the web design and development industry. I generalized each complaint, grouped them together with my own personal pet peeves in web development, and organized my thoughts on what a CMS should look like. I only wanted to write this software product one time, never requiring a major rewrite. The time spent upfront gleaning information was absolutely invaluable.

There was a secondary goal during this process: I was also waiting for enough stable third-party components to come into existence such that my life would be made easier. Barebones CMS relies on several bleeding edge components to get the job done. Three to four years ago, Barebones CMS would have been impossible. Only last year did the final pieces fall into place. There are some pretty obscure components involved.

One of the key goals that became apparent during my listening phase was that equal treatment needed to be given to programmers and web designers. I like to hand-code my websites as do many other people I know. This brought me to the realization that Barebones CMS needs to cater to three different and unique groups of people: Programmers, web designers, and content editors. Microsoft caters and shows love to developers, developers, developers, developers. That formula works really well for them. So, Barebones CMS caters to each group with a unique default widget: Programmers get the Code Widget, web designers get the Layout Widget, and content editors get the Content Widget. Each widget is incredibly unique and accomplishes the tasks of each group's needs.

I quickly realized that I was developing this product for a team environment. Each person has different talents they bring to the table and has different needs as a result. However, each person needs to be able to interact well with the other team members and I ended up deciding I wanted to make Barebones CMS a supporting tool in that team environment rather than creating my own team environment. What I mean is that instead of forcing developers to take a specific approach to developing a website, I made almost every feature of the Barebones CMS completely optional. This leaves the door wide open to developing a site in an infinite number of ways. Teams will feel right at home in Barebones CMS because they don't have to change how they do things if they don't want to.

I also wanted the core features of the product to not get in the way of creating and maintaining a website. For example, I've discovered the frustration of caching systems that tend to get in the way of developing a website. I made it a very specific goal to never have the caching system get in the way or any other system for that matter.

Finally, I spent almost three months carefully documenting each aspect of the Barebones CMS - attempting to make the documentation easy to understand for everyone. During the documentation process, I found numerous annoyances in the product itself that I fixed. When all was said and done, I had gone through 21 release candidates before releasing Barebones CMS 1.0. That is not a typo. Twenty-one release candidates. Most of the issues were annoyances - a few of the fixes resulted in new bugs that got squashed shortly after they were introduced.

If I had to summarize Barebones CMS and compare it against some other professional product, it would be this:

- Barebones CMS is Dreamweaver in your web browser.

However, I prefer not to make such comparisons. Please don't quote me on that :)

How does Barebones CMS relate to the future mobile development? Well, Barebones CMS is a website creation and editing tool inside a browser. With Barebones CMS, it is entirely possible to develop an entire website on a mobile device. Not that you would necessarily want to do that on a mobile device. It is possible though. And possibilities are the drive behind innovation and are what push us forward. The reality though is that mobile platforms have a long way to go yet.

Barebones CMS is a serious, professional, and ambitious project. It is professionally developed and produced, is heavily tested, everyone who has followed my efforts on this project absolutely loves it, and it meets and exceeds what I expect from a CMS. I personally find it an absolute joy to use. Plus I get to eat ice cream and homemade pie tomorrow at a celebration launch party for the Barebones CMS. I'm completely psyched about that. (I still have to make the pies though.)

There are so many more things I could say about Barebones CMS but, since I spent a year on this project, the least you can do is visit the Barebones CMS website. There is a ton of introductory material I haven't even covered here. I really just wanted this post to be a rationale on why I vanished for a year. If you look at my posts over the past year, you can see Barebones CMS in those posts. Anyway, go visit the Barebones CMS website.

One last thing: Barebones CMS is just a tool. My hope is that you will add Barebones CMS to your collection of tools for creating websites and that you will always use the right tool for the job.

IE6 will be dead in 6 years!

2010-03-20T17:48:00.008-04:00

IE6 will officially die six years from now (i.e. it will finally be completely dead in 2016). It is mathematically proven by this chart:

(Download Excel Spreadsheet)

The data comes from the W3C Schools browser statistics page. The "Months from 10.2% Overlap" is referring to the two points where browser usage of IE5 and IE6 were both at 10.2% (time-shifted so you can see how similar the data is). The W3C site appears to stop tracking browsers at 0.5% (I'm calling that dead). My goal with this chart is to show that the data is rather similar at this point in time.

Six years is a pretty bleak outlook. I also had Excel fit a line to the chart and came up with "y = -0.0093x + 0.085" with a R^2 of 0.9854 (a slightly better fit with the available data nodes). With the line, IE6 is declared dead by the W3C Schools' standard of 0.5% in 8.5 months. So, the real answer is probably somewhere between 8.5 months and six years. I'm leaning more toward five years. Come on - that's a curve and we all know it. It won't matter how many websites put up "You are using a dead web browser." People clearly are ignoring those signs and only upgrade when they want to.

Pseudo-transparent 24-bit PNG in Internet Explorer - No hacks

2010-03-20T13:11:00.018-04:00

I've been on a kick lately with transparent PNGs. Due to recent discoveries involving PNG8, it is high time to revisit the world of PNGs and see what else there is out there. In my previous article on PNG8, I discussed how to get nice-looking PNG8 images by using Photoshop and pngquant. Most articles on PNG8 only cover either Fireworks or straight-up pngquant/some other command-line tool. The results aren't that great looking.

Today, I stumbled across this website and tried it out in IE6. Surprisingly, the first 24-bit transparent PNG "worked":

Why is this interesting? Because a pink background shows up. We are all so used to that ugly gray background in IE6 that no one has stopped to think that maybe the background color can be altered. Instead, people have gone for various hacky solutions to get full transparency. Well, now there is an intermediate, no-hacks solution between the full transparency hacks and PNG8.

Here is the image I will be working with:

(This is a licensed icon from my massive icon collection. It is ILLEGAL to copy this icon for your own use.)

This image has a nice drop shadow but being the transparent image it is, it looks hideous in IE6:

Looking in Wikipedia's article on PNG, there is a lot to learn about the PNG file format. However, the only thing relevant is that PNG is made up of 'chunks' and one of those chunks is called 'bKGD' and sets a default background color that web browsers aren't supposed to display but...IE6 does. We can use this to our advantage!

Running a few Google searches for the specific chunk name 'bKGD' turned up a Windows-only tool called TweakPNG. It, among other things, can add and manipulate a 'bKGD' chunk in any PNG image.

Step 1: Make a backup of the original image as TweakPNG will overwrite it.

Step 2: Start up TweakPNG and load the image (drag-and-drop or use File->Open).

This tool is designed for modifying PNG files at the 'chunk' level. So be careful. If you mess up, simply duplicate that backup image you made and start over.

Step 3: Select "Insert->bKGD (Background Color)".

Step 4: Double-click on the new chunk, select the desired background color (defaults to black and the color picker uses the hideous Windows color picker dialog, but whatever), click OK, and save the image.

Here is what the result looks like in Firefox:

The 'before' image is on the left. The 'after' image (after adding the 'bKGD' chunk) is on the right. The blue background is intentional to show that Firefox correctly ignores the background color that was just set.

Here is what the same images look like in IE6:

The image on the left has the familiar, ugly gray. The image on the right has the white background that was set using TweakPNG. Both are on the same blue background as before. As you can see, this method only offers pseudo-transparency. IE6 still does not consider the page background. So this approach WILL work for solid backgrounds but NOT backgrounds with patterns. This approach also makes it a bit difficult to change the background later on. However, TweakPNG is a very useful tool to have in one's arsenal of tools.

Setting a background color in every PNG is a good fallback position. For example, this will degrade nicely in IE6 where Javascript is disabled and Javascript-based hacks are being relied on for transparency support (e.g. DD_BelatedPNG).

One final note. As I was searching for a tool to add bKGD chunks, I ran across a CodeProject article entitled "Writing a Background Color (bKGD Chunk) to a PNG File" on the topic. The article is pretty basic and only has source code (no binaries) but does mention that the 'gAMA' (Gamma) chunk that Photoshop outputs can cause images to be "darker" in IE and should be removed for consistency. TweakPNG can be used to remove such chunks if you so desire.

Of course, if full 24-bit transparency is needed in IE6, the only solution is still a tool like DD_BelatedPNG.

If you think this effort is a silly waste of time, keep in mind that Internet Explorer 6 is going to still be around for another 6 years.

Transparent PNG8 is THE Solution To IE6 Transparent PNG Woes...For Photoshop Users!

2010-01-20T08:27:00.018-05:00

IF you know what you are doing, PNG8 can be THE answer to IE6's broken support for transparent PNGs. Ironically, the web is discovering PNG8 just as we are finally phasing out IE6. If you develop for the web occasionally, you know what a pain IE6 is. I'm currently working on a very hush-hush project (hence no blog posts lately) and it is almost the last time I have to ever support IE6. After these projects, I will never, ever, EVER have to look back (woohoo!).

Background

If you do a lot of web development, you know about all of the hacks in IE6 to introduce PNG transparency. My current favorite full-blown solution to 24-bit alhpa transparent PNGs in IE6 is DD_BelatedPNG. However, even that has problems and is still a hacky workaround for a problem that shouldn't have even existed for more than two minutes. The moment it was discovered, it should have been fixed.

I initially dismissed this Transparent PNG8 approach because I wasn't going to spend $300 on a piece of software (Adobe Fireworks) just to be able to do something a certain web browser should already be capable of doing.

Anyway, this post is about something I found out about a few months back but only just tried out today. Today, I will show those who only have Adobe Photoshop how to get the benefits of Adobe Fireworks Transparent PNG8 support (only not slightly broken?) without spending $300 on Fireworks. If you aren't familiar with or ever heard of Transparent PNG8 (or even Adobe Fireworks), don't worry, you aren't alone. I discovered the technique by complete accident. Here are two articles on the subject:

Degradable PNG Transparency for IE6
PNG Alpha Transparency – No Clear Winner

For the second article, the author doesn't understand what the Transparent PNG8 buzz is about and seems to assume that the final file size is important. If the goal is to get a transparent PNG in IE6 without ANY hacks, then file size is NOT important (well, if it output a 1MB image, then we would care - it doesn't). I include the article here because one of the comments points out something VERY important about PNG8:

PNG8 is GIF but you can have multiple indexes that are partially transparent.

In other words, if your image looks like crap in GIF, it will look like crap in PNG8. Gradients and full color photographs will have issues.

That said, though, most of the images I have converted to PNG8 actually look pretty darn good. I'm a quality nerd/geek, but I understand the restrictions of PNG8.

And without further ado:

The Step-by-Step Guide to Transparent PNG8 Images with Photoshop

Actually, you can use any image editor to accomplish the ultimate goal of Transparent PNG8 images, but I'm going to show how to do this with Photoshop.

Step 1: Open the image you want to convert to a Transparent PNG8 image in Photoshop.

(I'm assuming all layers have been flattened and we are working with a static image.)

Small images (e.g. icons) work best because they will generally have few colors and artifacts won't be nearly as noticeable. Ever try converting a full-color photograph to GIF? Yeah...same thing here. I'm using a 24x24 icon for this demonstration (zoomed in so you can see details).

Step 2: Duplicate the image/layer to a "New" image.

Basically, this is a backup copy. You HAVE to use a separate image and not just a new layer. In fact, it is a good idea to backup the original file just in case you screw up and need to start over.

Step 3: Prepare the image for IE6 users.

Image -> Mode -> Indexed Color... (in Adobe Photoshop CS3)

If you notice weird artifacts of creating an indexed palette, check to see how many colors are in the image. In a high-color image, the entire 256 color palette will be filled in. My tests show that the best transparency results occur when you leave about 10 palette entries open for the final step.

The goal of this step is to create what you want IE6 users to see. In this case, Photoshop picked up part of the drop shadow and put it into the indexed image. In the event you are deleting pixels, you should undo the indexed color palette step, delete the pixels, and re-index the color palette. You may also want to add some missing pixels (fairly rare in my experience). Do so with the current palette (copy and paste pixels).

You may have to play with the quantization settings (technical term for the dialog's options) to get the look you want IE6 users to see.

Step 4: Prepare the image for everyone else (IE7, Firefox, Safari, Chrome, Opera, etc).

Now, switch the image back to RGB mode.

Remember that copy of the image you made back in Step 2? Copy it back into the current image in a new layer. Position the new layer underneath the first layer.

Now save the image to a PNG.

Step 5: Converting the image to PNG8.

If you've been wondering how we get the image into PNG8 without losing transparency information and then scratched your head at the end of step 4 because the output of Photoshop's PNG format is a regular transparent PNG, then wonder no more. The purpose of all those steps is that I've found that it helps to help out this next step. Plus, you get super-refined control over the final output (remember, I'm a quality nerd/geek).

This step has a dependency on a tool and perhaps an environment you aren't familiar with (the "dreaded" commandline). I discovered the tool used here from this post on Stack Overflow.

Despite people's recommendation for 'pngnq', my recommendation is 'pngquant'. The reason is simple: The Windows EXE (I run Windows) is a self-contained unit that simply works without having to hunt down two additional DLLs. The author of 'pngnq' tells the user to go find the missing DLLs out on the big bad Internet (a VERY bad idea). pngquant also seems to produce more correct PNG8 images that don't do funky stuff in IE6. In addition, we've already quantized the image once using Photoshop to our liking. The built-in quantization algorithms in 'pngquant' aren't going to improve anything (i.e. the quantization debate on Stack Overflow is rather moot if it doesn't display correctly in IE6).

Anyway, download pngquant and unzip/extract it to a new directory. If you don't know what a directory is, uh, ask someone else. You really can't call yourself a web designer unless you know what a directory and the commandline are and know how to use them. I expect fewer know what the commandline is.

Copy the image to the directory you want to convert to PNG8 (or make the path to the image easy to get to from the commandline).

From the commandline run 'pngquant -ordered 256 [imagefilename]':

You can also try 'pngquant 256 [imagefilename]'. This is the same command but uses a different dithering mechanism. Since we have already dithered the image once, a second dither of the same type usually looks pretty bad. Ordered dithering seems to work better (in general).

(The '256' option in both commands says how big to make the palette. The maximum number of colors in PNG8 is 2^8 = 256.)

You can also try skipping steps 1-4 and take your images directly through pngquant or pngnq. I've found that, in general, these commandline tools end up creating horrible looking PNG8 images in IE6.

Step 6: Testing the results.

Open the output image (there should be a new image in the directory) up in both IE6 and Firefox to make sure it actually looks good, integrate it into the page, and then you are done!

I've found that most images work pretty well in IE6 as Transparent PNG8 images when they are carefully dealt with. The only case is where someone wants to get really fancy and make the entire image partially transparent. The indexing process removes that partial transparency because IE6 will only display fully-opaque pixels.

If you do need full transparency support in IE6 (and the evils dragged along with it), DD_BelatedPNG is pretty much the best solution out there.

Edit - March 20, 2010: Today, I ran across a way to get pseudo-transparent PNGs to work in IE6 with 24-bit images without hacks. The article goes into great detail on the topic but offers a possible middle-ground solution between the various hacks we have a love/hate relationship with and PNG8 with its limitations.

Time Waster: X-PHP-QUESTION - Yes, I did notice

2009-06-20T10:43:00.006-04:00

Welcome to Time Waster article #1. Time Waster is going to be where I put weird little easter eggs as I find them. Either programming hiccups or weird software oddities that caused me to waste time. Today's Time Waster involves the PHP website itself. To participate, you will need Firefox with the Firebug plugin installed.

Visit the PHP website in Firefox.

Now, open up Firebug and you should see something like this:

(Why, yes, I did notice.)

A few other people have noticed this. Looking at the source code for their site doesn't have any explanation for its existence. Someone on the PHP developer team thinks they are cute.

Bad Design: Windows Task Manager

2009-06-12T10:55:00.004-04:00

I've decided that Windows Task Manager is ill-conceived, poorly designed, and extremely misleading. Especially the following:

Most people have seen the "Performance" tab and that memory usage chart. Charts have a more powerful influence than a bunch of numbers and statistics. Take a good look at the chart and answer one question: What is the user going to think?

Most users have no idea what a 'page file' is. But most can understand the terms 'memory' and 'CPU'. The chart is extremely misleading. The chart above this chart is 'CPU' usage. In terms of all the charts in Task Manager, that chart is the most useful.

Think in terms of the average user trying to figure out why the computer is extremely slow. Now, based on this image, what can you tell about the current system?

That is right - the average user cannot figure it out. This means Task Manager is poorly designed and is downright misleading. The user is mislead to believe that the page file chart = RAM usage. Additionally, there appears to the user to be plenty of RAM for programs (220MB), so that couldn't possibly be the problem.

In all actuality, RAM usage _IS_ the problem for why the computer is so slow. There is 1GB of physical RAM available to the computer, 1.6GB total is being used, 2.1GB was used at one point during the session, and 2.5GB was used sometime in the computer's past. The solution to this computer's woes is to chuck in more RAM. But the average user won't know that because Task Manager is not showing them useful information via the chart. Using the page file as a chart is a bad design decision. At least change colors showing critical cutoffs so the average user can decipher that RAM usage is the issue when hunting down the reason for why everything is so slow.

Maximum failure: PHP 6 deprecates short tags

2009-06-08T09:31:00.010-04:00

I just discovered that PHP 6 is officially deprecating short tags. I'm sorry, but short tags are incredibly useful. They should be turned on for every server. They simplify coding dynamic actions within HTML. Plus, you get the added bonus of using:

<?=$x?>

Which is short for:

<?php echo $x; ?>

Compare: 7 characters vs. 17 characters. The latter is over twice the length, quite unreadable, and, from personal experience with large code bases, unmaintainable! Typing 'php echo' in every time I want to execute/display data that is hosted within the PHP engine is ridiculous. This is a step backwards going in the wrong direction.

The lame excuse for removing short tags is XML. PHP is for HTML, not XML.

I write PHP code quite regularly. Prior to PHP 5, PHP stunk. PHP 5 really changed my view of scripting languages. PHP 6 is going to ruin that view as we will return, once again, to the dark ages of software development. What really gets me is that someone made a decent recommendation and the PHP development team scoffed at the idea and one person even had the gall to suggest that syntax highlighting is all that is needed - let me point out that blind PHP programmers probably don't want to hear JAWS attempting to speak '<?php echo' over and over again and syntax highlighting is a useless recommendation to a blind programmer. I can imagine a small army of blind PHP programmers beating up the PHP developers (especially the one developer who made the recommendation to use syntax highlighting) while blaring an audio recording of JAWS speaking '<?php echo' over and over.

XML seems to be the pervasive reason. I personally dislike XML and wish it would go away (I also wish Ruby, Perl, and Python would go away too - all three are equally terrible languages - and, yes, I've used them all). I've used XML enough to know that it is a terrible storage mechanism for any data. JSON is better but still not perfect. The PHP serialize() function gets even closer but is a language-specific solution. XML is hereby a lame excuse for not coming up with an alternate short tag strategy.

If XML is sooo important, then simply recognize the sequence of characters '<?xml ' and ignore it. It is already invalid PHP code, so fix the problem transparently. Oh the horror! Three lines of code at most is all that is needed. This whole issue is over a data storage format that isn't commonly output by PHP, is a terrible storage format to begin with, and is over the two starting characters at the very beginning of the storage format. Most people DO NOT EVEN USE XML in the first place! Requiring '<?php echo ' every time output is needed from within the PHP engine in HTML is a surefire way to tick people off. Especially when they learn that the reasoning is the two measly characters at the start of a XML document.

Good grief. I can't believe I'm having to tell the PHP developers how to write software. I'm now going to start tagging posts with "maximum failure" for areas where programmers have failed spectacularly to see the simple solution to a problem. End-users (PHP programmers) want short tags in PHP, the PHP developers see a singlular issue with XML and proceed to blow the whole thing out of proportion, and there is an easy fix (see previous paragraph) that allows both camps to be happy - it is as simple as that.

I'm hoping that as soon as PHP 6 comes out of its internal development cycle and becomes the official release, there will be a significant (read: massive) backlash in the short tag department. In other words, I'm hoping the PHP developers will be forced to add short tags back in and undeprecate them. The only reason there hasn't been a huge backlash already is because: No one knows they are already deprecated, deprecated functionality is rarely enforced, and people are too lazy to rewrite code that works just fine. So, I have one question:

Can the PHP internal developers mailing list handle a million programmers dropping by and asking the same question over and over and over and over and over and over again?

Edit: To date I've rejected all comments to this post. People are attacking me personally and completely ignoring both the purpose of this post and this blog as a whole. This is my rant blog - a place to let off steam - if you don't like it, then don't comment and simply leave. Sheesh.

unsigned long long long long long int

2009-03-16T10:24:00.006-04:00

Congratulations! You've encountered one of the more hair-brained stupid schemes of the past century. There is no way in ANSI C/C++ to say, "I want X bits of storage for this integer."

Your initial thought might be, "Wait a minute, that can't be right...right?" Let us look at the various data types available to an ANSI C/C++ programmer:

char
int
float
double
struct/class/template/union
pointer

A 'char' lets us declare a single byte. A byte is 8 bits...or is it? CHAR_BIT (in limits.h) is typically defined as 8 bits (and the Standard requires it to be defined as a _minimum_ of 8). However, it could be defined as 9 and there has been some hardware where it is defined as 32. Additionally, it is up to each compiler to decide whether just saying 'char' is signed or unsigned.

Thankfully 'float' and 'double' use the IEEE floating point standard. Edit: Due to some response, I should clarify: Every compiler I have ever used (and I've used quite a few) relies on IEEE floating point for float and double. However, the ANSI C/C++ Standard does not require IEEE floating point to be used (it just happens to be convienently used basically everywhere).

The struct/class/template/union are not technically 'data types' but more of a grouping mechanism to group logical chunks of data types (and data) together.

Pointers can point at any of the other data types.

Now that I've worked through all the other types, we are left with 'int'. People first get their first taste of this problem when they discuss sizeof(int) on the various forums out there. They learn quickly about 'short', 'long', 'signed' and 'unsigned'. Below is a chart of what the ANSI C/C++ Standard says is the _minimum_ number of bits for each type of 'int':

short int - 16
signed short int - 16
unsigned short int - 16
int - 16
signed int - 16
unsigned int - 16
long int - 32
signed long int - 32
unsigned long int - 32
long long int - 64
signed long long int - 64
unsigned long long int - 64

Those are the minimum number of bits. Each compiler author chooses what various forms of sizeof(int) are in terms of bits. So, some compilers are 16. Some are 32. Some are 64. And there are even a couple compilers that define sizeof(int) as 24 bits. It varies so widely.

The 'long long' type was added so that 64-bit hardware could be programmed for in C/C++. And, to accommodate this hacky extension of C/C++, you use the printf()-family format specifier %lld, %lli, or %llu to display the integer.

So, now here is a question to ponder: What happens when native 256-bit data types show up in hardware? Are we going to create more hacky accommodations and sit all day writing 'unsigned long long long long int/%llllu'? What happens if some popular compiler vendor decides that 'unsigned long long long long int' is 384 bits? Or are we going to start mixing 'short's and 'long's - 'unsigned long long short long int'? How much longer will this "minimum number of bits" foolishness continue? The lack of being able to declare how many bits/bytes we need for integer representation is silly and only going to get sillier as int sizes get larger.

Someone will point out bitfields. I hereby point any such someones at the numerous issues surrounding bitfields. Bitfields are typically restricted to the size of an 'unsigned int', have poorer performance (compared to doing the same things they do yourself), are only available
inside 'struct's, etc. They are a modest solution, but hardly workable in the long run.

'int' (implicitly 'signed') and 'unsigned int' should be all we need and should mean, "I don't care how many bits there are for this integer and trust the compiler to generate enough bits for the target platform for most common operations." That is - 32 bits for 32-bit platforms, 64
bits for 64-bit platforms, etc. For specific declarations where we need exact bit sizes and refined control, I'd like to see something like:

int@64 x, y;

To declare a couple of 64-bit signed integers. The '@' is illegal C/C++ right now (i.e. won't compile if you try it) but would be a pretty good candidate.

For printf() and similar routines, %@64i seems like it could work well enough. The '@' symbol also avoids existing printf() "extensions" such as multilingual parameter support. This suggested approach would make things more readable and this approach also allows us to get rid of the near-worthless 'short' and 'long' keywords from the language.

I'm definitely agreeable to restrictions on such an addition to the language such as "implementations may choose to place a limit on the value after the '@'". And such limits could be "multiple of 8" or "power of 2" or "between 1 and 256 inclusive" (or a combination). For where a compiler has restrictions and does not necessarily provide native support, it should be possible to do something like this:

#if !native(int@512)
typedef __Int512Handler int@512;
#endif

int@512 x, y;

The 'native' preprocessor keyword would mean, "If the compiler supports this type natively" (the above example means, 'If the compiler does NOT support this type natively'). The above example would allow printf() to assume that the value passed in will be of the specific class if the @value falls out of range of native values. The class would provide the necessary logic to handle all normal integer operations.

Even with the above, the compiler author should still be able to put restrictions on what can be declared. Let's say the compiler author only wants to support "integers that are multiples of 8 and powers of 2 up to 512 bits" and the hardware only supports everything in that range up to 64-bits natively, then the author only has to define 128, 256, and 512 bit support. As each platform comes into existence, native support can be added and, in the event older native elements disappear (e.g. 16-bit), support can be relegated to a class. Old code would just simply need to be recompiled for the new hardware.

Sadly, though, the ANSI C/C++ committees aren't likely to ever see this suggestion. So, we will likely end up writing 'unsigned long long long long short long long long short short long int' in the not too distant future.

Edit: Someone pointed out stdint.h to me. I feel slightly embarrassed for not knowing about this BUT implementations are only required to provide definitions for types that the hardware and compiler supports. My approach does something similar EXCEPT it allows us to get rid of a potential plague before it becomes quite troublesome: 'short' and 'long'. The whole point of this post is that we are using 'short' and 'long' for something they are NOT intended to be used for - to define integers of specific bit-lengths and if we keep on going like this, it will come back around to haunt us later. stdint.h is definitely a good starting point but we need to get rid of 'short' and 'long' in favor of a more generic approach.

"Hair-brained stupid schemes" is a great phrase.

Into the Deep Fryer

2009-03-14T23:52:00.005-04:00

I've ranted before about the poor quality of today's constructed components. I have another rant today that isn't really computer related but still interesting nonetheless.

Back in the day when IBM reigned supreme, they came up with a piece of "big iron" that still runs much of the financial world. Most people will recognize the following instantly: AS/400.

What crossed your mind when I said that? "Old" and "Ancient" come to the forefront of my mind. But are immediately followed by the words and phrases, "Reliable", "Solid", "Well-built", "Well-engineered".

Now think about today's software and hardware. Do you think the same things about reliability, being solid, well-built, and well-engineered? Probably not. This is a plague in our world and it boils down one simple truth: People don't care deeply about their jobs today. And that lack of caring results in slipshod, poor quality design and construction. Worse, people don't take responsibility for their lack of care when something really bad happens. Even worse, we have laws in some cases where people who fail at life can simply point at the law and say, "Well, the law says we have to do that, so we did........and therefore I'm not at fault."

And this brings me to deep fryers. "Haha" jokes aside (yes, I realize I could do a number of humorous bits here), a deep fryer is a pretty dangerous piece of equipment. You have to be responsible to own and operate one. You've got a heating element that can melt and/or set fire to virtually anything, really hot oil flying everywhere capable of inflicting 3rd degree (or worse) burns and/or starting grease fires, and typically some (usually flimsy) electrical cable that gets feedback from the heating element and gets rather hot too (electrical fire anyone?)

I'm looking for a deep fryer at the moment and, while I'm not too terribly shocked because I know no one cares about their day job to actually make a quality deep fryer, I am a bit shocked at what I am finding. I decided to go on to Amazon.com to look at deep fryers. I am a bit old-school I guess because all the deep fryers I've worked with in the past reach a nice toasty 375 degrees Fahrenheit. Anything less than that will cause whatever you put into the fryer to soak up the oil. Now here's something you probably didn't know: Fried foods can actually be healthier than baked foods, but only if the oil is hot enough and recovers quickly after putting the food in. The reason why it is healthier is because it won't absorb the oil and the oil will burn off the fat during the cooking process leaving you with a nice, lean dish that is moist and tastes great. 375F is the recommended temperature, but 400F is the ideal initial temperature because the temperature will drop rapidly upon putting the food in (about 50 degrees in a matter of seconds). Anything lower than 375 will cause the food to absorb the oil and keep the fat intact, which is worse (healthwise) than if you baked whatever you were trying to fry. Additionally, vegetable oil is healthier for you than any other type of oil (e.g. peanut oil is really bad for you).

It took me looking through about 20 different fryers to finally find one without a 1-star review stating "the unit doesn't get above 325F and takes an hour to get even that high". I've never owned a fryer that didn't get up to 400F. We can blame the stupid "safety" laws for this dumb and, ironically, unsafe engineering decision. Many of the "modern" deep fryers I found on Amazon.com apparently have to be plugged in for an hour just to reach the low temperature of 325F. Most people will leave the fryer unattended during that time - great for small, curious children and the nearby hospital burn center!

Some of the fryers made now have "magnetic cords". I guess some engineer saw the Apple laptop cable and thought, "Hmm...I can do that for our new deep fryer. It will be a cool feature." The reviews on such fryers reveals that it is not only a bad design decision, it is a dangerous design decision. People end up burning themselves trying to keep the cable plugged into the unit or use electrical tape to keep it in place, which has a fairly low melting temperature - ah, I love the smell of burnt electrical tape in the morning! Great fire hazard for the whole family! Hospital burn units love this "feature" too.

Now here's a doozy: T-Fal FR7008002 Ultimate EZ Clean Pro-Fryer - check out the 1-star reviews on that one. Apparently the automatic oil filtering/cleaner feature on that unit is flawed in a serious way. Hot oil is bad enough inside any deep fryer. This deep fryer apparently leaks hot oil everywhere even during cooking. 325F oil. And this unit apparently only reaches 325F despite saying 375F. It makes a mess and melts your kitchen (I feel sorry for the one poor soul who had this shoddy device destroy their perfectly good kitchen on first use). Additionally, this amazing, disaster creating device can be yours for only $117.49. That's right folks. For only $120, you can turn your beautiful kitchen into a disaster! And likely set it on fire in the process.

Surprisingly, the cable on the deep fryer (you know, that thing connecting the fryer to the wall) has not changed in the past two decades. What is the whole point of an electrical ground on sockets if the engineer isn't going to design for it? Deep fryer electrical cords get hot enough to melt themselves. For only $49.95, you too can have an electrical fire!

The only "innovations" deep fryer technology has made in the past four decades is the grease splatter "window" (tiny glass cover), which gets so covered in grease you can't tell if it is done without pulling up the lid. The other "innovation" is the charcoal filter, which traps most of the odor produced by the frying oil. That is the good part. The downside is the filters have to be replaced and are hard to find and sometimes expensive. It might make more sense (and be a lot cheaper) to use a grease trap (wire mesh screen) and stick some regular charcoal on top of that. Just saying.

Okay, now that I've covered all the bad issues with today's "modern" deep fryer. The main issue boils down to what I mentioned before: Shoddy construction due to people not caring deeply for their job. So, let me, a software developer, tell you, Mr. Deep Fryer Engineer, how to do your job.

First, scrap your existing design because it more than likely sucks. See Amazon.com 1-star and 2-star reviews for why your design sucks.

Let's start with the base. It needs to be solid, stable, and capable of supporting 15 pounds of metal. The heating element should be at least a good three inches off the ground. The base should be made of a material capable of absorbing, drawing away, and dissipating the heat so it doesn't melt countertops. Take a serious look at computer processor and graphics processor cooling technology. They've got to keep super hot processors from frying themselves. A similar approach could be used to keep the base super cool while keeping the main reservoir hot.

Based on what I've read, the heating element needs to be capable of handling and drawing 1800W of power. Don't skimp here and use quality components. Some places are using cheap-o heating elements and then creating lame excuses in the documentation later with stuff like, "Don't use shortening with this deep fryer." Make this full-on awesome. Test the heck out of this thing to make sure it reaches temperatures in excess of 400F easily (450F would be great) and does so as quickly as possible. It should be capable of withstanding its own heat for a solid year (i.e. you leave this plugged in for a solid year and doesn't burn itself out before even considering going to market with it).

The heating element should be tied to a controllable thermostat. Dials are time-tested and proven. Digital is not. Less things to go wrong = more awesome (and less likely of a fire breaking out). The thermostat should be accurate to within a degree and stay that way over thousands of uses.

A thermometer capable of taking the heat. None of this "when the light goes out so that you can no longer tell that the unit is on in the first place" or "when the unit beeps at you through digital technology" junk. Let the user see for themselves that the temperature is correct.

The cord should basically be one of those bright orange power cords you find in the hardware store. You know the kind: Grounded, three prong cable that is a good solid 1/4" thick and can take a licking and keep on ticking. Made of material capable of withstanding the temperatures and not melting and shorting itself out...and if it does, it is grounded so you don't start an electrical fire.

The main reservoir should be made of roughly 15 pounds of steel (let's assume for frying up to 1.5 pounds at a time). If your deep fryer doesn't have 15 pounds of steel, you are skimping. 15 pounds will keep the oil hot and have quicker recovery and it has the added bonus of not going anywhere very fast (e.g. small children with the insatiable desire to land themselves in a hospital burn unit). Okay, maybe 15 pounds is a bit much and maybe not steel (Cast iron is harder to keep clean and tends to rust too easily though), but you get the idea. Heavy enough it isn't going anywhere when a small child tugs at the cable and holds enough heat to recover the oil temperature very quickly when the hot food is put in.

Grease trap flip cover. I don't have the foggiest idea why engineers use glass covers. And plastic and heat never mix (plus melted plastic in your food probably causes cancer). Metal mesh grease traps are, IMO, much more reliable and you can still see the food. This grease trap should extend and arch down instead of just barely covering the top (as most engineers who design one are prone to do). There should be a handle. Rubber would be okay but that might melt given the heat conductivity of metal and the high temperatures involved. Maybe a way to quickly snap-on attach and detach the handle (push button on the handle to lock/unlock?), but you deep fryer engineers are likely to screw that up, so just forget I mentioned it. Rely on oven mitts or really fast fingers, I guess.

Test all of this stuff for a solid year before going to market. No part should break, bend, or die on the prototype model. It should be left turned on 24/7/365 @ 375F with fresh oil several times daily. If you can pass that test, you are ready for the consumer market. Consumers are tired of stuff that breaks.

Now for a few features I want to see that you'll likely screw up.

Rapid cooling system. A spiral around the outer edge that acts like a hose (open at both ends). You put the deep fryer in one half of a sink and insert one end of a metal tube into the upper part of the spiral and hold a funnel-like end under cold running water in the other half of the sink. The two ends can be connected by some sort of flexible heat-resistant plastic. No water touches the oil and yet the container cools very quickly. The talented individual may use a large spoon to stir the oil to help it cool more evenly. Maybe create an attachment for the funnel end so that the user doesn't have to hold it but it stays under the faucet and above the upper end of the spiral (otherwise you'll be fighting gravity). The $.02 hack is a couple twist ties to tie the funnel end to the running faucet. I'm sure you can come up with something far more professional. This, however, is probably a feature that will make your brain explode trying to understand it. So, don't bother implementing it because you are likely to screw it up.

Easy oil emptying and filtration system. Some engineer already screwed this feature up (see the T-Fal link above). You can simply avoid their costly mistake by following this simple idea instead: A siphon. That's right! I'm referring to the ancient (several thousand years old), time-proven, stable concept that works! Except for one little problem: As a deep fryer engineer with numerous past mistakes and doesn't test your products, you are likely to screw this up too. The easiest solution is to provide the user with TWO of the metal, flexible plastic, funnel devices except this one will have the funnel facing the tube, have a filter, be sealed, and possibly be a lot skinnier. So, if you perfect the first feature above, you have a decent chance of getting this one right too. Cooling comes before emptying out the oil anyway, so this is a logical expansion. Don't skimp on this and do something funky with the first tube combination (e.g. a reversible funnel). Deliver two complete sets of tubes and accessories to the customer.

Once you have completed all of the above, hammered out all the bugs, and have zero complaints on Amazon.com, then and ONLY then should you consider adding more features beyond what I've listed.

Rhetorical: Why do I keep having to come up with what should be blatantly obvious? Why the heck am I designing your deep fryer for you? Isn't this your job? Then do your job already and I won't have to tell the world how lousy you do your job.

Deep Fryer without any 1-star complaints

Which is the best thing I've found so far. It took me a lot of digging around to find that one particular device. I looked at over 20 different deep fryers with 4+ star ratings on Amazon.com before finding that one. This tells me that Deep Fryer engineers don't care about their jobs and users don't bother taking the time to measure oil temperature and then wonder why everything comes out soggy (but the same people were giving it 4/5 star ratings anyway, ruining Amazon.com's rating system from the high-level perspective). It comes the closest to what I described and, according to an Internet post I found elsewhere, reaches 375F easily, thanks to the 1800W heating element. According to Amazon.com, and I'm not surprised, the digital components falter (digital = some embedded software/hardware engineers were involved = shame on you for making junky software/hardware). For hardware like this, analog is still where it is at folks.

And now, my dear programmer types who are reading this, make sure your software is quality. It has been said, "code like the next person who will maintain it is a serial killer who knows where you live." "Eat your own dog food." Use your own Deep Fryers.

Perhaps we should deep fry the Deep Fryer engineers? Into the Deep Fryer you go!
(More jokes could ensue, but I figure that is plenty).

Habits of video game designers...

2009-02-21T19:00:00.004-05:00

I always enjoy a good video game. I don't get a whole lot of time to play them but when I do, I tend to enjoy playing them. My reason for playing video games is to relax and enjoy the game. Mostly to relax though. Relieve stress. That sort of thing.

I'm going to simultaneously review three different games I've played over the past few months in regards to how they SHOULD have been developed. The three games are Bioshock, Assassin's Creed, and Crysis. There are LOTS of spoilers if you haven't played them. I tried to not ruin the final bosses though.

Which brings me to my first point: If you are designing a video game and the person playing the game selects the "Easy" difficulty, it should be nearly impossible to die. Nothing is more frustrating than waiting for a game to reload the last saved game for the 10th time just because the character died again....on Easy. I'm not the greatest gamer in the world, but I'd like to think I am better than average. However, I am the sort of player who enjoys sneaking around and getting headshots through covert means. In that department, Assassin's Creed offered me the most satisfaction. Being able to just walk around, use the from-behind-knife-stab-kill and then nonchalantly walk away into the shadows before the target hit the ground was definitely entertaining. Bioshock and Crysis offered no such options. Ironically, Crysis was a military strategy-esque game and, from my perspective, the most elite units in the U.S. forces are SEALs - and they don't run around like idiots. They do things covertly - so I figured that was the best way to play Crysis. I found that the only way to beat Crysis was to essentially run around like an idiot. Until I did that, I died on "Easy" so many times it was silly. You can't sneak around Crysis and expect to win without dying a zillion times.

Also in this vein, I play on "Easy" for several reasons: The enemies are supposedly easier to beat. It gives me plenty of time to get used to the controls. And - I get to enjoy the environments the designers created. What is the point of playing a game if you beat it in 3 hours? I like to take my time - LOTS of time. All three games took me at least 40-80 hours to beat. I like to walk up to the trees or climb to a high mountain and just pause and look around. Most gamers out there on forums are all along the lines of, "I spent $50 and beat the game in 5 hours". Not only is that an insult to game designers everywhere, it is a waste of perfectly good money.

Now onto the next issue: All the latest games seem to go from completely normal (or at least plausible) to really bizarre just so there can be a final boss. Obviously games aren't meant to be realistic, but I didn't see Assassin's Creed's ending coming. It was completely normal right up to the final boss but then all the laws of reality suddenly went out the window.

Bioshock also had an equally weird ending - the designers at least kept the whole game bizarre so you at least knew the final boss was going to be odd. But from the first sight, it was obvious to me that the main character and the final boss were in two completely different leagues (even on "Easy"), so my character shouldn't have even survived the first round of the Adam fight.

The first Crysis boss should have basically torn apart the front half of the aircraft carrier I was on or at least sank the whole thing just from its own weight - especially since the ship was already taking on water. The final Crysis boss could have just simply squashed the aircraft carrier and instantly sank it by landing on it. Sometimes I wonder if game designers do this in a meeting: "Let's create a game engine with real physics and then throw both the physics engine AND logic out the window at the end of the game........ Awesome!"

Nest issue: Level loading. I will keep saying this until someone pays attention! Game programmers - start loading the next level while playing the previous level! This, of course, requires cooperation with the designers - some sort of waypoint/trigger to begin loading the next level. If you can't do that, then please, please, please load some sort of wireframe overview of the level and load the textures for the level as you go. I refuse to accept that such as I described is impossible. Bioshock was the worst in terms of level loading times. It used to be that games loaded instantly or it was assumed something was wrong with the game and the reset button on the computer was pressed. It should be possible to load a level instantly without a splash screen. Just do it.

Next issue: Hardware adaptation. Game designers seem to absolutely love pumping as many texels into models as possible. Programmers, conversely, love putting as many of the latest graphics card features into the game (shadows, blur, HDR, etc.). Fine, but don't make me guess which feature/setting is causing my framerate to drop to 1 fps. Bioshock and Assassin's Creed were fine - in fact, my hardware was capable of running the game smoothly at maximum settings and did not require restarting the game to get them (and I was also able to run them both at desktop resolution). Crysis, on the other hand, basically required restarting the game whenever I tweaked the settings, which I had to do several times. For the final aircraft carrier deck scenes, I had to turn several options to Low just to be able to play the game - looked really bad too. I now understand the Crysis supercomputer "jokes" I've read plenty about on the Internet. The Crysis game engine leaves something to be desired in terms of performance (poorly written). I was able to, however, play almost the entire game with the default settings - right up to the aircraft carrier deck scenes. However, even that was insufficient at times (occasional lag). Most users won't even understand most of the settings in the settings panel any more (they have weird, technical names) and even fewer understand which features/settings will affect performance the most because each game is different.

So here is a great solution to this problem: Create an frames per second range selection and automatically adapt the quality settings accordingly. So, let's say I'm willing to accept framerates from 15fps to 30fps under non-battle conditions and 23fps to 50fps under battle conditions. Basically, this means that the game should track the frames per second at all times and whether or not the main character is under attack or about to be under attack. It would also be extremely useful for the game engine to know how difficult processing the scene through the hardware will be. What this should boil down to is a threshold system. If the framerate falls below or rises above specifically set conditions by the user, automatically adapt the game settings starting with the most expensive settings and working down to least expensive. Crysis, however, didn't handle setting changes very well (caused weird flickering to take place requiring a game restart), so care obviously needs to be taken in the programming (the other two games seemed to handle most changes just fine). Obviously, quality setting changes will cause most games to pause...BUT it would be very useful if such changes could be phased in (like the level pre-loading idea). Unless, of course, the game is lagging so badly that the player is getting less than 5fps - then pausing while drastically altering the settings would be better instead of staggering the changes.

The user should also be able to configure how long the framerate has to be outside the range before actually doing anything. In my experience, noticeable framerate drops typically only last a brief time (a half second or so and then the game continues for another half hour without problems - no point in altering settings at that point just to have them altered again a few seconds later).

The user should also be able to configure the settings thresholds for online play as well. Single-player lag is fine by me but multiplayer lag is not good at all. I don't play online very much though. Too many 12 year olds with foul mouths if you know what I mean.

Next issue: Stop with the NVidia stuff. I'm sorry, but I've used enough NVidia hardware to know that I can generally get a better video card for the same price with ATI. While NVidia is generally ahead of the curve, ATI cards are cheaper for the same performance as NVidia. Used to be a NVidia fanboy too. So, the switch to ATI was basically an ego-bruiser but way easier on the bank account and I was able to afford decent video cards. NVidia hardware runs hotter than ATI hardware too. All three games were pushing the NVidia brand for no particularly good reason (I doubt Crysis really runs well on ANY hardware where you spent less than $8,000 USD - I built my own system for $1,500 and Crysis is the only modern game I can't max. the settings out on). Also, the drivers for NVidia are written by programmers who clearly don't know much about authoring software - what good is awesome hardware if the software (driver) is causing the OS to BSOD every time you go to play a game? Never had any stability issues with ATI - meaning they have better programmers and QA personnel who know how to author a stable software driver.

Next issue: Foul language. Bioshock and Crysis (especially the latter) have foul language throughout the game. Assassin's Creed was pleasant in this regard. Users should have the option to turn off the foul language and multiple voice recordings should also exist. I'm tired of foul language being in everything ranging from games to movies to people. The English language is diverse, rich, and powerful enough to avoid dropping four letter words every few minutes. Show some class when designing games. Thank you.

Next issue: Game startup screens that I can't skip. All three games had them. Bioshock and Assassin's Creed were the worst. I understand companies want to have their logos displayed. Fine. Display your logos the first time I fire up the game. You can even force me to look at them the first time. But after that, let me press ESC to jump to the main menu. If you absolutely have to show me logos, why not display them right at the beginning of the game itself? I'm not talking about the intro. screen, I'm talking about an overlay of sorts in-game at the beginning.

Next issue: Vehicle controls. This is pretty much a Crysis-specific thing but both Bioshock and Assassin's Creed had "vehicles" too. When you are in a lumbering, hard-to-control vehicle, it helps to know what in the world you are stuck on. The specific vehicle I'm thinking of is the tank in Crysis. I died so many times because I got stuck on a little bush it wasn't funny; it was incredibly frustrating (it was a tank, for crying out loud - tanks can flatten pretty much anything - sheesh). Going third-person in a tank would be silly though - third person worked well for Assassin's Creed, wouldn't work for a tank in Crysis since there is a battle still going on around you. I should have been able to press a key (say, Ctrl) and be able to quickly glance around without having to rotate the main weapon.

Next issue: Equal distance from start to finish for each level. Game designers seem to put a ton of effort into the first few levels, make them highly polished, extensive, and creative but then give up on later levels. Occasionally I get stuck in a game and go view a walkthrough. I don't immediately view walkthroughs - I only do that after dying 30 times and trying every strategy I can think of. Usually the solution is obvious. What I typically discover is that I am close to 1/3rd of the way through the game. I usually think, "Wow, this is a pretty long game if I'm only 1/3rd of the way there." I usually stop after using a walkthrough just to give my brain time to think about strategy so I won't have to use the walkthrough again. Assassin's Creed was the most balanced of the three games - but only because of the issue I will cover last. Each level was of roughly the same length in terms of time taken to complete. Crysis was the worst - I literally finished the last two-thirds of the game in 8 hours - a fraction of the time it took to complete the first third.

Next issue: Hurrying the plot along or not providing a good enough "memory". I've said before I like to take my time in games. Crysis was the worst in hurrying the plot along and Bioshock was the worst in regards to knowing where to go next for the plot to continue. There was always someone in Crysis yelling, "Come on, we don't have all day." There were times in Bioshock where I was supposed to find things without a directional arrow (which was fine, but did get tedious wandering everywhere). There were also times in Crysis where the HUD stopped showing where to go next, but it was rare. If I were to stop playing during those times and returned a week later...would I remember where I left off? Assassin's Creed's plot was fairly obvious but dull.

Next issue: Linearity. This has been repeated so many times by so many people, it is ridiculous. There is only one game ending to almost every game made. Replay value of one-game-ending games = zero. The only reason I replayed Half-Life 2: Episode Two a while back was to get the Garden Gnome Achievement. I got it but it was the most hideously difficult achievement ever.

Supposedly one of the absolute best games ever is a game called Chrono Trigger (unfortunately not for PC). The graphics were bad, the gameplay repetitive, and the story was not the greatest (but was also heart rending in some spots), but what made it special was that there were 13, yes thirteen!, different endings. You could choose to meet the final boss at basically any point of the game. For the singular reason of 13 different endings to the game is why it shows up on "Top 10" games-of-all-time lists...every...single...time. There was supposedly only one ultimate path to the game though. I'd love to play a first-person shooter where you get to make several choices and each choice has consequences later on in terms of what levels you get to play, people you meet, stuff that gets said, and the number of endings that can happen. Supposedly Mass Effect is like that but limited and still supposedly ends up having just one ending.

Next issue: Achievements should be possible to get without having to pull hair wondering what you missed (particularly for the "find these 50+ things" hunts). I mentioned HL2: Episode Two, but Assassin's Creed also had an achievement system as well. I like the concept just because they are sheer fun to get when you get them (usually I have some sly smile on my face) but you should be able to select the achievement(s) you will be attempting to get - that way you don't miss them - perhaps directional arrows/reminders. It would take the fun out of some achievements by doing that, but you could collect all the flags in Assassin's Creed much more easily. Or know that the garden gnome fell out of the car for the millionth time (which begs the question - surely the designers KNEW of that issue and could have designed a working trunk).

Next issue: Limited weapons. This is related to the final issue but if you are a gamer, you know exactly what I'm talking about. Assassin's Creed was actually pretty good about weapon selection, but some Greek Fire (or a bit of flint) could have been nice for a "create a diversion" Challenge.

Bioshock came in second place. The Plasmids were interesting but expensive and I ended up setting everything on fire most of the time since ice didn't drop items and the rest were lame. I ended up just shooting Big Daddies with the RPGs and didn't really care for killing the little sisters (even though I could have gotten more Adam for it). Most gamers tend to settle into a groove once we know some specific approach works well. Upgrading weapons was useful but not many choices were given (the game brochure prides itself in how you can custom upgrade and make enemies do things they wouldn't do otherwise). Your average Joe-schmoe RPG game has more weapon customization options than Bioshock. The camera "weapon" was an inconvenience. In order to get some Plasmids and upgrades, you had to take pictures - and the best time to take them was in the middle of a firefight.

But Crysis was the worst. Ironically, you could hold nearly unlimited ammo but only two weapons. The RPGs were even more annoying - why you couldn't carry more ammo was silly. You also had to bend over and pick up all the ammo. Kind of a nice touch at the beginning, but got tedious really fast. There should have been a "Maximum Magnetism" mode. The main character was basically made of metal. I only used grenades one time in the entire game. I find grenades annoying unless they are of the explode-on-contact-when-fired-from-a-projectile-weapon type. Grenades are usually lobbed around corners in real life with a known threat inside - a feat that is VERY difficult to accomplish in games (especially since grenades in games are never as powerful as they should be). The general solution most games use is to face the enemy, throw the grenade, get out of the way, and then hope you get lucky. During which time you could have already killed them and moved on.

Even more annoying was the TAC cannon. I was pretty annoyed with the game by the time I reached the TAC cannon since my weapons had been taken away from me for the second or third time (I like to build up an arsenal when ammo for each one is limited - taking all my weapons and ammo away annoys me). Then I got somewhat excited by the prospect of being able to fire nukes from a cannon. Then I discovered you could only fire nukes at an enemy that previously got more powerful when a nuke was fired at it, absorbing the nuclear energy. That was a pretty serious plot hole there BUT I was willing to overlook that as long as I could fire nukes at anything. It turned out that I could only fire it at the final boss, which, according to the plot hole, should have been able to absorb the energy of the nuke. Plus the final boss was ridiculously easy. I didn't mind too much about that though. It looks like the end of the game was given significantly less creative license and there are glaring storyline oddities/inconsistencies/plot holes that make it seem like it went through multiple iterations without proper quality assurance and beta testing (e.g. the island "shield" on the display expands BEFORE the nuke hits and then does the same expansion animation AFTER the nuke hits. Also, the Gauss cannon is an energy weapon - which should have had the effect of making the enemy stronger).

Final issue: Repetitiveness. I tire quickly of games that become overly repetitive. Bioshock was the worst in this regard. Close to the end of the game I swore that if I heard "Welcome to the Circus of Valuuuuuuuuuues" one more time I was going to find and put my fist through the person's face who vocalized that for the vending machines found liberally throughout the game. Actually, I was ready to do that after the second time I heard it. The ONLY interesting vending machine in the game was the one that delivered a live grenade at your feet for free (can't beat that price tag!). That one was fun. Not as fun as throwing bunnies into a spinning fan on a vehicle destined for a cliff dive, but still fun. (Can you name the game I just referenced in that last sentence?) It is the little things that make a "good" game "great".

Crysis was repetitive in that it was shoot bad guys, wait for health to replenish itself, pick up ammo off the ground, repeat. Or, die, try something else, repeat. The only "cool" weapon in the game was the Gauss cannon and the occasional RPG. Everything else got boring really fast. The vocals in the game, "Maiximum [whatever]" were cool but equally repetitive. I left it on "Maximum Armor" most of the time since it was a pain to change the settings. Every reviewer of Crysis seems to complain about the "temple's" lack of gravity making the controls harder to use. Maybe on consoles but the PC was nice and smooth AND offered relief in an otherwise dull set of repetitions. The vehicles, on the other hand, were impossible to control on the PC and I got very tired of my tank blowing up for the stupidest reasons. The ground-to-air fights were also fun breaks in the monotony (although at the end of the game, everything flew). But, for a first-person shooter, it was better than most games of the genre. I liked the bosses at the end but bosses are really what keeps games interesting because otherwise it is the same thing over and over again. So there should have been some mini bosses to help balance things out. I did like the changing terrain and day/night cycles though.

Assassin's Creed was the second worst in terms of repetitiveness. I got VERY tired of climbing buildings to each and every lookout location and saving citizens by starting massive, bloody battles (some of those battles took a while and, when I was done, I'd look around and people were running around screaming at the massacre of at least 25 corpses - which should have likely drawn the ire of even those people). Then I'd have to run off to the next "save the citizen" battle. I also wanted to find the people who created that annoying woman, the one who would constantly get in the way saying, "Please sir, just a little coin. I'm hungry and starving..."; find the programmer, designer, and voice actor responsible for that and put my fist through their faces. I couldn't shove her out of the way, I didn't have any money, and killing her reduced my health and created a mess because she was a "civilian". The things I most enjoyed in that game were the various Challenges - in particular the Silent Assassin and Merchant Challenges. They offered the necessary diversion from an otherwise repetitive game. I did like climbing on the walls too, but that got tedious when it involved the eagle nests and the scenery didn't really change all that much. I would have liked a bit more variety - such as a "find the missing brick(s) needed to reach this birds nest" Challenge or a "Where's Waldo" Challenge or a "follow this person" Challenge or a "this Challenge is the basis for unlocking another Challenge [or two]" Challenge. The eagle eyes concept was cool but WAY underutilized. I used it once at the beginning and once at the end but otherwise didn't need it nor found it very useful. I didn't really care for the end of each level. I simply secretly stabbed the guy and then ended up having to run around each level for a while. One level I literally was running around non-stop for a good hour-and-a-half before I was finally able to stop. The only thing good about Assassin's Creed's repetitiveness is that each level was roughly the same length and thus took about the same amount of time to complete. An assassin is supposed to blend in (more or less). The main character stuck out like a sore thumb in his outfit. It would have been interesting to have participated in the evils consuming the land prior to carrying out the assignment. Get to know each target on a personal level by working with them, thus making it harder to complete each assignment but easier to gain the trust of the target beforehand. Lots of possibilities this game did not explore. The absolute ending leaves you hanging, though, so perhaps Assassin's Creed II will correct these problems and create an immersive and less-redundant experience. The problem with a sequel is there is basically no room for a clever past life like this game was. A past life experience will be difficult to pull off convincingly. More than likely, the present will be focused on (and probably do more with the eagle eyes), which will likely be riddled with plot holes. Surprisingly, right up to the final boss, there were few to no plot holes in this game (with the sole exception of the beginning where you come back to life without ever receiving a solid explanation). Of the three games, this had the best, most complete game storyline I've played through in a really long time.

If you are a game designer who worked on any of these games, you should know that I still enjoyed the games but they could have been a LOT better. Graphics are good in all the games but good plot consistency, a solid, emotional storyline, immersiveness, and not annoying gamers (especially PC gamers) should also be at the top of the list.

Consider adding me to your external beta testing team for your next game. I've played thousands of PC games ranging from Sopwith and Striker to Monkey Island I-IV to Total Annihilation (STILL the best RTS game ever) to Narbacular Drop (Portal's predecessor, Portal was made by the same indie team) to the latest and greatest. I'm also a software developer and have developed my own games (mostly for ancient handheld devices) - so I know how difficult making a video game actually is.

Too Many LEDs

2009-01-03T04:00:00.004-05:00

I was having a conversation yesterday with someone and we somehow managed to get onto the topic of LEDs and how there are too many of them. I wholeheartedly agreed because I've been getting annoyed lately at how bright everything is when the lights are turned off at night.

This morning I walked around the place before I turned anything on. I could see my way clearly everywhere. Whatever happened to all the fun we used to have with being able to stub our toes on something in the dark? Light Emitting Diodes. Pff. Whatever.

I counted them before turning anything on: 47 LEDs all either steadily on or going blinkety-blink in the dark. No one is looking at them. In terms of brightness at night, the bright blue and bright white LEDs are the worst (i.e. they output too much light). Green falls behind at a close second and red is last. I'm sure there are other colors and maybe different perceptions of how bright they actually are but all this light is wholly unnecessary. Especially since this wastes continuous electricity (47 * 30 to 60 mW = 1.4 to 2.8 Watts of continuous draw - not a whole lot, but it does add up) AND I'm pretty sure they also disturb sleep. Yup, I wouldn't be surprised to find some study that LEDs are the single greatest cause of sleep deprivation.

What are LEDs on all our devices good for anyway? One-glance diagnostic purposes. Now let me ask all you users: How often do you actually need to look at a device's diagnostic lights? Probably a few times per day for a phone but almost never for most other devices. The worst offenders - in order of offenses: My USB thumbdrive (a steady bright-blue fade in, fade out), Qwest DSL modem (I recently switched away from Comcast - a story for another time), a D-Link router, VOIP box, and the Ethernet connection in the back of the computer.

It really isn't hard to fix this problem: Add a button on the device that turns on the lights for a few seconds and then they - here's a shocker - turn off again. What a brilliant idea! Too bad I didn't think of it first: Wristwatches do this already. Engineers: Go steal, er, "borrow" the necessary technology from those who make wristwatches.

I swear Internet-connected components are the worst in terms of the quantity and annoyingness of their use of LEDs. Internet component engineers use the brightest LEDs possible and then make them blink incessantly. When was the last time you actually needed to look at the LEDs on those devices? Exactly. You can't remember. Neither can I.

Why Unicode stinks

2009-01-01T13:33:00.007-05:00

Happy New Year! Cliché, but awesome nonetheless. I hereby resolve to blog more. Nah. Scratch that. Who keeps their New Year's resolutions anyway? My family recently discovered that my grandmother, for instance, had a diary that she started every January in the early 1940's and the farthest she ever got was June (about mid-1940's - after that it looks like she permanently gave up).

Anyway, onto the actual topic of discussion. What follows is a summary - it isn't accurate, but that won't really matter too much.

A byte, for all intents and purposes of this discussion, is 8 "bits". A "bit" is a 0 or a 1. 8 bits offers 256 combinations (2^8 = 256). The smallest logical unit that can be used in a computer program is a byte.

A computer screen is made up of a whole bunch of pixels, typically arranged in a 4:3 ratio (e.g. 800x600).

With that knowledge and assuming you were designing a computer, how would you display a letter of the alphabet to the user?

The first computer engineers didn't think very far ahead. In fact, I'm pretty sure they were fairly brain-dead when they created what is now known as the ASCII character set. The ASCII character set was supposed to be temporary until something better came along. Guess what? ASCII is unfortunately still in extensive use. English is easily represented in 256 characters. Many Latin-based languages are as well.

Instead of fixing the problem, it was exacerbated when some idiot came up with the concept of "code pages". As I just said, most Latin-based languages fit comfortably in 256 characters. The problem arises is that there are more than 256 characters in all Latin languages for all the characters that need to be represented (or something like that). ASCII is actually two parts - the Basic character set and the Extended character set. The first 128 characters are part of the former and the latter 128 characters are in the latter. So, when you booted up the computer, you would load the code page for your language, which would display different characters.

Why was this a problem? Well, let's say you wrote a document using the Spanish code page and wrote an English document and sent it to someone who uses the English code page. When the person opens the document, anything outside of the Basic character set would appear as jibberish. To read (and edit!) the document properly they would have to first load the Spanish code page.

After about a dozen code pages had been cooked up for various languages, someone finally got the idea to create multibyte character sets (MBCS). A MBCS requires more than one byte to represent a single character. The first MBCS implementations were poorly designed and got a bad rap for being difficult to manage and hard to program for. However, for non-Latin (in particular, Eastern) languages, MBCS was the only way to actually encode characters.

After many years of struggling, some people formed a committee and Unicode was born. Unicode was supposed to be the magic bullet for all problems ever involving the representation of all known languages. I say "supposed" because Unicode stinks.

For instance, you can't represent Klingon in Unicode. This is a silly case, but what happens when we encounter intelligent alien life and we want to send them a document? Unicode isn't cut out for that. Or what about the zillion dead languages no one uses that exist? Or how about the characters of the Chinese language that can't be included because they aren't "common enough"?

And don't get me started on programming Unicode stuff. Writing programs that don't chop up a grapheme cluster (I'm still trying to wrap my head around that one) into little pieces is a nightmare. There are about a dozen different Unicode Transformation Formats (UTF) and strings may or may not start with a Byte Order Mark (BOM). Joel Spolsky, while I respect most of his views, is just plain wrong about Unicode.

Of everything I've seen in Unicode only one positive stands out. UTF-8. UTF-8 is on the right track but still misses the mark by a long shot. It is still difficult to program for as it is a MBCS (but then so is the rest of Unicode if you don't want to destroy grapheme clusters) but it uses single byte characters - no big/little-endian mess - and that makes it a zillion times better than any other incarnation of Unicode. But even UTF-8 is still limited in the number of characters it can represent.

Here is what I want to see: Unlimited character set definitions. So if I want to display Klingon on my computer screen, I can. And the implementation is blindingly simple:

Bit 7 - Grapheme cluster
Bit 6 - Continuation
Bits 0-5 - Type and/or Data

That's a byte broken down into bit groups. All a programmer will be interested in is Bit 7. If Bit 7 is a 1, they move on to the next byte(s) until they find a byte with Bit 7 set to 0. That is a complete grapheme cluster. A grapheme cluster is a complete character - the boundaries of which you can slice and dice to your hearts content. You could, for instance, copy and paste or concatenate two of these or delete it entirely from a string. LOTS of possibilities. With Unicode, finding the boundaries of a grapheme cluster in a string of bytes is complex.

Bits 6 is a little more complex to understand. Basically, it indicates if the current byte is a continuation of the previous byte (and type) or if it is a new type.

0 = New Type
1 = Continue current Type

For the Type, there are two bits means there are four possibilities for values. I'd like to see these as such:

0 (00) = Code Point
1 (01) = World
2 (10) = Metadata
3 (11) = Custom

The Type is an implementation-specific view. A grapheme cluster without a World type is assumed to be the current World default (e.g. Unicode). A Code Point is implementation defined (e.g. a Unicode Code Point). The Metadata type is implementation-defined metadata that gets applied to the grapheme cluster as a whole or the following Code Point. The Custom type will probably never get used but it is implementation-dependent. It could be useful as an application callback to display some custom character that is very specific to the application.

The remaining 5 bits are used for data. Need more space for data? Well, then continue with the next byte. An example is in order. Let's say Unicode is World 0 and Klingon is World 1 and Code Points 154 and 22500 represent a single grapheme. This could be encoded as such:

10010001 (Select the Klingon World)
10001010 (Last 4 bits of Code Point 154)
11001001 (First 4 bits of Code Point 154)
10000100 (Last 4 bits of Code Point 22500)
11111110 (Previous 6 bits of Code Point 22500)
01010101 (First 5 bits of Code Point 22500)

This is one possible encoding. Code Point encoding is entirely up to the World designers.

This approach allows the Unicode World to coincide alongside any other World. From my perspective, programming becomes really easy. I don't have to care about the actual data I'm manipulating. Here's an example of what I mean:

size_t my_strlen(const unsigned char *str)
{
size_t x = 0;

for (; *str; str++)
{
if (!(*str & 0x80)) x++;
}

return x;
}

For those who don't read C code, strlen() determines the length of a string of characters. The code above does the same but on a grapheme cluster level using the definitions I've provided. Doing the same thing in Unicode is about 50 times more complex.

Let's say, for instance, you want to convert a grapheme to "lowercase". When you start messing with anything lower than the grapheme level (as a programmer), the waters become really muddy real fast. Converting a string to lowercase is not clear-cut and has all sorts of unintended consequences. For this reason, World designers MUST declare all transformations in clear-cut, dead-simple transformation sequences. Such transformations, for instance, could come in the form of a Standard set of APIs or libraries that are required to be used for conformance.

The approach above is superior to UTF-8. It uses a little more storage space to represent the same characters, but let me introduce you to this wonderful thing called "compression". My recommended approach will still compress quite nicely and you can do things with this you will never be able do in Unicode. But, as I've worked with Unicode, I've come to the realization that it doesn't matter how many bytes there are as long as the text gets displayed properly in the end.

Just a thought. It may turn out that this is a bad idea too. 20/20 hindsight is funny like that.

A Real "Green" Computer

2008-09-08T21:50:00.006-04:00

I hate the phrase "green computing". Why? Because the more stuff we have, the less environmentally friendly we actually are. In other words, if you can afford a computer, you ain't green. That would mean you sitting in front of the computer screen reading this. Yes, go ahead and squirm.

However, that doesn't mean I shouldn't be aware of the world around me and therefore try to be responsible with my resources. And with those resources, I realize that if I can cut costs drastically, that would be a huge help to my fellow man.

So, I have one question for you: How many Watts does your computer draw from the wall?

Don't know? Neither do I. And I leave my computer on all day and night. There are important things my computer has to do every single day (besides filtering through thousands of spam messages) that requires it to be powered up and running. Of course, this usage comes at another cost - the room the computer sits in has to be within a tolerable temperature range. That means AC in the summer and heat in the winter (despite the computer generating plenty of its own heat).

Most people don't do much beyond the four basic tasks: Read e-mail, surf the web, IM, and word process. (And maybe listen to music on itunes). Basically, any ol' computer will suffice just fine. My needs are even simpler - just about anything that connects to a standard Ethernet cable will work.

So, I've been on an extensive quest for "green". Or, let me put it in terms anyone can comprehend: Something that will significantly lower my monthly bills. I'd wager about 1/3 to 1/2 of my electric bill is solely because I leave my computer on all day.

Now, if you examine the average electric bill, you find that the electric company measures things in something called Watts. Or, more specifically, Kilowatt-Hours.

So, the obvious solution to lowering one's electric bill in this particular scenario I've set up is to look for a computer that has almost no power utilization at peak usage. There are some pretty amazing things being done with computer systems these days, almost none of which are power-friendly.

Over the past year I've looked on-and-off for a solution to my problem. My preference is a headless system (i.e. no monitor) as I've got no problem with remote connecting into the system. That pretty much takes laptops out of the equation (but not entirely). We (the computer industry) keep coming up with faster and faster components, but what we need to do is step back and go slower with the goal of less power. We can build fast components that use a lot of power, but can we build a slightly slower component that uses no power? There's the challenge for the next decade. Google would be a huge customer of components that used negligent amounts of power. Low power typically = low heat. Low heat = additional cost savings.

Recently Dell, in particular (but other companies are following), has pushed the whole "green computing" thing, but the question is: Do the Dell systems that are being put together actually save you money? That is, how many Watts do they draw from the wall?

Below is a small list of a half-dozen boxes that I've kept track of for various measuring purposes to determine where we are in terms of power draw overall.

First up are the gaming consoles. I'll get to the 'why' in a bit. That is, "Why are you telling us about the power draw of gaming consoles? I thought this was about PCs?" It is and isn't. I'm after the most efficient hardware for my needs. If it happens to be a PC, great, but if you truly want to be power efficient, you have to be open to all the alternatives.

The XBox 360 is a fairly popular gaming console. Unfortunately, it weighs in at a 160 Watt power draw. Over twice the power draw of the original XBox (74 Watts). (And the XBox 360 Elite weighs in at 194 Watts).

The PlayStation 3 is another popular gaming console. I've found conflicting numbers as to how many Watts it draws. One site says 171 Watts, another says 380 Watts of power (peak). I'm guess that 171 Watts is an average draw.

Next up is my reason for even looking at gaming consoles. The Wii. Get ready and brace yourself...

...A paltry 14 to 17 Watts, depending on who you ask. I assume the 17 Watts is peak and the average is 14 Watts. Supposedly it also consumes 11 Watts if the Internet connection thing is left active while the unit is in power save mode and a mere 2 Watts if that feature is turned off while powered off. 14 Watts is definitely not bad. And, when you consider that WiiLi is being worked on (Linux for the Wii), that is not a bad platform.

The weird thing is that people love the Wii and think XBox and PS3 are so-so. In other words, less is more and more is less. If you can play fun games on 14 Watts of power, then that is sufficient to run a lightweight application over an Internet connection. Nintendo is onto something here that could be adapted to the computer industry.

So, those are the baselines. There is a huge rift between the two types of gaming consoles. Price is also a factor - the Wii is cheaper, especially when you consider the cost of plugging it into the wall and keeping the area's temperature regulated with AC if used in the summer.

Now, let's stack up regular computer systems. This is actually a LOT harder. No one publishes metrics on power draw. And that is not done probably because we don't want to be embarrassed by how much power we actually use (abuse?).

First up is Dell's own brand new "Hybrid". This computer is claimed to be "environmentally friendly" - a "green computer". I talked with a Dell sales rep. about this and it took them about 15 minutes to come back with the answer: 350 Watts. I'm going to assume that the number is "peak wattage" and not "average wattage". Let's assume that the number quoted to me is similar to the PS3 peak and we can guesstimate that the average is more like 160 Watts.

Since no one else is very forthcoming on average and peak wattages, I'm forced to skip the other companies like HP and move on to the next best thing, Apple. In this case, I've had my eye on the Mac Mini for several years. This little gem weighs in at an average of 20 Watts and a peak of 50 Watts. Not bad considering you get OSX with it (i.e. a decent OS instead of having to hack something together as I would with the Wii - a fair tradeoff).

For the Mac Mini, someone kindly measured this with a very nifty little tool called "Kill-A-Watt" that measures power draw. Given how easy and affordable these are (from the computer manufacturer's perspective), they should be required to run all computer system configurations through it. I've been thinking about getting one but it won't likely change my habits very much. It might also move me more quickly to buying a new computer. Then I think, "Just what I need...one more computer!"

Finally, I came across perhaps the ultimate goal in terms of achieving nil energy usage: OLPC. Also known as One Laptop Per Child or the 'XO' (er, 'hug and kiss'?). It comes in puce gr...I mean...ugly green. Whatever you know it by these days, it consumes a mere 2 to 3 Watts of power. XO-2 (i.e. version 2.0) supposedly will only use 1 Watt of power. It blows the Wii out of the water in terms of power usage but is apparently quite sluggish. But I don't need a powerful computer.

It is also interesting to note that the OS you use also causes a power draw. A couple years ago I experimented heavily with VMWare Player. Ironically, the free VMWare Player actually has a feature that the full version product does not: A CPU measuring device. I've suspected for several years certain applications ([cough] Visual Studio [cough]) and OSes draw more CPU than is reported by the OS. VMWare Player reveals their usage completely. I was on my Linux rampage at the time and tried numerous OSes, I also tried the beta of Vista and a few Windows OSes I had lying around. Windows 98 was actually more CPU-friendly than any other Windows OS (i.e. Microsoft lied during the XP install, but no one should be surprised, the company always lies). The public Vista beta kept the CPU indicator incredibly busy even though absolutely nothing was going on in the system that I could tell (not even moving the mouse and no applications running). In the Linux realm, all but one of the OSes behaved identically - even the much touted "Xubuntu" as being "performance friendly" actually behaved the same as all other distros. The only OS that used absolutely zero CPU while sitting idle was: Damn Small Linux (DSL). I had to make a serious effort to get the CPU indicator to budge. If you want an OS that is power-draw friendly that also has some semblance of usability (i.e. a GUI and boots blindingly fast), DSL is it. There is no other OS worth attempting to use that has zero power draw on the CPU (assuming your goal is zero CPU power draw on idle).

In conclusion: Let's suppose you want some power, need some user-friendliness, but don't want a power-hungry computer. The Mac Mini is, to date, the best choice from a "balanced" perspective. You get a well-recognized, easy-to-use OS (OSX) that boots up fairly quickly, is easy to update/patch/install new software, and has enough 'oomph' for most people while being power-draw friendly. Not quite at the Wii or OLPC level but definitely a good start in the right direction.

This angst has been brought to you by someone who got fed up with Dell advertising their PCs as being environmentally friendly. Lies.

What to do when a computer arrives.

2008-07-12T20:49:00.004-04:00

This post is also known as "Your computer could be a chemistry experiment waiting to happen".

In my previous post on how to buy a computer, I briefly covered what to do when you receive a new computer but focused much more heavily on the topic of how to actually go about buying a computer. In this post, I will cover what to do before you use the computer for the first time to make sure what was built and sent to you is actually a quality product. You spent good money, let's make sure that no one cut corners where they shouldn't have and that all the parts are indeed correct.

This first part requires you to be static free. First, pull the computer out of the box. Now go touch a doorknob while grounded. Now, read the computer manual for opening up the box and follow the directions. Usually thumbscrews or screwless entry is used these days making it easy to open it up. Yes, it is scary, but opening the box won't void the warranty or service contract. The computer won't bite. Promise. Besides, in many areas of the world, the computer will likely have been sitting in an extreme temperature. Opening the box up will allow it to reach room temperature much more quickly.

Once the box is opened, check for condensation. If you see condensation, don't plug the computer into the wall for several days. Water and electronics don't mix.

The first thing to do is to take out one of the sticks of RAM and look at the pins. Gold should only touch gold connectors. Tin should only touch tin connectors. If the manufacturer mixed the two metals, you will get very slow corrosion that eventually introduces data errors, application crashes, and eventually blue screens. Like metals should only touch like metals. I picked this odd tidbit of information many years ago when I was building a system and the motherboard specifications said "gold pins" for the RAM. I did some Google searching and apparently manufacturers don't pay attention to this. If your manufacturer mixed metals, they are obligated to provide replacement RAM. Return the RAM to its cradle.

Next, go over and make sure all connectors are firmly in place. This includes all expansion cards.

Look at the direction of the CPU fan and where the graphics card is (if any). If the CPU fan is going to be blowing air onto the graphics card, then the computer is poorly built and will likely cause the video card to lock up. Send the computer back to the manufacturer or look for an alternate way to keep the system cool (e.g. liquid CPU cooler).

Check to make sure all hard drives are well-ventilated. Hard drives that get too hot have a reduced lifespan.

Make sure cables are out of the way of the main cooling system.

Check the CPU for anything suspicious. By this, I mean a loose heatsink, nearby wires that could melt, etc. CPUs and GPUs run hot. Most are in factory-sealed containers, but if they are open, it should be easy enough to tell. Most major manufacturers are usually pretty good about case design.

Make sure the motherboard is properly seated. This is harder to tell, but it should be firmly in place. If not, and you feel brave enough, take a screwdriver and tighten down the screws.

Search the Internet for each card and component on a separate computer. Make sure they are without fault. Be sure to check forums to see what people are saying about each item. Most items are OEM instead of retail. Cheaper, but sometimes lower quality with a higher fault rate. If possible, also try to determine the brand and model of the motherboard. Download the motherboard specifications from the manufacturer's website and make sure images match up. Figure out who exactly manufactured your video card (remember the GPU and the card itself are manufactured separately) so you can head off any trouble in advance.

Also, keep in mind that there is such a thing as "pirated hardware". Only a trained eye can spot the difference but if you experience problems with a particular component and replace it and the official component works as expected, perhaps you were using pirated hardware (even lower quality and very hard to detect).

Close up the box. Follow the directions to set up the computer.

Now reinstall the OS. If the manufacturer did not send you an OS reinstall disk, call them up and tell them to send one. Manufacturers are obligated to provide you one backup disk if you ask for it. If you have never installed an OS before, it is somewhat scary. Microsoft Windows is actually wizard/GUI-based and fairly automated once you get past the scary text-based interface. Reinstalling the OS removes all the junk the manufacturer installed that would significantly slow down a perfectly fine PC. Reinstalling the OS is usually fairly simple - turn on the computer, put the CD/DVD in the DVD drive, turn off the computer (hold the button for 5 seconds - don't worry, this won't hurt anything since you're going to wipe the drive anyway - it just forcefully turns off the computer), and then turn on the computer again. You may need to reboot again and press F2 or F12 or something similar to enter the system BIOS to change the boot order so that the CD/DVD drive is looked at first in the boot process instead of the hard drive.

After reinstalling the OS, you will note that it likely looks "ugly". Now you need to reinstall missing drivers. Typical drivers include audio, video, and display drivers and any drivers for third-party components (e.g. a digital camera, printer). I recommend installing the video driver first, the display driver if the monitor(s) is not recognized, and then the audio driver. If you have dedicated video and audio cards, go the respective manufacturer websites and download just the drivers for those devices (e.g. ATI has a big ol' GUI interface download and a drivers-only download - you really only need the drivers). I've found that after installing updated video drivers, it takes a couple reboots to "fix". Repeat the procedure for any third-party components you wish to use.

Depending on the OS and patches installed (e.g. Service Packs), you may need to install the latest updates for the OS before you install a specific driver. Pay close attention to what is the minimal needed.

Once you are done installing drivers, go to the OS update site and install all updates for the OS. This may require repeating several times and several reboots until all updates are installed. You should be connected through a router to the Internet. A Windows-based computer (especially an unpatched computer) not behind a hardware firewall will be compromised in under a minute on the big-bad Internet. The only way to clean a compromised system is to reinstall the OS. You have been warned! Get a good consumer router and put it between your computer and your cable/DSL line.

Once you have reinstalled the OS, the drivers, and it has been fully patched, reboot one last time. Now, install all the applications you want to use.

Once all the applications have been installed, check for updates to the OS and then each application. Repeat until the entire system is fully updated and set up.

Reboot one last time. Now you are ready to use the computer.

That is a painful process, but one you won't need to repeat for many years to come. This should produce a quality experience even for the cheapest of computer systems.

Every once in a while (once a year), you should open up your computer and take some compressed air for computers and, with the computer turned off, blow out most of the dust buildup. Also, look for the button battery on the motherboard and make sure it isn't corroding (don't remove it to check! It keeps the system time alive!). Remove a RAM chip to make sure the pins aren't corroding and then put it back into the slot. Make sure nothing looks "burnt". Close the box. You should backup the data on your hard drive (burn it to DVD or copy it to an external USB hard drive or use a secure online storage service or...LOTS of options). After backing up your data, do a defrag.

How to buy a computer

2008-07-12T16:29:00.008-04:00

This blog entry isn't technically about software development, but every once in a while, we all need a shiny new computer. With all the latest overhyped buzz about the economy and gas prices (I've heard predictions that gas could easily be $10/gallon next summer - there is no "ceiling", but I've never deluded myself into thinking there was), if you are thinking about getting a new computer, it might be time to take that thought a tad more seriously. If gas prices skyrocket, then today's $3,000 PC could be $10,000 tomorrow. Maybe. Maybe not. Just a thought.

At any rate, I'm not wanting to talk about the economy. What I want to talk about is the infamous question, "What computer do I need to buy?" I get asked this question frequently. There are a LOT of choices, so it is no wonder that people get confused and just want some guidance without being made to feel "dumb". Most people buy without thinking and then later regret the purchase.

The first step is to determine your needs. If you develop software, then you need a machine that can crunch through compilations fast (nothing is worse than waiting for a compiler to do its thing). If you play games, then you need a "gaming rig". If you just surf the Internet and word process, then almost any computer will be fine. If you plan on doing multimedia, then you need to look at hardware that will support that...and possibly not even a PC.

The next step is to determine "build vs. buy". That is, do you want to build it yourself or let a company like Dell or HP do it for you? There are some advantages to building it yourself, but, from personal experience, it tends to be cheaper to buy from a major manufacturer and takes a lot less time too. The hardest part with building is getting the base box built properly - the case, power supply, and motherboard don't always work nicely in terms of layout. And then you have to be willing to sit down and sift through the motherboard manual and know what "jumpers" are, know how to seat a CPU, etc. Basically, one big pain (an even bigger pain if the wrong motherboard gets shipped to you). But you do learn how to build out a computer. I highly recommend having a good "geek" friend to help you out when you get stuck if you decide to build instead of buy. If you decide to buy, be sure that the first thing you do is reinstall the OS. Make sure the computer manufacturer sends you a CD/DVD with the OS on it - even if you decide to not reinstall the OS, some geek in your not too distant future will likely recommend an OS reinstall and will ask for said disk. Reinstalling the OS removes all the junk the manufacturer put on the computer, which, BTW, will drastically improve system performance out-of-the-box. The downside is reinstalling the drivers for the hardware can be complicated.

The next thing to do is to determine your budget for the computer. A lot of companies (e.g. Dell) have payment plans. I personally prefer to pay for things up front, all at once - otherwise, you end up paying a lot more for a lot longer. $400 will get you something you can surf the Internet, check e-mail and Word process with. You can do multimedia, programming, or anything else too, but either performance or quality will likely suffer. A $400 computer might not be very specatular, but it'll work just fine for almost any task. $3,000 is about where you start getting into high-end hardware (e.g. gaming systems). $5,000 is going to get you bleeding-edge hardware/software designed for Enterprise-level applications (e.g. servers). You want to buy something that will last about 5 to 10 years and yet meet (perhaps exceed) your needs. So plan your budget for the computer such that you will have paid off the computer long before 5 years is up. Keep in mind that this industry moves so fast that a mere 6 months from now, any computer bought today will likely be considered "outdated". But as long as it continues to meet your needs, then it isn't "outdated" for you.

The next thing to do is determine what OS and software you want to run on the hardware. Microsoft Windows isn't always the answer. And neither are PCs. Apple has iLife. There is no equivalent of iLife for PCs, and even if there were, I wouldn't touch it. The hardware (drivers) and software (DirectShow) interfaces for Windows are too..."flaky"...for lack of a better word to do any serious multimedia work. By multimedia, I mean audio recording, editing, and mixing, video transfers, editing, and publishing, creating video DVDs, transfering/uploading to iPods, YouTube, etc. Still images from digital cameras work just fine under PCs, but when you start fiddling with real multimedia, you need to have system stability and performance. Application crashes (even OS crashes) and lousy performance runs rampant in the multimedia department thanks to the lousy nature of DirectShow - one of the most horribly written pieces of software ever. I'm a big Windows fan, but even I know its limitations. Windows is great on the business application and gaming frontlines. Mac OSX is great on the multimedia front. Linux is great on the server front. Oh sure they can each do some of it all, but those are the areas in which each specific brand is strongest.

Now that you've got the basic idea of what you are looking for, it is time to get to specifics.

If you just need to surf the web, check e-mail, and word process, then any PC (or Mac) will work fine. You can save a lot of money by NOT purchasing Microsoft Office. Open Office is generally "good enough" and is free. Thunderbird is a free e-mail client. And Firefox is a free web browser that tends to be more secure than Internet Explorer (but I personally like IE for surfing, FF for web development, and both are free under Windows).

If you need to interact with various devices such as a digital camera, then you might want a teensy bit more power behind the computer and a card reader. Not really required. A $400 PC will do photos just fine - although, you will probably eventually want to invest in an external USB hard drive if you take a LOT of photos.

If you need to do multimedia, take a serious look at a getting a Mac instead of a PC. If you are familiar with Windows and PCs, Macs are somewhat foreign and do have their own sets of issues but, by far, have a superior multimedia suite (iLife) when compared to the out-of-the-box Windows multimedia support (basically none, Movie Maker is about the limit and even that is lousy).

If you are a student in high school or college and are thinking about a laptop, you might want to take a look at what is called a Tablet PC. A lot of tablets now are "hybrid" laptops with a regular keyboard that can be swiveled out and used. Combining a Tablet PC with Microsoft OneNote creates what I'd consider to be as close to "digital paper" as is possible. Most tablets have built-in microphones that can record the classroom while you take notes. You can then play back (within OneNote) the notes and audio later and they are synchronized. If you hook up a webcam, you can also record a synchronized video of the session. This is a huge plus over a laptop since you draw on the screen, (IIRC) it OCRs the handwriting to make the notes searchable, and you can quickly play back audio at a specific point in the notes to determine why you wrote down something (or, if you have terrible handwriting like me, make out what it is you wrote). Laptops don't cut it in a classroom setting - there are all sorts of charts and graphs to copy off the board, which can only be drawn by hand. The only downsides to a tablet are that they have the tendency to have lower lifespans than regular laptops (typically from dropping them) and they tend to be slightly more expensive.

If you need performance (gaming, writing software, serious data crunching), then you need to start looking at customizations and more expensive gear. This is where this blog entry starts getting interesting.

The major bottlenecks in any computer that determine overall and perceived performance are between the three most critical resources in the computer: CPU, RAM, and hard drive. If you are a gamer or want to buy your first real gaming rig, then you have two more critical resources to consider: GPU and APU.

Let me start with the CPU (Central Processing Unit). Most people think the speed of the CPU defines the speed of the computer. In the past, that used to be true. But today's CPUs tend to sit around idly doing a whole lot of nothing. Not entirely true, but the CPU is almost always the fastest component in the computer, typically waiting on other components to finish doing stuff. When selecting a CPU, you may want to rely on CPU benchmarks to help you decide. cpubenchmark.net tracks CPUs in use and how they stack up against each other. The battle is usually between AMD and Intel - one or the other has the lead. Now we probably all want the latest and greatest, but keep in mind that the high-end CPUs at the top of the high-end chart on cpubenchmark.net are typically multiprocessor (i.e. more than one CPU) and are fairly costly systems as a result (expect to pay around $1,200 per CPU in the #1 position). While a good CPU does have advantages where heavy number crunching is involved, the CPU isn't typically where bottlenecks reside. Additionally, most applications don't take advantage of multiprocessor systems (including video games).

The major performance bottleneck is actually RAM (or, perhaps more accurately, motherboard bus speed). It isn't how much RAM you have (well, anything under 2GB is likely to take a performance hit), it is about how fast the RAM is. CPUs have onboard memory caches that are incredibly fast because requesting data from RAM is many times slower. When selecting RAM, watch for the speed of the RAM. The faster the RAM, the better performance you'll likely see when moving large chunks of data around. Of course, the RAM has to match the motherboard or the motherboard won't recognize the chip and/or do weird things. Also, you need to decide how important ECC RAM is. ECC RAM is self-error correcting RAM. It costs more, is typically slower than non-ECC RAM, but if you need reliability and stability above performance, ECC RAM is the only way to go.

The last major performance bottleneck is the hard drive. The hard drive is actually the slowest component in any computer. The faster that data can be read into RAM off the hard drive, the faster a program will load. Unfortunately, hard drive speeds are relatively the same. However, keep hoping that 64-bit OSes finally take off. Major computer manufacturers are shipping computers that are 64-bit capable and can handle more than 4GB RAM but they install a 32-bit OS and that, in turn, restricts the limit to 4GB RAM. Back in the DOS days, we had a RAM drive device driver that turned a section of RAM into a drive letter. It wasn't widely used for various reasons (mostly because RAM was very limited) but was very fast. It disappeared with the advent of Windows 95. With 64-bit OSes on 64-bit hardware with 64GB+ RAM, it is conceivable that the RAM drive could make a comeback. It wouldn't solve the hard drive bottleneck, but could alleviate it.

Now that I've covered the basic performance bottlenecks, onto the gaming system-specific ones. I love a good video game, but figuring out which video card to buy is complicated. I don't really track the industry too closely and generally have to re-learn it when I go to buy a new video card. The 3-D graphics card industry is littered with confusing acronyms and letters and manufacturers. For this reason, I typically look around for a good comparison chart and hope that someone has one with my current video card and the absolute latest, bleeding-edge video card (used to be a problem - seems someone has stepped up to the plate...more on this in a bit). Another problem with video cards is that you have to match specifications exactly. The standards for video ports are constantly changing and making new acronyms (PCI, AGP ...x, PCI-E ..., etc.). Of course, this usually means that the latest and greatest cards won't work on existing hardware - requiring a whole new PC to play games. Very annoying. And expensive. Anyway, I digress.

Thankfully, there are only two real players in the GPU (Graphics Processing Unit) market: ATI (merged with AMD) and nVidia. I used to be a nVidia fan, but, for the same amount of money, ATI seems to have higher-end hardware that also seems to perform better. That can always change through competition, but I got fed up with nVidia cards giving me consistently lousy results on 3D benchmarks and performing poorly in games despite supposedly being more capable than ATI. I plopped in an ATI card once just to see if it would be different and it performed way better. I don't like ATI's control panel though - too invasive. This is my personal opinion - many people like nVidia.

There are two types of GPUs: Gaming and CAD. Gaming GPUs are the most common and are for cards that will be used in gaming computers. CAD GPUs are geared for specialized CAD applications and tend to be more expensive due to their relative rarity. A gaming GPU can be used for CAD but might not render some things "properly". A CAD GPU can be used for gaming but will have reduced framerates and possibly introduce odd artifacts.

One more thing: You have to be aware that GPU manufacturers don't actually make the video cards themselves. They merely make the GPU. It is up to other manufacturers to make the boards the GPUs go on and then sell them (what card you buy doesn't actually come directly from ATI/nVidia). Some manufacturers are better than others. Some manufacturers overclock. Others don't. Some even underclock. Some cards are better built. Some cards have poor cooling.

Picking a GPU is hard enough. Finding a reliable graphics board manufacturer is harder. You also need to decide how many monitors/displays you will be using. One? Two? Four? That will also affect your decision on which manufacturer to buy from.

Do lots of research before picking a card. It used to be pretty cut-and-dry...whoever could crank out the most texels won. Now there are a zillion factors to consider.

ATI comparison chart
nVidia comparison chart
Compare two cards side-by-side

The first two links allow you to figure out what is the "right" GPU, which, as always, is as clear as mud. As far as I can tell, they (Hardware Secrets) have kept up with every change over many years. I used to have to hunt forever to find a good website to figure out what the latest video cards are. The last link allows you to compare two cards side-by-side and then see prices for the card and whether the manufacturer overclocked/underclocked the card. My personal recommendation is to find cards with comparable prices and see how the GPUs stack up against each other. Expect to spend about $400 for a bleeding-edge card and $250-$300 for a good card that will likely last a few years. Keep in mind though, you can have an awesome video card, but if the CPU or RAM are performing poorly, so will the video card. Also keep in mind that the graphics card market moves faster than anything else in this industry. A new card comes out every couple months. Although, that breakneck pace will hopefully change as the manufacturing die for the GPU gets smaller and heat-dissipation issues increase.

Finally, we reach the APU (Audio Processing Unit). Or lack thereof, as is in many cases. This enters into the obscure world of dedicated CPU vs. letting the main CPU process data. Audio processing is CPU intensive. If you are playing video games, you will want a dedicated audio CPU (APU, or DSP, depending on who you ask) to be sitting on a dedicated audio card. A lot of motherboard manufacturers include what is known as "onboard audio". Anything "onboard" means that the main CPU has to process the data. A dedicated APU is more costly but frees up the main CPU for other tasks. Soundblaster/Creative Labs/whatever-they-are-named-this-week is currently the leader in putting together decent audio cards, but they are getting lazy and complacent and that behavior shows through their lack of concern for their bloated audio drivers and problems people encounter with their cards (lower manufacturing quality). The only reason they are still in business is because of the Soundblaster 16. Basically, the same reason why Borland/Inprise/whatever-their-name-is-this-week is still in business (Turbo C)...customer loyalty.

A word on overclocking. Overclocking is a strange word until you explore its meaning. Every computer has an internal clock. Basically, a timepiece like a watch but much more precise. Overclocking goes over the recommended clock speed. Overclocking is, IMO, a very dangerous practice. Computer systems run hot as they are now, but overclocking pushes the thermal limits of the hardware, introducing failures at higher rates for only slightly modest gains in performance and typically burns out components sooner than their life expectancy. Many companies that overclock include a "liquid cooler" system to counter the overclocking. Liquid cooling may become defacto standard eventually, but for now it typically indicates that the system is overclocked. Play it safe and don't buy an overclocked system.

Most people buy a computer and a monitor at the same time. I've discovered that sometimes that works out, but most of the time, the monitor isn't very good or could have been gotten more cheaply elsewhere. Or, in the case of multiple monitors, it is ideal to buy all monitors at the same time. The reason to do this is so that you have identical colors and brightnesses on all the monitors. Even if you buy the same monitor several months apart, the manufacturer could change the way they make that model that noticeably alters the quality and/or color accuracy. Additionally, as a monitor ages, the backlight/tube fades/dims.

Finally, don't buy accessories from the major computer manufacturers. Or, if you do, make sure you are getting a good deal by shopping around first. Oh they will highlight various items and tell you how great a deal they have on those items, but more than likely you can get them much cheaper elsewhere on the Internet. Both hardware and software accessories. Don't get sucked into thinking they are the only place to buy the item. You can easily save $20 to $250 on most items by looking elsewhere.

Whew! That is a rough overview of how to buy a computer. If you made it this far without your head spinning, then you should be more capable in making a confident purchase. Buying a computer is an expensive endeavor. Treat it like any major purchase - with carefully thought out actions (or inactions - perhaps you'll decide you can't spend money), not being upsold/marketed to. If emotions come into play or you feel pressured, back off from the sale until you've had some time to ponder it. Upsells will easily go way beyond your budget. Be very careful and watch the pricetag closely.

Adobe sucks

2008-06-28T11:12:00.006-04:00

Let's say you walk into a store and you go to purchase a $100+ item. This item comes with one free bonus item that gets shipped later that the item you are buying is supposed to be used with. During checkout, the clerk sneaks in and scans an identical item through and you swipe your card and pay and go home without realizing what happened (let's say something distracted you). Both items are in tightly sealed containers. You get home and you realize that you bought two items. What would you do? Well, you would immediately return to the store with your receipt and ask for a refund for one of the items. Let's say you take both items back just to prove that you haven't used either one. The receipt also contains the exact date and time of purchase (well within 24 hours). The clerk can see that both items are sealed, have never been used, that the product they are supposed to be used with hasn't been delivered yet, and that the product is being returned within 24 hours of purchase. The sales clerk simply refers you to a support clerk since sales clerks don't handle refunds. So you go over to the support clerk and retell your story. Next, the support clerk proceeds to state that they don't handle refunds directly and that you have to go to a specific website URL, fill out a form, and fax that form to prove your identity and that the form will take two to four business weeks to process before you can get your refund. All the while, the support clerk can clearly see the proof in front of them.

You would be furious. You would then demand to see the manager. However, first the support clerk attempts to pass you off to their supervisor - who says the exact same thing. You would then repeat your demand to see the manager to the supervisor. In this case, however, the store manager proceeds to say the same thing despite hearing the exact same story and seeing the same exact proof sitting right in front of them. You would become irate. Some of you would then proceed to pummel both clerks, the supervisor, AND the manager, making sure all of them discovered how to make holes in the wall with their head.

This story fairly accurately describes my experience with the Adobe Online Store and associated personnel. NEVER, EVER buy directly from them or their online store. The experience is miserable. You'll regret it.

I don't know how their site could be so badly programmed to sell a person two licenses when the person clearly only wants one, but it is. The employees at Adobe are blockheads and the store is a horribly written piece of software. Everyone involved in this mess should be fired immediately. I'm slightly surprised that someone hasn't "gone postal". I'm also still waiting for the day when someone in the military who operates a tank decides to go AWOL and level a large corporation's headquarters for having lousy customer service.

The document that you have to fill out and send in is here.

Then you have to wait two to four WEEKS to get your refund. Meanwhile, Adobe is multiplying floating money that rightfully belongs to you. And who knows how many other people they are doing this to. Tens of thousands? Millions? What they are doing is immoral and probably illegal.

How hard is it to revoke a software activation license that HAS CLEARLY NOT BEEN USED and refund the money? Adobe HQ's official response to this is: "You could be using a 30 day trial with the activation key on a non-Internet connected computer." Uh. That doesn't make a single darn bit of sense. If the activation code was being used with a 30 day trial and the trial expired and you tried to activate using the rejected activation code/license, then the application says "rejected" and you simply...USE THE OTHER ACTIVATION LICENSE! Duh! So, how hard is it REALLY to revoke a license...it takes five seconds, if even that. Click a couple times and, POOF!, rejected. Activation systems use backend SQL databases - and rejecting a license is as simple as one UPDATE SQL query. Anyone who says otherwise is lying through their teeth.

This brings me to an important message about customer service:

THE CUSTOMER IS ALWAYS RIGHT! ALWAYS!

I'm the customer. I'm right. Adobe, you are wrong. Period. All I'm asking for is a little common sense to be applied to their refund policy - I asked numerous times to think logically and simply refund the money and they continued to be blockheads. They have the number of times I downloaded the product in front of them (zero). They have the number of times I logged in to download the product in front of them (zero). They have the number of times both serial numbers have been activated in front of them (zero). They even have the dates and times of when I purchased the item and when I requested the refund (within 12 ridiculously short hours of each other!). Any reasonable business that wanted to stay in business for very long would see all of this proof and refund my money right away. Not Adobe. All of this I pointed out, but they continued to spout the same nonsense about filling out that document and that it would take 2-4 business WEEKS to refund my money. Everyone involved in creating this policy and those who interacted with me at Adobe HQ and Adobe Lame-o Indian Support should be fired immediately for not using their brains for their job. Common sense says I should have been immediately refunded my money with no questions asked and no jumping through hoops.

I should have even been allowed to request a refund right from within the website (not that I can tell, I haven't ever logged in) with a hyperlink that says "request refund" for any licenses with zero activations. The result is that it instantly rejects the license so it can't be used and puts the refund request in a queue and sends an e-mail to the purchaser. If I didn't intend to refund my money, guess what? I can simply go purchase the product again or, even simpler, cancel the refund request within 24 hours, which would instantly unreject the license and remove it from the refund queue. If it is activated, THEN AND ONLY THEN should I have to go through the process of filling out and sending in the "Letter of Software Destruction" (LSD?) form. The sort of programming I just described is incredibly easy to do and yet apparently no one at Adobe has any level of intelligence to figure this out on their own.

All I'm asking for is service with a smile.

Get ready to spew your morning coffee...

2008-05-31T12:40:00.005-04:00

...onto your computer monitor and keyboard. Literally.

Following hot on the heels of the "Perl is a terrible language" post comes the reason why I had to re-learn Perl in the first place.

I just barely released MyProBB 2.3. It took nearly two months to get this release ready. 1.4 months of that 2 months consisted of writing a single plugin for the forum. What follows is the shamelessly copied portion of text from the MyProBB 2.3 announcement post:

---------------
[This plugin is] perhaps the best little gem to hit the Internet since AJAX-driven websites, er, Google...

The Official Instant Message plugin. Yup. That's right. I single-handedly hold the distinction of having the first web software package that sends Instant Messages. I hold the distinction of being the first to send IMs to five major IM networks from a web forum. I also hold the distinction of having the only web forum software package that sends real Instant Messages. I even hold the distinctions of having the most advanced Pidgin plugin using the Perl plugin for Pidgin, the first Perl plugin for Pidgin that contacts the Internet, and the first ever multi-protocol automated IM web-driven bot.

That's a lot of firsts. I've done something no one else on this planet has done. It took a lot of sweat, tears, and one-and-a-half months to get here, but I did it.

Despite making the plugin as simple as possible, the plugin itself has over five pages of documentation. However, while it does take a bit to set up, it really does send IMs. To five different networks: AOL IM (AIM), MSN Messenger, Yahoo Messenger, ICQ, and Jabber/XMPP.

There are three parts to the plugin: The 'User Profile' part (what the user sees), the Management Interface part (handles requests), and the Pidgin part.

The first two parts are straight-forward (more or less), the Pidgin part is not so straight-forward. Pidgin is a multi-protocol IM chat client that has Perl scripting support for writing plugins (it also supposedly supports Tcl plugins, which I don't know). I looked at the documentation and knew it would take a while but a Perl plugin for Pidgin was the best route. A Perl plugin for Pidgin would allow users to use the plugin with Pidgin on any Pidgin supported OS (which is all the major OSes).

My first step was to get Pidgin installed with the Perl plugin. That took several days. (The instructions cut through the awful steps I went through). Then I had to re-learn Perl. For the zillionth time. I hate Perl. Once I figured that out, I had to download the source code to Pidgin and learn how that worked. I was digging around in the C code far more than I cared to just to figure out what in the world was going on. I ran into so many ridiculous bugs (fully documented in the plugin source), I ended up writing all sorts of cheesy hacks to get the darn thing to not do weird things. Then I had to wait for Pidgin 2.4.2 to be released because they were making changes to the Perl part of the plugin, which resulted in more hacks. Then I had to figure out how to communicate between the Perl and PHP parts of the plugin over the Internet in a bandwidth-friendly fashion. And then I ran into rate limit and size limit issues of sending messages over IM. By the time I figured everything out, a whole month-and-a-half had passed.

Sorry for the rant. But hopefully that thoroughly explains the reason for the delayed release. Instant Message support for a web forum: A very cool feature. And it totally abuses someone else's software application (Pidgin).
---------------

And now here is why you should spew your morning coffee onto your computer screen: The Instant Message plugin significantly lowers the bar to creating IM spam bots for every protocol Pidgin supports. Pidgin supports every major (and a few minor) IM protocols on the planet.

I wrote the plugin such that it ties into MyProBB and requires verification, but I've basically done the hard work of figuring out how to turn innocent, sweet Pidgin into an evil IM spam bot. My purpose is for good. Someone else's purpose will be for awesome.

If you were holding it in, you can spew your coffee now.
Feel better? Good.

Every new innovation is a double-edged sword...

Microwave ovens can be used to cook food or they can be rigged to stop air-to-ground missiles from homing in on and hitting your very expensive radar dish that the enemy wants to take out. The missiles take out the microwave ovens instead. Fun.

Perl is a terrible language

2008-05-31T11:57:00.004-04:00

Every time I go to use Perl, I end up having to re-learn it entirely from the ground up. That is how bad the language is. Most languages I can come back and look at some code and say, "Oh, I remember what that does." Not Perl. Perl is the only language I've ever used that I come back to the code and say, "Huh? What in the world did I do there?" And then as I read yet another Perl tutorial (how many do we really need?) to re-learn Perl for the zillionth time (okay, more like 25 times), I say, "Good grief. This language is terrible."

Plus Perl has these weirdly named modules and sticks everything, and I mean everything in the global namespace. Including variables you define inside a function. Perl is the only language I know of where, if you forget to use the word 'my' before using a variable for the first time, it royally messes up the entire script execution and takes hours to diagnose.

Additionally, every last Perl module reeks to high heaven of bad design. An example of a poorly designed Perl module is LWP. Whoever named it that should be soundly beaten. Not that I'm advocating violence. Microsoft would have named it Internet. Or Web. But, no, it has to be an acronym for "libwww-perl". Totally obvious. Plus the module itself is poorly written for the average programmer. What, precisely, does the average programmer want to do with a web module? Well, probably access the web and, almost always, a webpage sitting on a web server. So, someone created LWP::Simple. That is great if all you want to do is run GET requests all day, but if you want to do a simple POST request? Not allowed - you have to go use the full-blown LWP library for that.

Here's a wonderful little function that does a POST request LWP::Simple style:

sub LWP_Simple_Post
{
my ($URL, $Content) = @_;
my $Request = HTTP::Request->new(POST => $URL);
$Request->content_type('application/x-www-form-urlencoded');
$Request->content($Content);
my $UserAgent = LWP::UserAgent->new();
my $Response = $UserAgent->request($Request);
if ($Response->is_success())
{
return $Response->content;
}

return undef;
}

Now, really, how hard would that be to have quickly thrown together in LWP::Simple? Probably not even five minutes. Perl programmers are clearly lazy. If you write a module for CPAN, please make it versatile enough to not be so incredibly painful to use. Thank you.

I swear that only bad programmers actually use Perl and swear by it. That statement includes you Larry Wall, Mr. high-and-mighty creator of the second worst but popular language on the planet. Only COBOL holds the dishonor of being worse than Perl.

PHP 5 is an infinitely better scripting language. I've settled on PHP for major web development projects and C++ for client-side projects. It is a pain to have to know more than that.

And now a word from our sponsors...

Oh. Wait. I don't have sponsors. Never mind.

I feel dirty after having to learn Perl again. I'll just go wash my brain or something.

Sins of improper website operation

2008-05-24T08:22:00.006-04:00

People who run websites are generally clueless about securing their websites. What follows is a short list of "seven sins" (I know...a cliché) that are committed by those who operate a website that use a dynamic backend scripting language (PHP, Perl, etc.):

1) Installing third-party components without first reviewing them for how well-written they are, if they have had major security vulnerabilities and/or exploits in the past 12 months, how well defended against automated scripts they are, and how well each component defends itself from known and unknown exploits. If you don't know how to do this, then employ the services of a security expert.

2) Not upgrading components the same day an upgrade becomes publicly available. PHP, MySQL, third-party components, etc. All major releases typically have fixes for security vulnerabilities. And most releases likely already have exploits for those vulnerabilities floating around in the wild. Security firms track both the vulnerability and the exploit and will themselves release the exploit a couple weeks after a fix becomes available (a couple ill-repute firms will release the exploit within 24 hours). Tracking upgrades is easy - use a website monitor program to watch for changes to the site - or sign up for e-mail notifications.

3) Choosing weak passwords for users with access to administrative interfaces (i.e. those with a greater than "generic user" status). A proper administrative password is at least 35 characters in length and completely random with care taken to ensure a crytographically secure random number generator is used (i.e. not based on a PRNG).

4) Writing PHP code without considering the security implications. File manipulation, handling a file upload, assuming that if someone can access a specific portion of code or script that they are allowed to execute it, and cross-site scripting vulnerabilities.

5) Writing code that interacts with MySQL (or other database) without considering the security implications. SQL injection, gaining access to a higher privilege level than allowed, and stealing the data in the database - which is usually critical to the business - if you lost your database data today due to someone stealing it, would you go out of business?

6) Not setting up firewall rules that block ports to anyone but authorized personnel. Blocking access to server resources is critical. If your web host control panel doesn't allow you to restrict access to server resources (e.g. FTP) on an IP address basis (e.g. restrict FTP access to one IP address), then you need to change hosts.

7) Not logging administrative activities somewhere. If something goes horribly wrong, and it can happen to anyone, a log can quickly tell you what the damage is and what needs to happen to fix it. If you don't have a log of all administrative level activities, then you are left to guess. Reinstalling a whole website after a hacker has done their thing can take weeks.

Well, hopefully this will improve the security of some websites and improve a few clueless website operators.

The day of the first mandated rolling blackout

2008-05-12T09:47:00.005-04:00

Fossil fuels are important to programmers. We use computers which rely on electricity which rely on power plants which rely on transportation which rely on fossil fuels. When we run out of fossil fuels is the first day you won't be able to turn on your computer. Or much of anything else. I try to avoid doom-and-gloom in general but this is something that has been on my mind for a while. Basically, if we do nothing, all we will be able to say is, "Well, it was fun while it lasted." So, what should we do? Let's start with automobiles. The biggest consumer of fossil fuel/oil. I'm going to go for a myth vs fact approach here.

Myth: We have many, many years left before we run out of fossil fuels.
Fact: Nope. We've got maybe 30 years left. If even that. Some very conservative figures state 15 years before the first mandated rolling blackout. 30 years would be entertaining for sure - the magical year "2038" should ring some bells. You know - when all 32-bit clocks roll around to 0. Computers that work tend to not get replaced. Some line of code might be, "if (lasttime > time()) LaunchNuclearMissile();" thrown in perhaps as a dummy line that some engineer thought would be funny. Two disasters in one year would be...hilarious.

Myth: Oil is unlimited.
Fact: When I found out about this completely off the wall idea that oil is unlimited, I laughed. Apparently NASA sent out a space probe to Titan (Huygens) and one module, called GCMS, collected data that suggests the methane rich environment is not organic in nature (no Carbon-12). People then extrapolated upon NASA's scientific observations that oil is not necessarily based on fossils as previously assumed - only they convienently left out the word "necessarily". Then other people (not scientists) further extrapolated that oil is infinite. Then the conspiracy theorists decided to have their own take and declare a coverup that Big Oil knew about this all along and that there is actually an infinite supply of oil and that they are merely taking advantage to make a huge profit. All scientists know is that the methane on Titan comes from below the surface of that moon but they never made ANY claims that oil on earth is infinite. All they were doing was casting doubt on the possible likelihood that oil did not come from fossils. Sigh. Leave it to people's wild imaginations... Anyway, we probably won't step away from calling oil "fossil fuel" any time soon, since that phrase is so ingrained in us.

Myth: Biomass fuels (biofuels) based on corn are the answer.
Fact: Nope. Video 1. Video 2. Video 3. Video 2 - gotta love politicians in the hotseat. It takes as much energy as, if not more than, to produce a gallon of viable biofuel. It causes more pollution, not less due to less burning efficiency (a problem I see resolving itself more or less in time - being a relatively new thing). But the food issue is the main problem. We don't have enough food to begin with and farming is unfortunately viewed as a menial and dirty task by us city-slickers.

Myth: Hydrogen cars are the answer.
Fact: Nope. While this is currently hip and cool and trendy, there are so many obstacles to overcome. Pure hydrogen is hard to come by. We know that water contains two hydrogen atoms and one oxygen atom. However, separating a water molecule into its component pieces takes a LOT of energy. The most common method for getting water to separate is to use electrolysis. And the most common method of electrolysis in hydrogen cars is to use a platinum-based alloy. Platinum is an expensive and rare metal. Storage of extracted hydrogen (to keep it from bonding to something else) is also an issue but someone seems to have come up with an interesting solution using titanium disilicide, sunlight, darkness, and heat. But anything viable there is a long way off from reality.

Myth: Electric cars are the answer.
Fact: Possibly. But the electric car unfortunately had an untimely demise. Stuff that has an untimely demise, typically caused by an idiot who claims they have the solution and can't reproduce it, usually takes decades before anyone seriously bothers again. The main problem with electric is distance and that they actually cause the same amount or more pollution than gas-powered vehicles (the pollution is more centralized at the power plant).

Myth: Wind power is the answer.
Fact: Er. Where are you sticking the windmill? On the roof of the vehicle? I don't know why I bother sometimes.

Politicians are great. They have a one-track mind that believes anything they are told to believe. The problem with energy is that you need a multi-prong approach to make it work efficiently and environmentally friendly. People love the idea of a "silver bullet" solution, but you would think that we would have already figured out that just isn't how the world works.

Let me now move on to alternate sources of electric power. Again, myth vs. fact seems appropriate.

Myth: Geothermal plants are permanent.
Fact: Geothermal is a good idea but hardly permanent. They are heavily dependent upon the stability of the crust. A single, small earthquake could easily and permanently shut down a geothermal vent. Additionally forcing the earth to provide geothermal power over the long-term has unknown effects. Geothermal plants are also required to stay up and running 24/7.

Myth: Hydro/Tidal/Wave power is a good idea.
Fact: Yes and no. The latter two typically sit in salt water, which is highly corrosive and will require significant maintenance. The former is typically a dam which can have undesired consequences both upstream and downstream in the long-term. But moving water does create significant amounts of energy.

Myth: Wind power is weak and towers are ugly.
Fact: Well, it depends on the area of the world and where the turbine gets placed. Most turbines are placed well away from residences and businesses, which makes getting power to the destination difficult. For the ugliness, I recommend the engineers at Apple, Inc. You know - the people who came up with the iPod, the iMac. So we need the iTurbine. Hardware engineers are generally not interested in aesthetics. They just want to get it working, whatever it is. While Apple is at it, they should make the iTower to get rid of unsightly cellular towers.

Myth: Off-grid solar is the answer.
Fact: A lot of people are making a big deal about going completely off-grid. Going off-grid supposedly means no more bills from your utility company. The downside is that, while it is being touted as being paid for itself in 20 years, what the proponents fail to mention is battery replacement. The energy during the day has to be stored somewhere to last the night.

Myth: On-grid solar power is cheaper.
Fact: Well, people still have to be paid. And setting up a solar operation is not cheap. And there is the issue of how to store the power, but the power company can store massive amounts of energy and shove it around the grid as needed. And the big ol' power lines still have to be maintained. It may not be cheaper, per se, but it is more eco-friendly. Usually. Some solar plants have emissions but are far lower than most power plants.

Myth: Nuclear plants blow up and irradiate stuff.
Fact: That hasn't happened for a while. It is true we haven't figured out what to do with all the waste material but that is the stuff of movies. Perhaps we shouldn't use nuclear fuel for power plants.

Okay, those are the basic myths I could think of off the top of my head. Wikipedia has a lot of information on all this stuff since it is a hot topic.

The world sees the problem either as one issue or two issues: Stationary and mobile energy. I see a third problem: Localized energy. A viable solution for clean, renewable energy is going to be able to combat all three problems.

Stationary energy is "solved" using power plants. Power plants have the ability to store and move massive amounts of energy across long distances. In terms of renewable energy, there are really only two options - wind and solar. Of those, solar is the most consistent. The amount of sun isn't predictable but day and night are quite consistent. Wikipedia has a visual map of how much centralized solar is needed to power the world's current needs.

Mobile energy is harder to "solve". The current hoopla is over E85 (Ethanol 85%, gas 15%). Ethanol in the U.S. is based on corn/maize, which is a primary food source for stock. Corn-based ethanol is less energy efficient, only fractionally cuts down on emissions, consumes much more farmland, and costs more to make. The problem is the basis of Ethanol being corn. Corn has to be converted to sugar before it can be processed. Brazil, on the other hand, also makes Ethanol but the basis of their Ethanol is sugar cane. The end result of Brazil's sugar cane Ethanol fuel is that it is significantly more cost-effective and efficient than corn. Why not import? Well, there is an import tax that currently makes it more expensive to import sugar cane than to grow corn. The difference is that sugar cane is not essential to life. We can do just fine without refined sugar in our diets. We can't grow sugar cane in the U.S. very well or at least not in sufficient quantity. We could annex Brazil but the world might not be too happy with us.

But there are still emissions. Electric cars would be better. But only if an emmisions-free source (i.e. power plant) was used and batteries were eco-friendly as well instead of leaking toxins all over the place.

And now I come to the fun part: Localized energy. All the current solutions (pun intended) try to integrate with existing systems. Remember how I said earlier that it takes decades to return to a "blunder" (untimely demise) and try again? The famous inventor Thomas Edison actually had his own blunder back in the day. When electricity was first being distributed, it came in D.C. form (Direct Current). The A.C. form (Alternating Current) that we use almost exclusively today had not been invented yet. D.C. was big, expensive, and only traveled short distances. A.C. was able to travel longer distances and therefore required fewer power plants and therefore electricity was cheaper. A.C. is great for high-power applications such as refridgerators, freezers, central air conditioning, furnaces, washers, dryers, etc. You get the idea. What A.C. is NOT great for is small-power applications such as computers, laptops, iPods, cell phones and pretty much anything else with near 100% electronic components inside. These items are usually combined with the infamous block...the power inverter. Power inverters convert between A.C. and D.C. even when there is nothing physically connected to the other side of the unit (I assume that is the case since I can touch the unit and it is warm/hot).

Now, why the history lesson? Well, solar power for the home brings in D.C. current. The panels that people advocate "require" an inverter to convert D.C. to A.C. A lot of energy is lost in that continual conversion process. If all you are going to do is convert that energy again from A.C. to D.C. with another inverter (along with more wasted energy) then why bother with that at all? I hereby propose the reinstantiation of the D.C. wall socket to connect to localized solar power to power your tiny devices. Then, combine that with centralized solar power plants for powering the big-ticket items. If we're going to stop wasting energy then we need to stop wasting it on silly things like power inverters.

Additionally, instead of wasting fuel to power things like the radio, clock, etc. in a vehicle, slap some decent solar cells on the roof and power them that way instead. Again, localized solar power for devices that make sense.

Just some stuff to think about as we code our way to 2038.
Launch that missile!

Google CAPTCHA broken

2008-05-04T09:55:00.003-04:00

CAPTCHAs are those annoying little images that we have to use now to stop spammers from creating free e-mail accounts on the Internet. GMail, for several years now, has been considered a "safe haven" free e-mail address site where only manual signups were possible. This was made possible via their own homegrown CAPTCHA technology. But now it has been broken:

Article on The Register

Google actually uses its own CAPTCHA technology across multiple sites. For instance, Blogger requires filling out a CAPTCHA when posting comments to blog entries or having an account, which, unfortunately, uses the Google CAPTCHA.

What triggered this post is something I heard combined with a recent comment on an older blog posting.

At the top of every blog on Blogger is a little "Flag This Blog" button. I suspect that if enough people click that, it causes Blogger to declare the blog spam. Or at least it factors in. It could also include sudden bursts in traffic and maybe actual analysis of the blog entries itself. I'm assuming the person has to be logged in to use the button, which means they have to have used Google CAPTCHA. If they don't have to be logged in, this paragraph will make me look really silly.

At any rate, what I want to talk about is when to use third-party components instead of rolling your own. The authors of the original CAPTCHA have a pretty good idea of how spammers think and operate and have created reCAPTCHA. Programmers tend to think this way, "I'm going to reinvent the wheel regardless of what is out there already." Google has a lot of people who think that the world will end if they don't build it themselves.

When I was looking to implement a CAPTCHA plugin for MyProBB, I went looking around for the best CAPTCHA I could find first. reCAPTCHA quickly came to the top of my list. It is secure (public/private keys), the concepts seem fairly sound, they are the creators of the original CAPTCHA (so they know what they are doing), it helps make the Internet a better place (reads books), it has lots of features, it looks good, it has the right amount of visual "pop", it is free (nice plus but not important), and it is hosted on mostly neutral territory (an educational institution vs. a corporation). Plus, implementing it is super easy. The end result is that the reCAPTCHA plugin for MyProBB is the simplest plugin for MyProBB in terms of complexity - it makes a pretty good example plugin to learn from if you want to make new plugins for MyProBB.

Google programmers could learn a lot from me. Doing it yourself is not always a brilliant decision. Obviously, if Google relied on the reCAPTCHA servers, that would be a bad idea (someone would take down those servers real fast) but Google might be able to license/buy the back-end source code for a hefty chunk of change.

Solving pesky LNK4204 warnings...

2008-03-06T22:02:00.005-05:00

So today I finally thought I had the entire build process worked out for the upgrade to VS 2008. Turns out I had one final little hurdle to overcome. After experiencing the mess of trying to integrate third-party libraries into my main base library, I decided to try creating a build hierarchy to significantly reduce build times and my base library project size.

Let me clarify what a build hierarchy is: Basically, I have a base library that I integrate with every software application I write. I've actually got pre-made projects set up that allow me to quickly roll out a new piece of software without wasting 30 minutes each time I go to write a quick little program. Occasionally I have need for a third-party library. Instead of dragging those VC++ projects into each and every solution so that the base library didn't break, I merged the solutions into one project. Yeah, after the fact, it turned out to be not one of my brighter ideas (it generally worked fine though but rebuilds were a huge pain). A build hierarchy is a separate solution where I build a static library or DLL of whatever it is that needs building. Then I just use the output libs in my main library's build process. The idea here is to keep the library layer completely separate from the application layer. I don't ever want to have to edit project settings for the application layer or the library layer.

Basically, I added the path and filename to the .lib file to Properties->Librarian->General->Additional Dependencies and then went and built the project. The test application built but then, for every last .obj file in the third-party .lib, the linker issued a LNK4204 warning despite the .pdb files clearly being in the same directory as the third-party .lib file was. The solution to this was pretty obscure. Apparently all .pdb files generated outside the current solution have to have completely unique names to be used within another solution. This is obscure because you would think that the output .pdb filename from each solution being unique would be sufficient as it contains all the .objs and .pdb data within the .lib and library .pdb. Apparently the linker gets confused if two .pdb names conflict at any level in the hierarchy. VC++ gives a default name to each .pdb, making it somewhat painful to go through each project and override it. Anyway, making all .pdbs have unique names (not just the top-level ones) made all the LNK4204 errors go away and build cleanly (did a Rebuild Solution just to make sure).

I still have quite a ways to go before my development environment is stabilized again but the major hurdles are out of the way. I've basically stalled all development effort until this gets done. I've put in countless hours getting this to work and probably lost some sleep too.

Sleep? What's that?

Sigh - library problems

2008-03-05T08:37:00.003-05:00

One the third party libraries I occasionally use is called zlib. It offers rudimentary compression capabilities. My upgrade to VS 2008 has been nothing but one headache after another. This latest headache was caused by the optional x86 MASM assembler module for zlib that drastically increases the performance of zlib. Hand-optimized assembler generally outperforms anything else but is tricky to get right in the first place.

Apparently, the optional zlib assembler integration module requires the now ancient and dead MASM32 assembler (remnants of Visual Studio 6 that I appear to have "migrated" to a later release of VS). This time around, VS 2008 includes its own version of MASM, which I didn't want to overwrite. So, the zlib build issued these errors:

inffas32.asm(594) : error A2070: invalid instruction operands
inffas32.asm(596) : error A2070: invalid instruction operands
inffas32.asm(610) : error A2070: invalid instruction operands
inffas32.asm(667) : error A2070: invalid instruction operands

It took hunting through a blog post in a Far Eastern language (probably Japanese) to figure out how to fix the problem. Edit inffas32.asm and change:

movd mm4, [esp + 0]

To:

movd mm4, dword ptr [esp + 0]

At which point, I said "duh" to myself, tried it out, and it worked. I know assembler and thought the line looked weird but couldn't pinpoint why. I figured I'd do my own blog post on this for two reasons:

1) The websites hosting various Chinese/Japanese language blogs are always ultra slow.
2) Not every programmer can "read between the lines". I didn't have to translate the blog to English to know precisely what was being said.

People complain about outsourcing but guess who is coming up with solutions to problems first? I frequently find the most interesting bits of code tucked away within foreign websites - particularly of the Japanese variety. I have never been to an Indian blog as a Google search result.

Sigh - crashes.

2008-03-01T21:51:00.003-05:00

Lots of crashes. I just "upgraded" to Visual Studio 2008 Professional a couple weeks ago. I've been working my way slowly through this incredibly painful upgrade cycle. I knew it was going to be painful in advance, so I have segmented the upgrade to span about three weeks of effort. Let me journey my experience thus far:

1) The first step in this adventure was to uninstall VS 2003. That took a while but actually wasn't very painful.
2) I then attempted to install VS 2008 Pro out of the shiny new box. I didn't want to fiddle with the whole upgrade process so I went ahead and got the full version.
3) Then the installer froze while trying to install. In particular, it froze on attempting to install the 3.0 .NET Framework.
4) This is where things got...complicated. I went out to Microsoft Update and had to do a ton of upgrades (including the annoying spyware install of Windows Disadvantage). After many hours I finally got the 3.0 .NET Framework installed.
5) I then started the install and ran it. It crashed halfway into the install of the main Visual Studio components.
6) I crossed my fingers that it would pick up where it left off, hoped nothing would go horribly wrong in the future, and then started the install again. It seemed to continue where it left off and finished off the install without any more problems.
7) I applied my special fixes to devenv.exe - a symbolic link/symlink patch (see my CodeProject article) - some debugging libraries - various include files and batch file modifications. Basically, a lot of grunt work to connect everything back up. I was hoping this would keep things relatively transparent. Er, that's not quite what happened.
8) I installed the MSDN Library documentation. This took a very long time to complete.
9) At this point I realized that the 64-bit components were not installed and had to go back and install them.
10) I then ran a full Platform SDK upgrade. This also took a while.

That was the end of week one. Roughly 10 hours of effort.

11) I imported an existing project.
12) I attempted to build the existing project. Noticed that Intellisense was busted. Had to delete the relevant .ncb files and restart the IDE to get Intellisense to work properly. Seems to be slightly more intelligent in this version.
13) Existing project libraries failed to build.
14) Upgraded/edited/modified libraries.
15) Multiple crashes occurred while editing property sheet pages. Discovered that VS now seems to back up project files. What would be better would be to fix the crash bugs but at least I lost only minimal data each time it crashed.
16) Managed to get the libraries to build with tons of warnings (even on warning level 3, level 4 warnings were being issued).
17) Discovered that upgraded applications attempt to still locate old VC libraries to link against and fail. Somehow the solution or project still points at the old VC runtimes.

And this is the point where another 10 hours have passed. End of week two.

I'm doing this upgrade in my spare time. My focus is mainly the web forum software MyProBB. I took a break today from working on that to get my development environment stabilized. Again. And I was hoping to work on something exciting too but I'm probably not going to get to that.

Replies to e-mails

2008-02-28T20:58:00.004-05:00

I was sitting and thinking today, "Why is it so difficult to find the reply I sent to that person?"

Most e-mail clients track who you have replied to and, almost always, that message is sitting in the "Sent Items" folder or something similar. GMail includes all replies to a message in the view but I don't really like that. If an e-mail client can track what e-mails have been replied to...how much harder is it to also store a pointer at the message that was sent as the reply? Then you could right-click in the e-mail client, click a "View reply" menu item, and immediately jump to the reply message. If you replied more than once, a dialog box would pop up asking which reply to jump to.

This feature could be implemented in probably 30 minutes. Yet no one has bothered to do so.

Time of day for conducting business

2008-02-23T11:45:00.004-05:00

It is useless to waste people's time. That statement is something we all can agree with. Yet, I have one question: Why is every last business open from 8 to 5?

There are several industries that we all agree have to be available and operational 24/7. Emergency centers such as hospitals, 911 responders, local police, and fire departments. And websites/web servers. Anything else running 24/7 is not essential.

Those businesses that are not retail and are not open 24/7 are open 8 to 5, Monday through Friday. Take banks, for instance. Let's say you are depositing a check into your local bank - assume someone gave you a check. When do you do it? Saturday morning, of course. Why? Because banks are open only during "regular business hours" and closed Saturday afternoon and all day Sunday. The only time available is Saturday morning unless you can go into work later or leave earlier. Here's a schedule banks should consider:

12-2:00 p.m., 4:45-7:45 p.m., 2-3 a.m. M-F (6 hours per day)
8-7 Sat (10 hours)

Total time: 40 hours.

Actually, similar schedules would work well for almost ANY service-oriented industry. Most clerks/tellers seem to twiddle their thumbs frequently during regular 8-5 business hours and are absolutely thrilled when a customer shows up. The customer gets way more attention than they really want and it is pretty obvious the employees are desperate to do anything - even "busy work".

Another example is dentists and other similar service industries. Dentists work 8-5, M-F and do the occasional emergency surgery on weekends. The problem with that schedule is regular 8-5'ers have to take time off from their employer to go to the dentist. Employers generally are not happy with their employees leaving to go to scheduled appointments even if they are aware of the appointment weeks in advance.

One major downside to the schedule is that many employees live a bit away from the employer. People would probably consume twice as much gasoline as they would likely go home to family and sleep after the second timeframe. To combat this, the 2-3 timeframe could be made to be every third day on a rotating schedule (1/3 the staff). Or fill that time-slot with part-timers (there are a bunch of older folk out there who are retired but who would also love to make a little extra money by working for just one hour a day).

The other major downside is school starts at 8 a.m. and lets out at 3 p.m. at most places. The above schedule works well for the 3 p.m. issue for most people but there might be some issues involving sleep with the 8 a.m. issue. Lots of things to consider.

Still, this is something worth giving a shot. It would significantly ease the "'rush here and there' before they close" mentality for the 8-5'ers. Less stress is a good thing.

Software companies should adopt a similar schedule but maybe provide some extra overlap with the 8-5 (shifting time away from Saturday). The 2-3 a.m. portion allows the software company to interact with people overseas. That is where many sales are mostly going to be in the not-so-distant future.

Video game annoyances

2008-01-07T09:17:00.000-05:00

So I was playing a video game called Team Fortress 2 yesterday (I don't play games often - except as a breather between releases) and it dawned on me that it took nearly five minutes to get to the main screen of the game, about another five minutes to load the level for the server I was joining, then another two minutes to find another server with the same level after getting kicked off for "filling a reserved slot".

The major companies out there, particularly Valve, make moderately fun games...but they take forever to load. Back in the DOS days, if we had to wait 12 minutes for a game to load, we'd be hitting the reboot switch. Compared to today's games, those games loaded nearly instantaneously. Even if there was a "Loading..." screen, we knew it was only a few seconds delay before we got to go blast our way through stuff.

Console game makers such as Nintendo about the same time weren't sitting around twiddling their thumbs on "Loading..." screens. Super Mario Bros. ran smoothly on lousy hardware and had near instantaneous load times.

Granted we've got 3D, textures, HDR, and other things that make stuff look "pretty" but I have one question for the entire game industry: Why not just load the level WHILE playing? Display wireframe models (or perhaps just simple blocks) and load the textures and other big items simultaneously while the user plays the game. All that is needed to play a 3D game are the level data and some basic wireframe/block models. Then, while the user is playing, in a separate, low-priority thread, load the rest of the level nice and slow. Sure - it will take about the same amount of time (a slight bit longer) for the graphics to load, but waiting for nearly 12 minutes to just start playing one level is beyond ridiculous.

You would think programmers would think, "Gee, this takes forever to load, let's do something revolutionary about the problem." But instead they are thinking, "It takes forever to load, but loading is going as fast as possible." No wonder the game industry is in a rut. No one thinks for themselves any more.

A new year. A whole new set of rants. Or something like that.

How secure is secure?

2007-12-29T11:37:00.000-05:00

People buy things online all the time. They trust the Internet security mechanisms that have been put in place and believe a little gold icon in the corner of the web browser equals security. What most people don't do is stop and think "gee, how does a little gold icon equate to security?"

So this wonderous blog entry shall take you through the world of "modern" security, how it works, and then how easily it can be broken. I'll also share some interesting tidbits of information and form an interesting connection that no one else has made that is worth thinking about. Just so you know in advance, I still trust the technology and make purchases online, but it is something that has nagged me for some time now.

People have been making stuff "secure" for a really long time. Other people have been trying to break security for just about as long - mostly so they could steal whatever was being made secure. Only in recent years has a third group formed. This third group of people try to break security only to make things more secure. The third group is considered to be a "gray" area and is loaded down with controversy. I'd like to think this blog entry classifies as none of the above groups and is merely a different perspective.

Anywho, security has been around a really long time. The earliest documented case to date of information being encoded/encrypted is Julius Caesar's Caesar cipher:

http://en.wikipedia.org/wiki/Information_security
http://en.wikipedia.org/wiki/Caesar_cipher

The Caesar cipher was a relatively simple encoding that involved shifting the letters of the alphabet 'x' positions. We laugh at the simplicity of this today given the sheer amount of technology at our disposal but, back then, it had to be an arduous process to encode and decode messages and the secrecy of how the encoding worked made messages appear to be gibberish.

Even as recent as World War II, cryptographic algorithms had to be kept secret. For if the knowledge of how the algorithm worked became public knowledge, the algorithm would be broken and all messages would be able to be read. An example of this was:

http://en.wikipedia.org/wiki/Enigma_machine

Jumping to today: All cryptographic algorithms in use today are known and published works that have been analyzed by both the second and third groups I talked about. Yet we remain secure and purchase things online using these algorithms. How is this possible? Introducing:

http://en.wikipedia.org/wiki/Public_key_cryptography

Public key cryptography. I'm not going into the math behind the concept but if you read the above page, you'll see mentions of "Alice" and "Bob". Think of "Alice" as your web browser and "Bob" as the website (web server) you are talking with. If you are the sort of person who needs a specific example, let's pick Amazon.com's purchasing system to be Bob. Most people are familiar with that site.

How public key cryptography works has been repetitively dumbed down so people like me can understand it. Simply put: Alice and Bob each have public and private keys. Their private keys are in their separate rooms. They both exchange their public keys in sight of everyone. Alice then goes to her room and encrypts a message using Bob's public key and then encrypts the result with her private key. She then leaves her room and publicly hands the encrypted message to Bob. He goes to his room and decrypts the message using Alice's public key and then his private key.

The most complex part of this is understanding that public and private key pairs have to be generated in such a way that you can't decipher what the private key is based on the public key. From here on is where most people launch into a tirade of how the math works. But since most people don't like math, I won't bother with the details. Just suffice it to say that it works as long as both private keys are kept secret.

When it comes to web browsers (i.e. Alice), though, you probably aren't aware of any private keys you have. That would be because you don't have any. At least not at first. You probably have noticed that it takes a while to connect to a secure website but after the initial delay, navigation goes relatively quickly. Well, part of that delay is your computer generating a private and public key pair to use to contact the website (i.e. Bob). Generating these keys can take up to 15 seconds plus a lot of CPU and they are usually generated every time you visit the site.

But here is where things get tricky. A lot of thought has gone into how the fictional Alice and Bob communicate with each other. The problem is this: Most discussions about Alice and Bob are how they know each other personally and trust each other implicitly. But if Alice is a web browser and Bob claims to be Amazon.com, how does Alice know that she can trust that Bob is actually Bob on the big bad Internet and not someone who has surreptitiously made himself to merely look like Bob?

http://en.wikipedia.org/wiki/Public_key_infrastructure

Public Key Infrastructure, or more commonly, PKI, is a fancy phrase for saying, "Use a third-party to validate the public and private keys". That is, someone other than Alice or Bob validates that the message came from Bob because both Alice and Bob trust the person/organization who signed Bob's certificate:

http://en.wikipedia.org/wiki/Public_key_certificate

That is about the most concise definition of a certificate I've seen to date. Since I can't say it better than that, visit Wikipedia.

So what's wrong with all this? It sounds all wonderful and like it actually works. The problem begins with deciding who to trust as the Certificate Authority (CA). Over 50% of the certificates issued each year come from Verisign and Thawte (also owned by Verisign). That means a LOT of trust is riding on Verisign - a commercial company. But they are fairly responsible with the trust given to them despite their commercial interests.

In general, the entire system is quite secure. So where's the problem?

The problem lies with Verisign's private key. Whoever has that key can generate certificates that are automatically trusted by any web browser. From what I understand, most trusted CAs use a non-networked computer with a single serial line that is dedicated solely to generating signed certificates. I have one question: What happens if the Verisign private key falls into the "wrong hands"? It isn't inconceivable to then create, say, a man-in-the-middle attack for Amazon.com (Bob) or your online bank and capture all the information in plain-text without you knowing about it. Digital security is one thing, but Verisign doesn't tell us how secure the physical security is. But if Verisign is secure physically (i.e. you couldn't get the private key guns a-blazin'), that doesn't mean other trusted CAs won't have weaker security.

Every web browser has a list of trusted root certificates. After this blog entry, I wouldn't want to work for any of those companies.

The physical route might be too obvious. ("Hmm...I wonder why they were only after the only computer that had just happened to have our most critical private keys on it that we use to just sign digital certificates with and nothing else?") Trusted keys in browsers would get changed and everything would be secure again. Even then, it may be possible to not actually have to do that. What if someone figures out a way to reverse-engineer a private key? Most certificates expire and have to be renewed annually, which makes reverse-engineering the private key of individual websites infeasible. However, trusted CA root certificates expire anywhere from 2020 to 2037. It may be possible to reverse-engineer a CA private key from a set of signed certificates and using a guesstimate of the rough date and time of when the CA generated their private key and what software package they might have used to generate the key. That is, re-create the private key they use. Given that there is a LOT of time between now and when the first major (most widely used) CA certificates expire, this may be a feasible route - and a lot harder to figure out "whodunnit".

It is interesting to note that the export laws concerning cryptography and cryptographic algorithms are now fairly relaxed. This is the interesting connection I wanted to make:

http://www.financialcryptography.com/mt/archives/000206.html

Verisign can issue certificates to the government that can be used for "wiretapping" purposes. That is, the government can spoof Amazon.com without prior permission should they choose to do so. Granted, a subpoena is required, but aren't that difficult to get if you have the right credentials. If the government can wiretap with Verisign signed certificates, then that means anybody could do that for any website if they had Verisign's private key.

If my second conjecture is correct - that is, the possibility of reverse-engineering the private key given the ability to roughly guess certain parameters and letting the computer brute-force the rest - that may be another explanation for the relaxing of export law in cryptography. That is, if the government already has the ability to do so and someone has figured out how to see 90% or so of transmitted data that has been encrypted as plain-text. Some people should find it odd that an increasingly controlling government would relax something like cryptography export law unless they already knew something the rest of us don't (i.e. the law didn't matter anymore).

At any rate, I'm not too concerned, but I do keep the stuff I do on the Internet that requires security (e.g. purchases) to a minimum.

By the way, the idea of physically obtaining Verisign's private key could make for one-third of a cool action movie - probably along the lines of National Treasure - assuming they made it as accurate as possible (National Treasure did actual research on how someone would go about stealing the Declaration of Independence). The other two-thirds of the movie would involve simultaneously breaking into major Internet eXchange Points (IXPs) worldwide (fairly secure facilities - from what I understand, a few are really deep underground) to utilize the newly acquired private key (probably would configure their setup to route specific traffic - e.g. Amazon.com secure web traffic - through an attached proxy for the ultimate man-in-the-middle attack). What happens after that is probably bad - ranging from identity theft to economic collapse. But since no one reads this blog (i.e. no more than five people), it is highly unlikely to come to the attention of Hollywood's script writers and therefore we won't have a cool movie made that has the nifty geek-factor in it. Then again, there aren't many of those movies made anymore. Either because Hollywood doesn't cater to geeks or the script writers hate having the details poked at by the geeks. I'd watch the movie though - it would likely be fun and entertaining.

Education and business

2007-09-13T14:14:00.000-04:00

I find it entertaining that people worldwide think they have the right to discredit Americans and make fun of them. If you are not American, you do NOT have this right. However, since I'm an American, I'm allowed to make fun of my own people and nation as a whole.

It is pretty well known that education in America, as a whole, stinks. This video is well worth watching because I believe it paints a pretty accurate picture of the state of the nation:

http://www.youtube.com/watch?v=pfRUMmTs0ZA

The video is from a show called 20/20 and it is over a year old but please return when you finish watching.

I was brought up in a family that cared deeply about proper education. I had learned to speed read by the time I entered school. I also remember at least four different forms of corporal punishment used on my behind/legs as a child. Boy grandma sure loved finding that freshly-cut switch. But it made me into a fine, upstandin', law-abidin' citizen who cares about the world around us. And history and math were rammed down my throat (actually, I enjoy history* and math). Fun is NOT what school is about - people are there to learn. Those that cause trouble and hinder the learning process should be publicly humiliated and, should it continue, be kicked out**. The same goes for teachers and administrators who don't know the first thing about education.

* Those who fail to study history are doomed to repeat it.
** There are plenty of ways to learn. For instance, I taught myself everything I know about Software Development through hard work and perseverance. (And, just to clarify, I was never expelled but did cause some trouble mostly because school was not challenging enough - I got A's in my sleep - another reason to have a fast-track program for brilliant individuals...most of them stagnate under the current system).

One of my biggest gripes about education in America is that it is NOT tailored for the individual. Everyone takes the same exact classes regardless of what industry they want to be in. Really smart people are kept together with the average people and those who constantly struggle. Let's take math - considered by most Americans to be one of the "hardest" subjects. The reason people find it hard is because they don't see any use for it. Our daily lives are not The Matrix and we aren't Neo. When someone has that mindset, you can't teach and they can't learn. Might as well be talking to a wall. Which, ironically, is almost always based on math.

The unruliness you see in the video is not too far off from the truth of most schools. Granted the kids in the video knew there was a camcorder in the vicinity, so their behavior was perhaps slightly more outlandish than usual, but it is pretty accurate overall (i.e. they did a decent piece of journalism). I know a bunch of teachers (and even one principal) who constantly complain about unruliness of their students and how their hands are tied. The nation is shocked when a teacher "snaps" (e.g. spanks a misbehaving student)...I'm not surprised. Teachers everywhere here are frustrated and how they manage to maintain a semblance of composure is absolutely amazing.

So what am I saying? Well, part of the problem lies in grades. I'm not talking about A, B, C, D, E, and F--. Nope. I'm talking about 1st, 2nd, 3rd, etc. The system restricts really brilliant individuals from proceeding ahead and forces people who aren't ready to continue to do so. It is important to study math, English, history, etc. But by the time educators finish allocating the basics to a specific grade, there is no room for customized/individualized education. Particularly those of a technical nature. Few people learn to write computer programs until college - as if to say "Well, college is the place to learn that stuff". Not true. Elementary school is the age to start programming if that is going to be your target career.

Here is what I propose: Drop the idea of grade levels entirely. Students take types of classes in some order relevant to interest and closest industry. If they don't pass, they retake the class. Some people are "prodigies/savants" though - so they might ace every math class but continually struggle through everything else. This is where colleges/universities and lower education have to agree to combine forces. Why have two separate institutions? It doesn't make any sense.

Part of the problem is also discipline. Teachers have no way to enforce it. I therefore propose to reinstate corporal punishment. As far as I can tell, nothing is stopping schools from writing up a legal document that parents have to sign to let their kids attend the school. In that document, simply have checkboxes as to what types of corporal punishment are allowed by the parent to be executed by the school teachers in the event of unruly behavior. If the parent won't allow any punishment, simply send a rejection letter (Thank you for applying to [Name of School] school. Unfortunately, we cannot accept your child at our humble institution as we have determined he/she is not a good fit for our organization. We hope you have success in your search). This unties the hands of the [currently very frustrated] teachers to actually do their job. And it has the side benefits of creating a more stable, honorable, and well-educated society. And it won't cost anything...the necessary material for switches is generally located around the outside of almost any building.

The last problem is relevance. Why separate and segment education from the work force? That doesn't make any sense either. When I was in school, I constantly heard the phrase "real world" in reference to the work force. What does that mean? We're already in the real world. Phrases like that say to me, "We in education have created a bubble for ourselves such that we instantly and purposely obsolete ourselves." In terms of technology, education is always lagging behind by at least 5-6 years (some schools don't even teach technology and just admit that it is "out there in the 'real world'"). Education is pointless if it doesn't keep up to date. Students today are bright enough to realize this...hence they have no desire to learn. And why should they? They are learning about and using outdated technology and methods. Businesses have already realized this and provide on-the-job training to compensate.

Proposal: If businesses and corporations cared anything about education, they would get involved instead of shuttling money into a generic fund (e.g. Bill Gates' Foundation). That could mean, for example, employees taking a day off here or there and stepping in to provide a history or math lesson to students. It would be designed and prepared by the teacher, presented by the employee (failure could mean they get the pink slip). It says "knowledge about the world around you is important to your future" to those students. The end result is they will pay closer attention to their teachers. Get enough businesses involved (who are there to teach the current lesson plan and not advertise their products) and teachers can take a much more sideline approach to learning (focusing a lot more on lesson planning and helping students achieve more).

Or maybe getting involved would just be an addendum to the teacher's lesson plan (e.g. 5 minutes at the end of every class). The business person would show actual source code that used a math concept the students learned about from a real software product their company uses.

Now: Why should a business even consider this? Well, the youth of tomorrow are generally creative and have plenty of ideas. Some students will be more than eager to share their ideas with you. Most businesses stagnate or get in a rut when their first cash cow happens. They just lose the drive to innovate because of the risks involved with innovation. Doing this also has the bonus benefit of keeping the employee's minds sharp. You will also be aware of what today's youth experience and desire. And then there is the cliche and obligatory: 'Helping to shape tomorrow's work force, today!' Or something like that.

Just a few thoughts that I have thought about for a long time.
(In general, I'm pretty well-behaved. Only saying things I regret later when I lack sufficient sleep.)

Standards documentation is annoying...

2007-09-10T10:24:00.000-04:00

Don't get me wrong, having Standards is a great idea. Being able to clearly communicate how something works is essential to daily life. If we, for instance, did not have the HTTP RFC Standards, you probably wouldn't be able to read this blog entry because every single web server would deploy their own idea of what a website is and there would be no single web browser to handle every single web server. (Or if there was, it would be a couple hundred gigabytes).

No. What irks me is the fact that very few Standards authors actually sit down and write binary data examples. That is, "Here is some sample data" and "here is how to read the sample data" and "here is some basic source code that reads and processes the data". While Standards should be about specification, of which they do a great job already, they should also be able to present an implementation or at least an example that could actually happen.

This is where Standards bodies and a community dedicated to each Standard should meet online to perform the latter. The actual Standard is supposed to be "set in stone" sort of documentation that changes very infrequently (if ever). What I propose to all Standards bodies everywhere is that the published Standard be closely associated with an online Wiki AND an official online forum. Dedicated community and Standards body members can then provide the essential examples and implementation that a Standard does not contain.

A perfect example of this is the C++ Standard. Some time back, some people came up with what I call the STL. Also known as the Standard Template Library, which, at the time, was not part of the Standard itself but was quite popular for providing additional functionality that avoided reinventing the wheel by using C++ templates. The ANSI C++ Standard committee adopted most of the STL into the C++ Standard. I still call this the worst idea they have had to date because the Standards body is defining implementation where they shouldn't be. The reason I think this way is because you cannot extend any of the existing templates (e.g. edit them) without violating the Standard. Because STL was no longer valid for new stuff, another really popular project called Boost has been made. What happens when the ANSI C++ Standards body decides to integrate Boost into the Standard?

The Standard should merely define what it means to be a template...not demand certain templates be included with every compiler. The STL should have been adopted into a Wiki and a related project. That way, it could have gained certain bits of functionality I consider essential. Take a look for a moment around the industry: C#, Java, PHP, Ruby, etc. are all gaining popularity and I say it is because they are not being hindered by Standards. Integrating STL into ANSI C++ was a horrible idea. Instead, the Standards body should have said "Hey, we really like this, let's point it out in the Standard as 'officially recognized' and dump it on our Wiki so the community can continue to develop it." Boost would have never existed.

The other thing I've noticed about ANSI C/C++ is that it continues to be geared for a line printer. This is 2007, not 1970. We have what is called the Graphical User Interface (GUI). You know? Mouse that points at and click buttons and hyperlinks and images and stuff? The ANSI C++ Standard and modern computing are at direct odds with each other. Because there is no central location for building on the ANSI C++ Standard such things like GUI stuff, threads, sockets, database access (!), etc. C++ has fallen by the wayside because to do anything you have to re-invent the wheel. Need socket code? Sorry. You'll have to go write that again despite already having been written thousands of times over and over.

As a result, people see other languages as being "easier to use" simply because those languages provide modern technology more easily. Sure C++ can do it, but you've got to lay the groundwork down before using it (even importing an existing library can be challenging). If another language already has the groundwork code written and included by default, then C++ becomes inferior.

Honestly, I'm not really sure what I'm looking for. Maybe I'm asking for two different things. That happened before when I wrote VerifyMyPC 1.0. I didn't really know what I wanted from the tool. It took a lot of feedback and self-analysis to finally figure out what I wanted and what other people wanted as well. I guess what I want is a Wiki for collaboration to create examples that people can use to understand Standards. However, I also want Standards to be separate from implementations and yet, at the same time, not distance themselves from the real world such that they obsolete themselves. I'm not really sure how to address the latter problem, but it is one that causes languages to become obsolete.

I don't ask for much, do I. :)

Setting up a wireless network

2007-09-07T16:26:00.001-04:00

Edit (July 12, 2010): Ruh-roh! This blog post was declared Dead on Arrival. Read the story on how my "secure" WPA-PSK wireless network got hacked before setting up your wireless network. My personal recommendation is to NOT use a wireless access point unless you do some real hard thinking and research.

Occasionally I will receive a request for help on wireless networking. Usually the person was scared by someone when they were told, "Wireless networking is insecure. Your personal computer data is at risk." The first question they ask me is, "Is my setup secure?" Well, I'm not a mind reader and usually not in front of the computer, but usually those same users are surprised even to know that they can log into the router.

Okay, so the first thing I have to do is explain what a router is. In layman's terms: A router is something that takes data from computers on a LAN and sends it out on to the big bad Internet. When a response comes back, it is responsible for making sure the data gets back to the computer. Routers also double as a hardware firewall (keeps the bad guys out). If you want the technical explanation, go here:

http://en.wikipedia.org/wiki/Router

Most modern Cable and DSL lines go into some sort of box that you then connect your computer to (or use wireless in it). These are usually bundled with some really lightweight router software stuffs.

If you really want to secure your wireless network, I highly recommend getting a decent router. Like an actual router instead of the fluffy stuff your ISP provided you with. Buffalo, at the time of this writing, makes a pretty mean wireless consumer-level router. You can get consumer quality routers at pretty much any computer and office supply store (Best Buy, Office Max, etc.)

When you first hook up the router, your network is vulnerable (i.e. completely insecure). The first thing to do is use the bit of patch cable (CAT-5) that came with the kit and connect it directly to the LAN port on the computer you plan on using wirelessly. Configuration should be done over the wired connection just to make life simpler.

Once connected, your computer will have an IP address. Now comes the technical part. Go to "Start->Accessories->Command Prompt". If you aren't familiar with this, don't worry. Type in "ipconfig /all" (without quotes). Something like this should appear in the output:

Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . : ****************.
Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 192.168.0.130
Default Gateway . . . . . . . . . : 192.168.0.1

The line you are after is the 'Default Gateway...: 192.168.0.1' line. Now start a web browser and enter:

http://192.168.0.1/

(Substituting whatever the Default Gateway is for you.)

A password prompt should display. The default username is usually something like 'admin' and the password field is left blank. This differs by router and model - read your manual.

A webpage should now display. This is usually called the Web-based Administrative Interface. What this looks like is usually different for every router. I refer you to your manual on your router for how to navigate the interface that shows up. Keep the command prompt open, more stuff from here is needed later on.

The first thing to do is to locate the wireless device name and change it from the default to something you will remember. This is also to avoid conflicts with surrounding devices. Also, if there are a ton of other wireless devices in the area, you should select a different channel. The default is usually 6. You can usually select a range from 1 to 11. However, because you are operating with radio waves, a channel is actually selecting a range in the 2.4GHz band. This range spans roughly 3 channels on either side. So to completely avoid signal overlap with channel 6, you need to select a channel that is at least 5 channels away: 1, 6, 11.

After making a major change and the router reboots, it is a good idea to close the browser completely, open a new browser window up, and reconnect to the router.

If you royally mess something up or get seriously lost, you can reset the router. There should be a little spot that you can push with a pen for 30 seconds to reset it to factory settings. You'll have to start over from the beginning if you use that (but at least you know it is available).

Now you are ready to start securing the device.

The first thing to do when securing a wireless router, is to go locate where the password for the router is stored. Change the password. Close your browser and open a new browser to the same location you just had opened. You will be asked for your new password. Make sure the password is complicated enough. Someone who breaks into the network will go straight for the router configuration to figure out where the weak points are within the network (along with making it easier to break in next time).

The next step is to make sure remote administration of the router is turned OFF. You don't want outside snoopers on the Internet to even know you have administration interface. Log into the administrative interface and locate the setting for remote administration and make sure it is off.

The next step is to go into the wireless configuration and enable some security on the wireless end of things. The default setting on wireless routers is to disable encryption. That is, everything is being sent in the clear and not encrypted. Someone could watch various bits of traffic such as e-mails, IMs, etc. without even being on the network.

There are various levels of encryption available and they have really weird, technical names for the non-technical person. So, I'm going to lay it out in a really straight-forward manner from strongest encryption to weakest:

WPA-PSK2 (AES) (aka WPA2-PSK)
WPA-PSK (TKIP)
WEP
None (default)

Some sites say that WEP is better than nothing, but it can be broken in 10 seconds. I leave it up to your imagination to come up with some humorous analogy. Note that as you increase encryption strength, the distance you can travel from the router (e.g. with a laptop) decreases significantly because signal strength drops off on both ends (power is diverted for encryption calculations). Also, you have to be careful with WPA-PSK2. You have to be using a network card that supports it and drivers that support it and have at least Windows XP SP2 and possibly a specific update.

When choosing a password for the encryption key, make it random. Completely random - letters, numbers, and characters. And store it in a file on a USB thumbdrive or write it down or something. And also make the password really long (at least 20 characters, preferably 40-50). NOTE: Some network cards do not like really long passwords in excess of 15-20 characters.

Once you apply the password and encryption changeover, wait for the router to reboot. You will have to reconnect to the wireless network using the new key. Right click on the "broken wireless connection" icon (has a red X through it) and select "View Wireless Networks". Select your network and click "Connect". Enter the password. It should connect. Now wander around and figure out the limits of how far you can travel before losing the connection. If necessary, move the router to a better place to cover more usable area.

This next step is optional but highly recommended. Log into the administrative interface and go to the wireless setup and locate something called "MAC address filtering" or "MAC filtering". This feature has absolutely nothing to do with the computer systems that Apple, Inc. sells. Enable MAC address filtering. Once enabled, there is usually a "Clone MAC address" feature with a dropdown list but, if you are physically connected, it will have to be entered manually. Switch to the open command prompt and locate the line that says "Physical Address...: XX-XX-XX-XX-XX-XX"...BUT make sure it is for the wireless card (NOT the Local Area Connection). Then switch back to the browser and enter the data into the fields manually.

What MAC address filtering does is say, "Only wireless network cards that have this MAC address are allowed to connect into this router." MAC addresses are uniquely assigned by the manufacturer of the wireless card (or any network card for that matter). Some OSes can use specialized tools to change the MAC address and "spoof" or fake a different card but conflicting MAC addresses makes it harder to get into the network without being noticed.

Optional: Locate the option in the administrative interface for turning off the SSID. The SSID is usually broadcast by default to make it easier to configure. It also makes it easier for the average snoop to figure out where wireless networks are. Those with wireless cracking tools, though, will still be able to figure out your SSID even if broadcasting is turned off.

Optional, but recommended: Switch the patch cable to the Cable/DSL box and connect into the web server that runs there (use 'ipconfig /all' again from before). Locate the a "Wireless radio on/off" toggle and turn off the wireless radio. You don't need it any more and will likely conflict with your newly secured wireless router.

Now that the wireless is completely set up and secured, you are ready to connect to the Internet and see if everything works. Up to this point, everything can be done and verified to work without connecting the router to the Cable/DSL modem. So go ahead and plug the patch cable (CAT-5) into the Cable/DSL. Test to see if you can connect to various websites. If you can connect and view websites, then skip the following troubleshooting section. Otherwise, read on.

Troubleshooting

When you can't connect to websites through a router but it works fine if you connect directly (a bad idea to begin with), then a couple of things could be happening.

The first thing to check is the obvious: Make sure the patch cable (CAT-5) is connected properly between the router and the Cable/DSL box. The patch cable should go from the port that says WAN on the router to a port on the Cable/DSL box. Also make sure to check that everything is powered properly.

The next thing to check is if your Cable/DSL provider has put a special MAC filter of their own and associated it with the IP. Every single networking device has a MAC address. Even the router has one. The problem is that the router's MAC address does not match any known/authorized MAC address. To fix this, go into the administrative interface on the router and locate the "WAN configuration" options (WAN = Wide Area Network...the device upstream, which, in this case is the Cable/DSL box). There should be a MAC address listed here. This is the address which gets broadcasted upstream. Now switch back to the Command Prompt and type in 'ipconfig /all' (without quotes) again. This time look for the wireless card's MAC address. Now switch back to the web browser. There should be a "Clone MAC address" feature with the wireless card's MAC address in the "WAN configuration". Use this option. When the router reboots, try connecting to websites again. NOTE: The MAC address you use should be the same one used when you used to connect to websites through the Cable/DSL box.

If the above fails, then you have probably run into the second problem: IP address range conflicts. The Cable/DSL box is issuing the same LAN (LAN = Local Area Network) IP addresses to the router that the router is issuing to computers on the actual LAN. In this case, the router is confused when it receives a request to connect to the outside world. To verify that this is the problem, directly connect the computer to the Cable/DSL box and use 'ipconfig /all' to see if the Default Gateway is the same as when you connect through the router. If so, to resolve this issue, go into the administration interface on the router and locate the "LAN configuration" options. What you are looking for is the base address of the router that matches the Default Gateway address. Here is where it gets tricky. Usually you will have a conflict of 192.168.0.* or 10.0.0.* (and rarely 172.16.0.*) - that is, both devices use the same first three numbers. To resolve this, simply increment the third number by one (1). So, 192.168.0.1 becomes 192.168.1.1. When you save this change, the router will reboot. A side effect of the change is that the web address to access the administrative interface changes to:

http://192.168.1.1/

Now try to connect to various websites. In most cases, this will work fine.

In the event that it doesn't work, you may be experiencing a synchronization problem between the new router and the Cable/DSL box. Power cycle everything. By this, I mean shut down the computer, router, and Cable/DSL box. Disconnect all cables (try to remember what goes where). Wait 30 seconds. Then plug the Cable/DSL box in and connect the line (from the wall) to the box. Wait for everything to be ready. Then plug in the patch cable between the Cable/DSL box to the router. Plug in the router and wait a bit for it to be powered up. Boot up the computer. Wait for the wireless connection to kick in and say you are connected. Try connecting to various websites.

If that fails, repeat, but leave everything disconnected for about 2-3 hours.

If that fails but connecting directly through the Cable/DSL box works fine, call your Cable/DSL provider and mention that you have a new router and "would like to know if there is anything special you need to do to get it to work because directly connecting through the Cable/DSL box works just fine". In particular, they may have a specific "MTU" setting. Whatever they tell you to do should be in the "WAN configuration" options of the administration interface.

If it still fails, then you could have faulty hardware. Especially if it works when you directly connect.

Final Security Check

Now that you are secure from the inside (including your pesky neighbors and wardrivers), let's see if you are secure from the big bad Internet. There are LOTS of bad guys out there and if you think your ISP is protecting you from them, you are wrong. If you sit on an open Internet connection with even a patched Windows-based PC (and even with many Cable/DSL providers!), you will be hacked and botnet'ed in under a minute..."up to one quarter of all personal computers connected to the Internet are part of a botnet" (botnets are usually protected by some form of rootkit making them nearly impossible to remove without reinstalling the OS).

The router you now have in place is there to defend you from all fronts at the IP packet level. So let's test that theory. Go here:

https://www.grc.com/x/ne.dll?bh0bkyd2

Scroll down the page past the useless stuff. Steve Gibson is a fanatic and not much of a security expert (moves his mouth a lot though - Bruce Schneier, on the other hand, is a REAL security expert) but his ShieldsUp! tool is extremely useful for finding any chinks in the armor of your router. Click the "Proceed" button. Click the "All Service Ports" option. Then sit back and wait as the website handles the rest.

A really good router will be completely "stealthed" (all green). However, many consumer routers have port 113 "closed". Port 113 is the identd service - some manufacturers do this so users won't complain about certain e-mail servers that require identd. If it was "green"/stealthed, sending e-mail messages could be very slow. Therefore it responds as being closed. This keeps the e-mail process moving along BUT at the sacrifice of having control over router security. Blame your manufacturer but also realize that port 113 isn't going past the router (i.e. requests stay on the WAN side - they never enter the LAN).

That's it! You're good to go. As far as consumer-level security, you've got the best there is for a minimal amount of tinkering.

If you want to do some more stuff, look into setting up firewall rules that block every outgoing connection except on specific ports that you use. This requires a fairly good understanding of how the Internet works in general. To try to explain it in English in a way that you could understand would take about two blog entries. Maybe more. And I would not try to explain it the same way Senator Ted Stevens did...The Internet is a series of tubes...it is not a truck...huh?

For commercial purposes and the adventuresome, take a look at RADIUS enabled hardware and software (along with various Linux/*NIX installations for certain router brands). The commercial hardware/software allows you to dynamically change the encryption key every 'x' minutes, which is a huge security enhancement. Also, you can set up those nifty redirection/login pages you see when you visit various establishments. You can also issue SSL client certificates for really secure setups. Actually, organizations are starting to put/are putting the wireless outside the LAN (usually squashed between two firewalls) and requiring people to VPN/SSH into the network to get on the LAN. This route is very secure.

This has been a very long message from your friendly neighborhood geek.

Cleaning a LCD display...

2007-09-03T16:15:00.000-04:00

It is interesting to note that many people don't really know how to clean things. This is especially true when it comes to electronic components. In particular, cleaning monitors on computer systems is a widely varied practice and no one seems to have a definitive answer on what the best method is. We pay a lot of money for our LCD flat-panel displays and then spray harsh chemicals on them that causes the display to go murky...where is the logic in that?

Some people might say that the safest chemical is water. However, is fluoride, found in most city water, really good for the plastic film on a LCD display? Probably not. How about all those minerals and "floaties"? Also, probably not good. Fluoride is for strengthening tooth enamel and the minerals probably contain corrosives. And water doesn't mix well with electricity and the delicate circuitry in the monitor.

Other people mention really harsh chemicals and household items as the solution to cleaning LCD displays. In particular, soap comes up often. Not only is soap harsh on the plastic surface, it tends to leave a nasty film behind that is usually worse/harder to remove than the original problem. Soap is for human skin, not delicate computer equipment. If you have ever gotten soap in your eyes, you know that it burns like crazy.

Also, soap contains numerous ingredients. Take Dove, for example, a Ph-balanced, hypoallergenic soap that contains:

- Sodium cocoyl isethionate
- Stearic acid
- Coconut acid
- Sodium tallowate
- Water
- Sodium isethionate
- Sodium stearate
- Cocamidopropyl betaine
- Sodium cocoate or palm kernelate
- Fragrance
- Sodium chloride
- Tetrasodium EDTA
- Trisodium etidronate
- BHT
- Titanium dioxide
- Sodium dodecyl benzene sulfonate.

I can't pronounce half of those chemicals. I wouldn't want them on my computer monitor. Would you?

Windex, on the other hand, contains ammonia (among other chemicals). Some people report success with this, but ammonia-based products have this amazing tendency to interact with plastics negatively (read: chemical interactions). LCDs have plastic...not glass coatings. Actually, I wouldn't even recommend ammonia-based products for CRT displays either. These displays usually have anti-glare coatings with a plastic to bond to the glass. Chemicals would damage this coating and possibly remove sections entirely from the glass, thus ruining the display.

There is a product called Klear Screen that is recommended by every major computer manufacturer. I have no idea what they put into it, but most people probably spray it right on the monitor, which will tend to do more harm than good. It also costs money.

Computer equipment is manufactured in a sterile environment and then used in non-sterile environments. People engineering LCD displays clearly don't think, "I wonder if we are making something the average person can clean?" Nope - their minds are clearly thinking of that sterile clean-room that Monk dreams of living in.

Here is how to clean any monitor safely:

1) Go to your local grocery store or Walmart.
2) Locate and purchase two items:

One or two "3M Microfiber Lens Cleaning Cloth"(s) (roughly $3 each).
A jug of Distilled Water (about $2).

3) Go home.
4) Put a little distilled water on the cloth so that part of it is damp and the rest is dry. (Not soaking wet!)
5) Clean the monitor. Don't apply very much pressure or you'll damage the display itself!
6) Dry the monitor with the dry portion of the cloth.

In most cases, this will clean the monitor (in some cases, just using the cloth by itself will clean it up nicely). If not, repeat once more.

Now you will remember I said water is probably a bad idea. There is a significant difference between tap water and distilled water: Almost all of the impurities are removed in distilled water. Tap water is healthy for people, but distilled water isn't. Distilled water is used in chemistry and biology experiments for its purity. I don't recommend using "double-distilled water" because it apparently interacts and reacts with carbon dioxide (which may or may not be "okay" for the monitor). Also, don't drink distilled water - studies have shown it to be detrimental to one's health if drunk exclusively - there are many important health benefits when drinking tap water that you don't get from distilled water.

The microfiber cloth is essential as well. It won't scratch the delicate surface. Generally you use these things for cleaning glasses and camera lenses (even digital camera lenses), but they also work well for computer monitors. Many people swear by 3M's microfiber cloth. T-shirts and most "soft" cloths are much harsher on the surface and will cause scratches over time.

If it still isn't clean, it is time to pull out better ammo. Go find a bottle of Klear Screen and purchase it. You can obtain it online and possibly at a local computer/office supply store. Again, just dampen your microfiber cloth with this mysterious solution. Repeat the procedure once more if necessary.

If the monitor still isn't clean, it is time to pull out the big guns. Go to the store and get a bottle of 90%-98% rubbing alcohol (about $3). Go home and create two different strength solutions in two sterilized containers:

1 part distilled water to 1 part rubbing alcohol. (Roughly 50% alcohol)
1 part distilled water to 2 parts rubbing alcohol. (Roughly 66% alcohol)

Try the weaker solution first using the microfiber cloth. Use a clean cloth - don't mix this solution with even dried-on Klear Screen - who knows what chemical reaction will occur! When cleaning the cloth via a washer/dryer, remember to never use fabric softener or dryer sheets (they will both ruin the cloth)!

If that fails, try the stronger solution. Remember! Never apply too much pressure or you'll damage the monitor itself (assuming a standard LCD monitor)!

If all of these methods fail, either live with the problem, get a new monitor, or go to a professional computer repair shop and see what they can do (they'll have access to chemicals consumers don't have access to - probably take them five minutes to determine if it is even feasible to attempt to clean).

DO NOT EVER use any ammonia based or strong household cleaners to attempt to clean a computer monitor screen. You will more than likely cause irreparable damage to the surface.

This has been a friendly computer geek announcement.

The new PayPal buttons...

2007-09-01T02:36:00.001-04:00

...are the ugliest things I've ever seen. The people PayPal has employed are clearly not graphics artists. And also don't have the web developer in mind.

About a couple weeks ago every business customer received a "teaser" e-mail from PayPal saying to get ready for new logos and buttons for use on websites. Mentally, I thought, "Sweet! Maybe they won't stink like the current ones." Today, PayPal sent every business customer an e-mail saying the logos and buttons were ready for use on websites and sent us all to their website via a link.

I clicked the link and, lo and behold, awful-looking images stared me in the face that are worse than the old ones (but that isn't saying much). The image above is the worst of the lot, but they are all pretty bad. Let me name off my reasons:

1) Every last image is a GIF image. Which makes the gradient look awful, severely limiting color choice, and offers extremely limited transparency options. PNG - enough said.

2) There is an ugly white border around the entire image. This makes it impossible to put the image on any color without making the ugly border instantly stand out.

3) Placement of the "icons". Oh where do I begin? It looks like a 'V' shape which just makes everything feel awkward. The icon sizing looks like a child did it. At least make them all the same size - bicubic scale them up to the largest length/width in each direction and then scale down. Yes it will make them look funny. But there is this tool called Photoshop that can do magical things like edit the images to extend them. At that size, no one will notice and everything will be pixel-aligned.

4) Speaking of Photoshop. Let us have PSDs of the logos and buttons already. That way WE can decide what the buttons look like and what file format. And, surprise! We don't like your buttons. You stink at graphics arts. Go back to school.

5) What is up with the extra inner space - especially all that space under "Bank"?

6) Putting PayPal above the icon work is just a bad idea. Sure it emphasizes that PayPal is being used, but it is completely unnecessary. All people want to know is "What sort of online payment methods does your business accept". Put PayPal first for all I care in the ordering, but PLEASE do not look ugly while doing it. Either completely encase the PayPal logo in the orange border or put it inside the box and make the border at the top thinner. What you have currently is plain ugly.

7) 3 of 5 of the icons have borders (natural or otherwise). Make them all have borders even if they normally don't. The "Visa" icon, in particular, looks awful against the orange background.

Digg this

Google Toolbar 100% CPU bug...

2007-08-30T12:24:00.001-04:00

Digg this

In general, I'm a pretty big fan of the Google Toolbar for IE. I use IE6 SP2 for pretty much all of my web surfing needs. I have Opera and Firefox installed too with plugins/extensions for the latter, but I just like IE better (although some sites are starting to be IE6-unfriendly). But this isn't about starting a "which browser is better" debate/flamewar.

Instead, it is about a little bug I found today while minding my own business. Actually, I found three bugs, but only one of them is a critical issue that should never have left Google's QA department.

So I was minding my own business and responding to e-mails and other such things. For many of the groups I'm on, I tend to do web searches just so I can paste a link or whatever into the e-mail. I run into an e-mail that I had indirectly answered on another mailing list several weeks ago and didn't bother pasting links so I could search the archives. But I left myself enough clues that I could figure it out again if I needed to later.

It was a pretty obscure topic so finding the links again was difficult. So, I decided to look through the Google Toolbar search history. The complete history. Google Toolbar saves every search you ever do...and I've never bothered to clear my history - so it is really lengthy.

Anyway, here are the three bugs in increasing order of how critical it is. I assume using a new browser session to make things easier for each test case:

1) Make the combobox dropdown window huge by grabbing the gripper and dragging. Now scroll through the history using the weird down arrow. The entire display flickers. This is a repainting issue.

2) Select a previous search item. Now use the down arrow button on the combobox to open the dropdown window. Scroll down a ways using the weird down arrow. Now move the mouse up but don't click anything. Move the mouse back over the weird down arrow. The history scroll starts over at the top of the history instead of continuing from where it left off.

3) Close all open browsers. Open Task Manager (trust me, you'll want it). Open a single IE session. Click the down arrow button on the combobox to open the dropdown window. Move the mouse over the weird down arrow and let it scroll through the history. As it scrolls, the current 'iexplore.exe' will slowly increase how much CPU it is using until 100% CPU is being used. Closing 'iexplore.exe' will close the window but the process will have to be manually terminated. If the end of the history is reached before maxxing out the CPU, use the bug from #2 to start at the top and continue (or use the weird up arrow).

Heh. I caught three bugs QA at Google didn't. [Sigh] Why is it I run into all the bugs no one else finds? Every last piece of software.

Labeled a spammer...

2007-08-30T12:03:00.000-04:00

Oh this is just great. I just got labeled as a spam blog. Here is the warning that I received when I logged into my blog today:

--------

WARNING

This blog has been locked by Blogger's spam-prevention robots. You will not be able to publish your posts, but you will be able to save them as drafts.

Save your post as a draft or click here for more about what's going on and how to get your blog unlocked.

--------

Here is what blogger says about why this happens:

--------

What We're Doing About Spam

Needless to say, we do not approve of spamming here at Blogger. Below are some of the things we've implemented to remove and reduce spam on our service. We will update this list as we continue our efforts.

Automated spam classifying algorithms keep spam blogs out of NextBlog and out of our "Recently Published" list on the dashboard.

The same classifiers are used to require an extra word verification field on the posting form for potential spam blogs. This makes it harder for spammers to set up automated systems to do their posting, since a human needs to complete this step.

The Flag as Objectionable button in the Navbar lets you notify us of problem blogs that you find, so we can review them and take appropriate action.
--------

Let's take a look at why this might have happened from least likely to most likely:

1) The sudden spike in traffic and comments.

2) A whole bunch of people clicked the "report this blog/flag as objectionable" link out of sheer spite. That would imply people don't like me...but, but, but...everyone likes me! I can imagine Microsoft employees doing that for discovering their secret Windows Update but they would first have to find this blog. That isn't likely either.

3) Yesterday's post on a new type of spam. But there is other content in that post, so it shouldn't be a problem unless they have lousy filters.

4) The whole "old hash"/"new hash" thing in the modified secret Windows Update post. I can see this as a major problem for various anti-spam algorithms. It does look like gobbledygook despite being perfectly legit.

Yet another reason I'm seriously disliking Blogger. Ooh! I've got a great idea. Let's add gasoline to the fire! If I get any more ticked off, I'll end up heading over to weblogmatrix.org and get something better.

Edit: Took almost 24 hours for Google to get around to declaring my blog spam-free.

A new type of spam

2007-08-28T14:27:00.000-04:00

Digg this

Just got this in my e-mail in-box:

---------------------
We Need Beta testers to try out our new software Office Tools Plus

This will help us get the software ready for consumer release. For helping out, you will receive a free edition and 5 years of updates.

1: Download the software 2: Try it 3: Tell us what you think Here is your chance. Follow the link to our secure download center:
http://68.202.*.*/setup.exe
---------------------
(IP address removed for obvious reasons)

A new type of spam has appeared. Instead of saying "Hey idiot, download this perfectly obvious EXE that is going to install a virus" they are covering it up with "You can be a beta tester for our new Office Tools product....the download for beta testers is here: [link]". Social engineering at its finest.

The average user who has heard of beta testing will probably be enticed into downloading the file and running it. Who doesn't want to be a beta tester for a product that won't cost them a penny and possibly improve their productivity?

This is a dangerous new type of spam. A number of people are going to fall for it - it plays on social desires to beta test/try out new products. My guess is that it contains botnet software. On the plus side, a lot of users won't fall for it because there is no associated "click here to learn more about this software" sort of link that takes them to a page that describes how the software works, complete with screenshots. I'm sure the spammers will eventually start doing that, but it'll take a while.

Of course, I use an anti-spam tool called Spambayes. It is free software and works really well. I just have to train it a couple times on messages like the above and all future messages that are similar in nature will vanish.

And don't bother telling me the merits of Linux, Mac OSX, etc. I've heard it all and I've even used them both.

A Most Coincidental Event!

2007-08-25T20:33:00.000-04:00

Digg this

Yesterday I spoke of a most heinous act of computer modification. Today Microsoft spent a huge chunk of the day attempting to solve a major problem.

http://www.boingboing.net/2007/08/25/microsoft_wga_server.html

The WGA (Windows Genuine Advantage - a.k.a. "Disadvantage") servers went completely down. Now I'm not a huge believer in coincidences but if Microsoft has ever had its pants down, this is perhaps a double helping (free wedgie!). Let's see here:

1) Push a secret worldwide update to Automatic Updates out to every computer on the planet.
2) My computer receives the update and VerifyMyPC flags it.
3) WGA servers receive the update completely unaware of what is happening.
4) WGA servers barf (perhaps something in the update they didn't like). All of the WGA servers go down.
5) User PCs attempting to connect to WGA servers can't and therefore are flagged as pirating Windows.
6) Microsoft catches wind of the problem and employees responsible for WGA head into work to solve the problem...and spend most of the day scratching their heads.

A PR disaster in the making if I've ever seen one. Had they had VerifyMyPC deployed throughout their organization, they could have avoided it or at least dealt with it a lot sooner (such as figure it out in 5 minutes instead of wasting hours in the office on a Saturday...time better spent at home).

Edit: Step 4 in the "sequence of events" is kind of vague. There is ALWAYS a reasonable explanation for what happens in a computer - it is just circuits and electricity after all - 0's and 1's. At the time I couldn't think of anything that would trigger a shutdown of the server. I was thinking more along the lines of "some application crashing or BSOD'ing" instead of, well, more reasonable ideas. After I thought about it a bit, perhaps step 3 made an incorrect assumption.

Suppose, for instance, you are in charge of the WGA servers and you are thinking about what hackers will consider a terrific target. The main Microsoft website is high profile but also extremely risky but breaking into WGA would be a great way to mess with a whole bunch of people at once and is a much more "backwater" system. So, as the server manager, you look for what is known as an Intrusion Detection System (IDS) and install one on the server. Then you set up a rule that says, "Should a file change, shut off this computer." Then you set up Automatic Updates to manual download and install (i.e. Ask me before doing either one). Then, you put into place a policy that when there is an update available via Automatic Updates one of the engineers (or a script) is to turn off/disable the IDS rule, run the updates, and turn the rule back on. This policy is then applied to all of the WGA servers to make the whole thing easier to administrate simultaneously.

Now Microsoft is huge and the left hand doesn't always know what the right hand is doing. So the Microsoft group responsible for Windows Updates releases a secret Windows Update that bypasses even the manual settings in Automatic Updates.

Every computer, including the WGA servers all download this update and install it. However! The IDS picks up on the fact that critical Windows files have changed. Each system then executes a perfectly flawless shutdown as per the rule set in the IDS system. Wam! Bam! WGA is completely down. The reason the engineers spent half a day in the office was probably to figure out what triggered the IDS rule to fire in the first place - even then they possibly didn't figure it out (depending on how good the IDS is - VerifyMyPC caught it on my system). Still, that's over half of their Saturday wasted.

I can see a number of people getting yelled at over this:

1) The Windows Update group responsible for the whole mess. First for issuing a secret worldwide update. And then for getting caught.
2) The WGA server group for having a single point of failure that can cause the servers to all go down.
3) Those who programmed the client-side of WGA for assuming that if the WGA servers are unreachable, because the servers are all down, that the person is pirating Windows.
4) The support group (in India?) who said that the WGA servers would be back up sometime on Tuesday.

Windows Update updating without permission!

2007-08-24T23:02:00.000-04:00

Digg this

Did I ever mention that I love VerifyMyPC? Oh wait. Never mind. I did that already.

It has been a while since I have posted but this one is too good to pass up. Every night around 10:30 p.m., my computer is set up to run a VerifyMyPC scan. About 11 p.m. the Scan Notifier runs and does the whole balloon pop-up thing. Normally nothing pops up because there is nothing to report (i.e. another day at the office - figuratively speaking).

When there is something to report, usually a little yellow triangle icon shows up and I say, "Yup, I remember doing that today." Or, "Those changes to my system sound about right."

Tonight, the special analysis mode of the Scan Notifier picked up on unusual behavior and popped up the Red-X icon.

If Microsoft ever wanted to get caught with their pants down, they succeeded. For most people, the above doesn't make a whole lot of sense past the "you might have a virus" part. VerifyMyPC requires a little extra knowledge about computer systems when dealing with the details. Google is your friend in these cases. Running searches for 'wups.dll' and 'wups2.dll' turns up something about Automatic Updates. In particular, those DLLs provide Automatic Update functionality for Windows.

In other words, the Automatic Updates utility automatically updated itself. Now this might not seem like a big deal but I have automatic updates set to manual (both download and installation have to be approved by me) and not the usual 'automatic' setting found on most user PCs. In other words, Windows updated itself without my express permission. Such behavior is right in line with spyware-like activity. Thus, VerifyMyPC is doing an accurate job in reporting such behavior to me. I love VerifyMyPC.

It is also interesting to note that Microsoft pushed out an update to Automatic Updates on a day other than the 2nd Tuesday of the month (also known as "Patch Tuesday").

Edit: The above image actually indicates that those files were 'added'. Drilling down, it shows that they were added to 'C:\WINDOWS\LastGood\system32\'. While 'wups.dll' and 'wups2.dll' were NOT modified, other files that are in the real system32 directory ('C:\WINDOWS\system32') WERE modified. What follows is a snippet of each file that was added and changed (files with the same name have been grouped together to help make it obvious that a virus or other piece of malware wasn't involved - malware authors wouldn't bother to copy the files to the "Last Known Good" configuration):

Add (Important)

C:\WINDOWS\LastGood\system32\cdm.dll (90.33KB)

Hash: 4E 68 B2 C4 4D F7 D2 58 16 8C 99 2C BA EC E9 95 53 33 05 86 C2 81 3B F4 B9 27 87 7C 0B 5B 51 A5

Change (Critical)

C:\WINDOWS\system32\cdm.dll (90.33KB)

New Hash: F2 2D 36 39 25 2C 01 76 40 0B 49 B3 06 2E B0 18 4B F1 F6 66 34 DD C7 F8 FD 69 73 23 9B CD 5B 98

Old Hash: 4E 68 B2 C4 4D F7 D2 58 16 8C 99 2C BA EC E9 95 53 33 05 86 C2 81 3B F4 B9 27 87 7C 0B 5B 51 A5

Add (Important)

C:\WINDOWS\LastGood\system32\wuapi.dll (536.83KB)

Hash: 07 A5 AF 93 9A 1D 28 5F 5B 08 BC 43 9B E5 57 EF 00 1C 4A D6 D9 E3 92 10 33 B2 D7 B9 E9 2C 42 C0

Change (Critical)

C:\WINDOWS\system32\wuapi.dll (536.83KB)

New Hash: C6 D8 44 CF CF BE 21 DA D0 3A 6E 75 7A A7 7B 06 DC 4E 3E 06 06 41 8B F9 E7 9D 91 13 29 17 5E C0

Old Hash: 07 A5 AF 93 9A 1D 28 5F 5B 08 BC 43 9B E5 57 EF 00 1C 4A D6 D9 E3 92 10 33 B2 D7 B9 E9 2C 42 C0

Add (Important)

C:\WINDOWS\LastGood\system32\wuauclt.exe (51.83KB)

Hash: A4 21 0C 3D 8A 99 75 97 E5 67 0B FA C2 46 6E 6A 0A FD C8 9B 2F 2F 6F 9C E5 88 63 3F 92 67 A5 9A

Change (Critical)

C:\WINDOWS\system32\wuauclt.exe (51.83KB)

New Hash: 46 DA FC 71 5B C2 BC BF D5 6A 3B 2B C3 DF 1D D2 C0 36 89 3E AB 2E 4F D6 E4 39 3E 08 10 54 D5 0D

Old Hash: A4 21 0C 3D 8A 99 75 97 E5 67 0B FA C2 46 6E 6A 0A FD C8 9B 2F 2F 6F 9C E5 88 63 3F 92 67 A5 9A

Add (Important)

C:\WINDOWS\LastGood\system32\wuaucpl.cpl (211.33KB)

Hash: 68 10 5C D1 BA 1D 73 48 02 31 DE 4C C0 F3 08 CF 15 3E EC 5B C9 F4 4D 2C 22 D0 D6 03 D8 59 C1 99

Change (Critical)

C:\WINDOWS\system32\wuaucpl.cpl (211.33KB)

New Hash: C4 0D 02 69 98 E1 9F 23 9F F9 5A 55 C1 33 4A E4 70 5A 8B 92 BF 4D DD F0 E4 42 3E 4F DA E9 D0 DA

Old Hash: 68 10 5C D1 BA 1D 73 48 02 31 DE 4C C0 F3 08 CF 15 3E EC 5B C9 F4 4D 2C 22 D0 D6 03 D8 59 C1 99

Add (Important)

C:\WINDOWS\LastGood\system32\wuaueng.dll (1.63MB)

Hash: 47 4F E9 97 52 0A 5C EC B5 CD ED 16 2B 32 49 61 AE 43 27 84 B1 82 11 66 6D D4 51 70 8A E6 C4 CD

Change (Critical)

C:\WINDOWS\system32\wuaueng.dll (1.63MB)

New Hash: 43 C2 26 22 FF C5 7E 8C 4F 54 C0 58 DA 30 D8 EA 57 BC 28 FF 43 CC 5C 85 17 DE C2 47 FF 2E 71 2A

Old Hash: 47 4F E9 97 52 0A 5C EC B5 CD ED 16 2B 32 49 61 AE 43 27 84 B1 82 11 66 6D D4 51 70 8A E6 C4 CD

Add (Important)

C:\WINDOWS\LastGood\system32\wucltui.dll (318.33KB)

Hash: 15 1D 34 E5 A4 3A CC DA B4 93 86 50 A0 99 70 6A 6B 6C 8E A5 D2 C5 83 25 EF 36 D1 AA 3B 46 9F 7B

Change (Critical)

C:\WINDOWS\system32\wucltui.dll (318.33KB)

New Hash: 51 12 24 6C 7B 09 54 21 ED 41 FA 90 B4 E8 CE 9D 00 3C DF A9 2F B1 DF 71 89 B8 CE 68 2D 8A 63 F7

Old Hash: 15 1D 34 E5 A4 3A CC DA B4 93 86 50 A0 99 70 6A 6B 6C 8E A5 D2 C5 83 25 EF 36 D1 AA 3B 46 9F 7B

Add (Important)

C:\WINDOWS\LastGood\system32\wups.dll (32.83KB)

Hash: E2 E1 5F 1C FB 8D 3F 38 15 89 F4 A1 05 6C 7C 22 6B 6A 54 EA 9A D4 FE 49 77 CE B4 96 8D EF 8E BF

Add (Important)

C:\WINDOWS\LastGood\system32\wups2.dll (42.33KB)

Hash: EF F0 03 E7 79 2B 94 C2 F5 3D 90 07 FB 9D 71 AD 2E 2D 3F 00 BB 8E B9 59 16 C3 F5 21 04 D9 7E FA

Add (Important)

C:\WINDOWS\LastGood\system32\wuweb.dll (198.33KB)

Hash: 12 72 88 FA C2 76 75 C4 51 69 A2 E3 BC B6 94 4B B3 91 C8 49 78 BC 2F DE 85 C5 B2 C4 2B D3 7B 93

Change (Critical)

C:\WINDOWS\system32\wuweb.dll (198.33KB)

New Hash: 5F B2 3D 83 EE 94 20 A6 0F 23 8F BF 5F 7E DD BC A6 8F 9A 9A CE 35 A8 F9 64 AF 88 A9 4D 4B E0 7C

Old Hash: 12 72 88 FA C2 76 75 C4 51 69 A2 E3 BC B6 94 4B B3 91 C8 49 78 BC 2F DE 85 C5 B2 C4 2B D3 7B 93

(The rest of the files have a .mui file extension and MUI apparently stands for "Multilingual User Interface" - probably just a bunch of language strings).

Change (Critical)

C:\WINDOWS\system32\wuapi.dll.mui (25.33KB)

New Hash: 42 46 98 4C AE 03 50 61 F4 E9 69 7A A2 38 A4 4B B3 A8 40 F1 39 3F 71 A7 92 78 42 28 5F 8F B9 33

Old Hash: 73 B4 BB 37 D4 FF 47 0B 61 78 73 AA 43 24 12 27 2C D4 B3 B2 9C 8E 6A 26 A6 78 1E A7 08 25 B5 36

Change (Critical)

C:\WINDOWS\system32\wuaucpl.cpl.mui (25.33KB)

New Hash: B1 6B F1 A9 5F 88 6F B1 8E B3 60 E6 42 2B AF B1 00 2D 9C 8A F1 17 C8 0D 6D 0E 23 24 6C CA 60 D4

Old Hash: EF E0 8D 82 AE F1 56 9B 55 C7 B6 CD CE 28 80 3F B7 26 20 84 EF 5C 4B 69 40 17 9C 4E 2F 67 97 58

Change (Critical)

C:\WINDOWS\system32\wuaueng.dll.mui (19.83KB)

New Hash: D9 B6 D9 FB 33 EA CB F3 DA 38 19 86 62 FE 70 16 6E 74 BC DC 4A 67 AD 24 A3 8A F8 8C 23 42 BA FB

Old Hash: D0 19 EC DA 02 E1 9F FD 30 C4 F4 06 90 A5 0F 97 76 59 81 B2 3A F1 BE AD 60 47 25 E5 63 7C 33 9B

Change (Critical)

C:\WINDOWS\system32\wucltui.dll.mui (33.33KB)

New Hash: 22 93 81 37 4F A2 81 38 D4 FC FB 07 69 A2 1F 6A 5D C5 7A 5C 44 78 F4 75 C0 3C 04 DC 6A 9C 45 B0

Old Hash: E3 BD 08 48 2F BF 98 68 AF 78 C9 17 A4 1B 1C 4E AD 64 D3 18 ED C5 06 BB 87 A2 93 52 2A A1 C5 F3

So there are plenty of other actual changes to Automatic Updates to back up my claim.

Also, while wups.dll and wups2.dll were not changed, it is pretty apparent that they were included in the update as they were backed up into the last good configuration directory...as if they were going to be changed. Also, VerifyMyPC only reports changes to files that have signature (hash) changes. A hash is a one-way cryptographic thumbprint of a file. If you want to verify the above you will need a tool capable of performing a SHA-256 hash and a computer you didn't reboot (last good configurations tend to vanish after a successful boot).

You should also keep in mind that there are Windows APIs to alter timestamps of files. Just because a file says it hasn't been modified or accessed since 2004 doesn't mean it hasn't been.

Update Sept. 14, 2007: Microsoft finally responded after some major publications also realized secret Windows Updates were pushed out...almost three weeks after I posted this. Here is the official response.

To this I say: "That is a bunch of baloney". If Microsoft wants to update Windows Update components, I want the choice to update that. The "Download and Install Notifications" option implicitly includes all updates. In my mind, the Windows Update utility itself is part of that 'all'. Don't update my system secretly. Ever.

And Microsoft still hasn't come forward to explain why the WGA servers went down. My guess is that would still be pretty embarrassed at this point to try to explain that "because they pushed out a secret update to Windows Update, WGA went down".

While I generally accept updates to Windows, I still want complete control over the entire process. The biggest problem I see with secretly updating is that it usually entails a reboot. I rarely reboot and if my system reboots while I'm in the middle of something, I will potentially lose a lot of work not to mention the time involved in bringing up all 20-30 programs I was running before the reboot. Secret updates might be followed by random shutdowns and reboots.

ASP Software Drawing at SIC 2007

2007-07-06T17:11:00.000-04:00

So this year I won't make it (again) to SIC. The 2007 Shareware Industry Conference is probably going to be lively this year given that this is also the 20th Anniversary of the ASP (Association of Shareware Professionals):

http://asp-shareware.org/

This year the ASP has a hospitality suite: Expensive, spacious, and only a few available each year. In other words, they went all out. Crazy people, those ASP members. Part of the events that will take place in the suite is a drawing in which they give away registered versions of shareware products. So while I won't actually be at the conference, my products will be. Let's see, I sent 5 CDs to the person in charge:

2 copies of VerifyMyPC 2.6 (registered version, of course)
2 copies of MyTaskFocus 1.1 (registered)
1 year license of MyUpdate Toolkit (a $250 retail value)

If you add it up, that's $310 worth of software product (excluding burning*, shipping, and handling costs). For me, being a small software company and all, I think that is pretty generous. I just hope the winners won't just stick the CDs on a shelf. I'd really like some good, positive press about my recently (as of yesterday) updated product line.

* Due to my lateness, I had to burn to LightScribe media - costs about $1 per CD and I had a bad LightScribe burn on one of the discs so I actually ended up using 6 discs.

Anywho, if you are going to SIC this year, be sure to blog about it. If you aren't going, read blogs and drool as thousands of dollars worth in software is given away courtesy of generous people like me. If you don't know what a blog is, this is a blog.

Microsoft patent (un)happiness

2007-06-25T12:51:00.000-04:00

http://mcpmag.com/news/article.asp?editorialsid=1320

Microsoft is out to make money. As are most businesses. Money is required for the basic essentials of life and I personally believe Open Source cuts into that. Until food, clean water, shelter, and clothing are free for everyone, Open Source is a great idea in principle but a bad idea in practice. The only thing you can do to make money off of Open Source is to turn it into SaaS...but how long can that model _really_ hold up? Making Linux and Linux-based products easier and easier to install (e.g. Ubuntu, OpenOffice) makes it more available to the masses and IT folk to do it themselves but if a person can't eat, drink, sleep, and they can't afford clothes (naked?)...is it worth it?

I don't have the answer to the question. Most programmers don't think about what effect their software will have on other people. Will developing for Open Source eventually cause all software developers everywhere to eventually lose their jobs? That's a loaded question but one that crosses my mind often. I use Open Source projects under Windows: FileZilla, TortoiseSVN, Subversion, Thunderbird, Firefox, and others. There are non-free products out there as well. Am I contributing to someone losing their job because I'm using Open Source?

Until someone figures out how to eliminate money from being required for the basic necessities of life, developing for Open Source, as a business model, is not viable. Microsoft is trying to kill off Open Source because they realize this and are trying to save millions of jobs in the process. The way they are executing it isn't exactly kosher, but can you blame them? I can't and I usually have something bad to say about Microsoft.

If you want your comment to get through on this post, it needs to contain the solution to the problem of money. Eliminate money (and all forms of trade) and Open Source becomes viable. I already know the solution but executing it is going to require 2,000 people, 5 years, and (ironically) $42 billion (US).

How to get squish just like grape

2007-06-19T11:52:00.001-04:00

When the other shoe drops...

http://www.channelregister.co.uk/2007/06/06/microsoft_mvp_threats/comments/

...it is going to hurt. A lot.

So this guy wrote a pretty popular add-in for Visual Studio .NET called TestDriven.NET. I've actually heard about this add-in prior to the whole mess he has currently got himself into so it is definitely popular.

Summary of how it has gone down thus far:

1) Developer creates add-in for VS.NET via COM because a VSIP license is expensive. Nothing in the EULA explicitly prohibits it.
2) People like the add-in and it becomes popular.
3) Microsoft gives him MVP status and then discovers the add-in works for VS.NET Express and asks author to remove support for Express.
4) Author refuses.
5) MVP status is revoked.
6) Author adds support for VS.NET "Orcas" Express.
7) Microsoft legal makes its move.

Frankly, VS is Microsoft's intellectual property, not the developer's. However, there are two parties at fault here:

1) Microsoft is at fault for leaving add-in COM support enabled in VS.NET Express. By doing so they left the door wide open for add-ins to function even though that wasn't their intention.
2) However, the developer is also at fault for not removing Express support from his product when Microsoft requested it. Microsoft only turned nasty after the developer refused.

The author shot himself in any legal foot he potentially had when he specifically added support for "Orcas" Express - a version of VS that isn't even out yet. To say publically that he had to add support means it wasn't working for some reason. That's all Microsoft legal needs to squash him like a bug. Or a grape.

Microsoft is out to make a profit. Anything that reduces sales for them is a thorn in their side to be removed. Going against Microsoft without billions of dollars in backing is about the stupidest thing anyone can do. Standing on principles alone is dumb - sure people are cheering, egging him on to go up against the software giant. But David was a slingshot expert when he went up against Goliath. That developer is winging it with almost no knowledge of the law - he'll have to be the luckiest person alive to just survive the onslaught that is coming let alone have any hope of winning. There is no backing out now either...he ensured himself of that when he included support for "Orcas" Express.

Let this be a lesson to all of you developers out there. Be smart. Don't do stupid stuff. Don't become the grape. Or if you do, figure out the fastest way to roll down the nearest storm drain to avoid being squished.

Why Windows Error Reporting (WER) does not work

2007-06-19T00:30:00.000-04:00

If you are reading this, you are probably coming from the CubicleSoft website to learn more about Windows Error Reporting. This blog entry will thus be a little more professional as a result.

Windows Error Reporting, or WER for short, is a set of technologies Microsoft put together for Windows XP and expanded upon for Windows Vista:

http://en.wikipedia.org/wiki/Windows_Error_Reporting

To summarize the Wikipedia article, WER gathers error reports in a central location (Microsoft servers) and developers of software then can log into the system and retrieve those error reports and thus fix bugs. End-users of Windows see something like this when the application crashes:

The user clicks the "Send Error Report" and the bugs get fixed.

The official website of Windows Error Reporting (WER):

https://winqual.microsoft.com/

All that sounds good in writing until developers start reading the "fine print" on what is required. By default, applications are not WER enabled. That means the WER data sent by the user is completely ignored by the WER system (i.e. information about the crash simply gets thrown out). One would think every software author would be on board with this WER thing.

So, what exactly does it take to integrate WER? First and foremost, to just gain access to the system requires a Verisign Class 3 Digital ID. That costs $400 [US] per year. That staggering pricetag alone causes most developers to be unable to join the WER program in the first place.

Secondly, once a developer gains access to the system, they discover they have to modify their source code. Significantly. Windows Error Reporting requires integration with various Windows APIs and many of them are quite difficult to use correctly. When used incorrectly (easy enough to do so), wrong information about the crash can be sent, or in the worst case, cause a second crash to occur.

Lastly, the developer has to create two installers: One being the usual release. The other containing the release plus the release's PDB files. A PDB file is used by a debugger to tell where in the original source code the program is at when debugging the program. Without the proper PDB files, the developer can't use any crash reports.

Those are just the steps to integrate WER. It is a huge undertaking and quite expensive. But that's just the start. Just integrating WER isn't enough. The developer has to "baby-sit" WER. Here's why: The WER system only tallies crashes until it is told to start collecting data for a specific crash. Only once WER has been to start collecting data does it do so. Until it is told to collect data, when the user hits the "Send Error Report" button, the error data is simply ignored.

Once the developer finally retrieves data from WER, the data is just a stack/variable dump and whatever information they put into their software to pass onto WER. There is a very good chance that the data will be completely useless. Could be anything from a bad PDB/EXE matchup, to a bad stack dump, to threading issues, to not having the necessary symbols for some system binary not available to the developer, to not knowing what the user was doing at the time, etc. Probably something along the lines of a 75% failure rate.

Despite these huge hurdles, some non-Microsoft companies DO use WER. Off the top of my head, Valve Corporation uses it in their Steam client. Their games still crash and, well, the bugs haven't been fixed yet. Microsoft uses WER for their own products and actually fix bugs, but that's about the only exception.

The end result is that no one sends error reports to the Microsoft WER server mostly because people have figured out that doing so is a waste of time. Now you know why it is a waste of time.

Hopefully I didn't go too far over your head with this blog entry. I did get kind of technical but if you came here from CubicleSoft, you wanted to learn more about WER. The Crash Reporting Support Tool that is used by CubicleSoft bypasses WER and sends error reports directly to my inbox. It is an elegant solution that does not have the problems WER has and it actually works!

http://www.cubiclesoft.com/Support/

And out of the blue...a job offer...

2007-06-13T21:16:00.000-04:00

Out of the blue I received today in my in-box (just now checking my e-mail - yes I'm still running on empty in terms of sleep) a job offer from Google. Well, not an offer per se, but somehow, somewhere I made a significant impression.

The amazing thing is that I haven't sent my resume anywhere. In fact, it is sorely out of date - been busy with the whole CubicleSoft thing. A Google Internet recruiter came to me. Could have been the article I just put up on CodeProject, but who knows? (Maybe I'm more important than I think I am...don't let that go to my head now :P ).

I'm not going to do anything about it today. I'm too tired. And likely to royally mess up something I'd be smacking my head against a brick wall over for the next 10 years. Best to get some sleep first before doing or saying anything I'll regret. Amazingly enough, I'm still pretty lucid.

UAC: The Definitive Guide

2007-06-13T11:00:00.000-04:00

I'm operating on zero sleep in the last 24 hours as I write this. So I'm going to keep this short. I just finished publishing a new article on CodeProject.com on Vista UAC. I call it the definitive guide because it combines every last bit of knowledge I've got on Vista UAC elevation, provides a really cool package called Elevate, and, well, it is everything a software developer needs to know about UAC and its quirks and workarounds...without having to spend weeks on hunting down the information:

http://www.codeproject.com/useritems/UAC__The_Definitive_Guide.asp

Plus it is by an author of a book called Safe C++ Design Principles. Oh wait. That's me. I must be tired.

While you are reading the above article, be sure to listen to:

http://www.jonathancoulton.com/mp3/Re%20Your%20Brains.mp3

Solving pesky LNK2005 errors...

2007-06-09T18:28:00.002-04:00

For those who read this blog and aren't technically inclined or simply don't use Microsoft Visual Studio (e.g. you use a different compiler suite), this entry isn't for you.

One of most annoying things to run into in Visual C++ are linker errors. They are obtuse, poorly documented, and double-clicking them doesn't take you to the source code (or the part of the object file) where the problem is occurring. One of the most confounding error messages is the LNK2005 error message. Usually something like this shows up:

nafxcwd.lib(afxmem.obj) : error LNK2005 ...something about operator new/delete goes here...

If you search Google, a Microsoft Knowledgebase (KB) article pops up (KB148652):

http://support.microsoft.com/kb/148652

A lot of people run into LNK2005 errors, find the above article, try out what it says, and discover the "solution" makes the problem worse, not better. That is the only KB article I've ever seen that provides an incorrect solution for a problem...usually Microsoft is pretty good about giving right answers.

There are four causes of LNK2005 errors that I've run into and actually fixed:

1) A bad build. For some reason or other VS just occasionally barfs. The solution in this case is to close down VS, wipe all the temporary files (e.g. .obj, .dll, .exe, autogenerated MIDL .c files, etc.), load up VS, and do a Build->Clean Solution, Build->Rebuild Solution. Depending on the size of your solution, this can take a while - go grab a coffee.

2) If it still fails with LNK2005 errors, then something is mismatched. Right-click on each of the projects and select "Properties...". Then make sure the "'C++'->Code Generation->Runtime Library" all say the exact same thing. If you attempt to use different runtime libraries, it may or may not link properly. The reason it will not link is if you allocate memory in one project and then pass it to another project and free it there (sort of like DLLs but weirder). If you are like me and have projects shared between solutions, you will want multiple project compilation types so that you don't have problems with linker errors (I have 6 of 8 possible for my base library - the base library is multi-threaded - I say 8 possible because you have /MD[d] with MFC, which is different than /MD[d]). Once you have types matching (don't forget to match the "use MFC" option if you use MFC!), make sure both debug and release mode are configured correctly with the configuration manager (Right-click on "Solution 'solutionname' (x projects)" and select "Configuration manager..."). Also, make sure the dependencies and build order are correct (same right-click menu but "Project dependencies..." and "Project build order..." instead).

3) If you've done all that and still get LNK2005 errors, then you've hit a scenario I like to call the "duh" scenario. An incorrect entry point for the subsystem selected will raise a LNK2005 error message. So, if you defined main() and don't use /SUBSYSTEM:CONSOLE, you'll get a LNK2005 error. Or if you defined WinMain() and don't use /SUBSYSTEM:WINDOWS, you'll get a LNK2005 error. Again, this doesn't always happen, but when it does it is weird (usually complains about new/delete duplications across object files instead of main()/WinMain()). To change the subsystem, right-click on the startup project and select "Properties...". Then go to Linker->System->SubSystem. Change accordingly to the entry point used.

4) If, even after all that you still get LNK2005 errors, then it is time to scratch your head and think, "Gee, did I or any third party put in any include files that force .lib files to be included in the linking stage?" Start scouring that source code for any #pragma comment(lib, ...) statements. This happens particularly with MFC libraries since there is code floating around out there that forces the MFC libraries to be linked in at link time first and then forces the other libraries to be excluded - including such code with a non-MFC project causes all sorts of headaches since the other libraries are loaded first and thus can't be excluded by the #pragma and then the MFC libraries attempt to get linked in and fail. But it can happen in other cases as well - even indirectly via a third-party library.

I've never experienced this, but according to what people say on the Internet, when they run into LNK2005 errors, they completely rebuild all of the projects in their solution from scratch (i.e. brand new .sln and .vcproj files). It would take me several days to do that, so that is not an option for me. But apparently it can work. My guess: They implicitly did #1 and #2 by taking that approach - only it took way longer than necessary.

A lot of people suggest turning on /VERBOSE in the linker options. In my experience, that rarely helps to do anything but create more confusion. The option exists, so it is probably useful in some cases.

Photoshop math

2007-05-31T18:15:00.000-04:00

When it comes to Photoshop, I'm an expert. Photoshop is designed, for the most part, for print design. It isn't exactly what I'd call web-friendly. Over the past few iterations Adobe has done things that help us web, icon, and what I call "scratch graphics" designers do stuff more easily.

However, I'm working in Photoshop right now and just realized that I started calc.exe for the zillionth time. Looking back over the years, I've realized that every time I start Photoshop, I have inevitably started calc.exe or pulled out my graphing calculator (now dead - a huge nuisance - should probably replace it but my experience was that it regularly chewed through batteries).

So, what I'd love to see in Photoshop is allow every field that takes a number to also take a mathematical formula and evaluate it. Real example of how I want it to work follows. Take the "Canvas Size" dialog:

For this case, I want to increase the width of the canvas to 886*2-1 pixels. Or 1771 pixels. Sure I can do the math in my head, but I save time by firing up calc.exe. I would save even more time if I could simply enter the formula "886*2-1" into that field. Or do what Microsoft Excel does and put an equals sign in front of the formula to indicate that it is a formula, "=886*2-1". I don't think I'm asking for a whole lot here. Writing a basic calculator program is a first-year college student assignment (every semester, a few requests from students pop up on mailing lists for someone else to do their homework assignments for them). This is Photoshop we're talking about. It is supposedly the cream-of-the-crop, top-of-the-line professional graphics editor and here I am running calc.exe to do basic tasks in Photoshop. Ironic.