Wednesday, October 27, 2010

KB976902 - "Black Hole" update

Update February 27, 2011 - Windows 7 Service Pack 1 appeared in my update queue with the checkbox unchecked. Hmm. To update or not to update?

Update January 14, 2011 - This appeared again in my Windows Updates. I figure installing it is okay now that Microsoft appears to have their ducks in a row. After the first fiasco, they probably took their time to release it correctly. This is the precursor to installing Windows 7 SP1. From Microsoft's website: "Windows 7 SP1 Release-to-Manufacturing (RTM) will be available in the first half of calendar year 2011. When released, it will be made available as an integrated release." This updates the installer in Windows 7 so it can upgrade to SP1 later. SP1 isn't available yet.

What follows is the original post.

Today, a mysterious Windows Update was released to all Windows 7 users. I'm going to preface this by saying what some Microsoft MVPs are saying: DO NOT INSTALL!

The update says, "Install this update to enable future updates to install successfully on all editions of Windows 7 or Windows Server 2008 R2. This update may be required before selected future updates can be installed. After you install this item, it cannot be removed."

This update appears to only be known as the "Black Hole update". No one knows precisely what this does but it appears to be a precursor requirement to installing Windows 7 SP1 - why it can't included with SP1 itself is unknown. There is no knowledgebase article (HIGHLY UNUSUAL!) despite having a KB article number.


Update (Oct 27, 2010): Korablikovas (see comments) pointed out that this update is not localized. In my extensive experience, Microsoft localizes just about everything that they publish. Which makes this update even more of an oddity and quite disconcerting. No official word yet from Microsoft on this issue (although someone decided to mark a reply an answer in the "semi-official" Microsoft forum post). Still no KB article. It appears that this update creates a System Restore point prior to installation, so if you accidentally installed the update and want to roll back the installation, you can use System Restore to do so.

Update #2 (Oct 27, 2010): Reports of this update disappearing from the "hidden" updates queue are cropping up. I can confirm that this update has vanished from my own queue and can also confirm that it wasn't installed. Looks like Microsoft redacted this update from any computer that hadn't applied it yet. As I somewhat suspected as I learned more from various sources, this was possibly just an accidental release of a piece of the RC build of Service Pack 1 - some sort of precursor component upgrade related to Windows Installer/MSI stuff. There's some people freaking out over this being a WGA update - apologies if anyone was lead to believe that from this post. And then there are the idiots who believed the TechArp April Fool's joke. It does bother me a bit that Microsoft can remotely redact updates but I've got mixed feelings on this particular update regarding redaction. I'd rather have remote redaction than have it still in everyone's queue who hasn't installed it. The problem Microsoft has to deal with now is those people who installed the update prematurely to roll them all back to a consistent state with the rest of us. That's probably a more difficult problem and Microsoft will likely take the approach of leaving well-enough alone until Service Pack 1 officially releases. I do recommend that Microsoft change the language of this update to not be so mysterious/suspicious and to get an actual KB article for the update put together ASAP. However, the potential disaster this could have been seems to have been averted for the time-being.

Update #3 (Oct 29, 2010): There is now a KB Article that goes along with this update. I didn't actually come up with the name "Black Hole", I just saw other people using it to describe the update because when a search was made for a KB article of the same number on the support site, no search results occurred. That is, the link and search went nowhere - a black hole. And, since it was redacted from the queues of those who did NOT install it, we can assume that Microsoft made a mistake by releasing it early. Of course, there is no word yet on what Microsoft plans to do for those people who installed the update prematurely.

What follows is original content for historical purposes.


The best information I've seen about this update is here, which says:

"In preparation for supporting a new Windows image, this update is updating the servicing stack to 6.1.7601.17105 including Component Based Servicing, Component Management Infrastructure, Package Manager, and Windows Management Instrumentation. This is to support Windows 7 and Windows 2008 R2 SP1."

I ran a quick Google search for portions of that text and simply came up with that blog entry again. So I have no idea how that person got that information (perhaps an internal TechNet memo). It looks like something Microsoft might write - just generic enough to be basically meaningless except to the few people who are nerdy enough to know what it likely means. I recognize a few terms but it is rather vague on the details.

Before people go crazy, I should point out that someone on that page linked to an "article" that will scare people who don't pay careful attention. It took me a while to figure out that it was an April Fool's joke. Bloggers and other media sources need to be VERY careful when republishing on this one.

The description indicates that it is an update to the installer components and therefore could merely be a mistake on Microsoft's part for not publishing a KB article. However, it could be a lot more sinister than that as well. As we already know, Microsoft is evil. Google, of course, does no evil.

Due to the speculation and the weird message in Windows Update, hold off on installing it until we know for sure what it is that is being foisted off on us. If anyone from Microsoft reads this - you need to work on being more transparent about your updates and seriously improve your public relations image. The way this update is worded and the mysteriousness surrounding it makes it look like another WGA-style fiasco in the works - and trying to sneak it by hoping it goes unnoticed. Guess what? I noticed it right away. Your pants are down...again.

Microsoft is baffled as to why people prefer Google. Straighten up, fly right, and stop treating your customers like we're unintelligent and incapable of running Google searches. Keep in mind that I'm a software developer. Here - have a reminder of who you actually cater to:

Developers! Developers! Developers! Developers!
Deodorant! Deodorant! Deodorant! Deodorant!

Tuesday, October 19, 2010

Easy Invoice Numbers

If you are building or using an e-commerce system and want to look all professional, you need invoice numbers. Invoice numbers are required, at the very least, for auditing purposes by most businesses. However, nothing says "not professional" quite like "dressing in a shirt that is flashier than a light pinstripe" or invoice numbers based solely on a MySQL 'auto_increment' field starting at '1' or, worse, a false arbitrary starting value.

Searching Google for an industry-standard practice of creating an invoice number turns up pathetic results. So this blog entry aims to correct this severe oversight of the Internet and bring it down to the level of the average programmer.

Businesses that are large enough have dedicated finance departments. These people like things to be EXTREMELY organized. If you are developing an application that is going to bring in money (e.g. an e-commerce solution), it needs to generate sequential orders that can be verified later should the organization be required to perform an audit. Audits are annoying. If you start numbering invoices/orders at 500, the auditor is going to want to know where invoices 1 through 499 are. So, you really should start numbering orders at '1'. However, from a customer's perspective, being the first person to order anything is viewed as rather "tacky" for the invoice. So, what should be done about this?

Invoice numbers should represent the order in some fashion that makes sense and makes looking up the order later on easy to do. It should also include some letters with the numbers. What follows is my own blend of parameters that creates a rather good-looking invoice number in most cases, makes it easy for programmers to extract the order number, AND meets the auditing restrictions imposed by most businesses:


So, a sample invoice number could be:


The example is the company "CubicleSoft" abbreviated to an acronym, the two-digit year, two-digit month, and two-digit day of when the invoice was created (November 19, 2010), a hyphen (-), and then the order number (Order 1). It could be the very first order ever in the system but the customer won't notice - they will likely just think it was the first order of that particular day. For programming purposes, it is easy to locate the hyphen, convert whatever comes after that to an integer, and then use the value to look up the order in the system. In fact, the only thing that really matters with the invoice is the number after the hyphen. The stuff before the hyphen can just be used for verification purposes.

The next issue is that finance departments require the ability to map an invoice number to the money. That is, they need to be able to exactly match the order to each and every penny. An e-commerce solution depends on a third-party to process the payment. This creates a disconnect between the order and the payment that must be resolved. The invoice number on the payment end of things must match the order entry system or auditors get unhappy. Therefore, be sure to send the payment processor the fancy invoice number so when someone later on goes to do a report, they can easily get that information, which makes that person's life easier and simpler.

In conclusion, I really like that video. :)

Monday, October 11, 2010

Forget flock() and System V Semaphores - use WebMutex instead

While developing my latest PHP-based project, WebCron, I ran into an issue that has bothered me for a while - atomicity. An atomic operation is one where only one thread of one process is allowed to execute some piece of code. Actually, even under C/C++, I've been bothered by this issue.

Unlike Linux, Windows really has the most friendly approach to creating an environment where atomic operations may thrive. Named mutexes is one area where Windows really, truly shines above all the OSes out there. Try porting CreateMutex() to another OS and you'll inevitably have some real head-scratching sessions when you try to do a cross-process, named mutex. So-called 'mutexes' under *NIX OSes are usually 'pthread'-based, which are really more in line with Windows "critical sections" than "mutexes". A programmer coming from the Windows world is going to be utterly confused because they've been pampered by Microsoft and no one in the Linux community sees any benefit of adding REAL mutexes to the kernel.

The problem lies in the fact that the Windows OS itself manages the mutexes. When you want to lock a mutex, a jump into kernel mode is made and the kernel manages the locking process. If a process or thread goes away while holding onto a locked (signaled) mutex, the Windows kernel potentially eventually cleans up the mess left behind. And here is the key point: There is no equal process on any other OS and definitely nothing that is cross-platform (natively). Or at least no single approach that has been "standardized".

Let's look at the options available for creating something close to a named mutex. I'll be specifically focusing on what is available for PHP:

flock() - This function is the first recommendation by the PHP developers as their "cross-platform" "solution". If you read the documentation and the comments carefully, it quickly becomes apparent that it does NOT behave the same under all OSes, especially Windows. Here are the core problems: The non-blocking option is ignored (i.e. doesn't work) on Windows, the return values are not consistent across OSes, it has issues with multi-threaded web servers, and it requires a file to already exist (in advance) on the system to be used as the "lock file".

System V Semaphores - These functions aren't available for Windows, requires an integer instead of a name to identify the semaphore (seems hacky to me), and the functions are not compiled into PHP by default.

For WebCron, what I needed was a mutex object that provided a cross-platform, cross-process, cross-thread, non-blocking, expire-capable, multi-lock capable, named mutex. When developing a product that will be used on who knows what OS, it has to at least support "The Big 3": Linux, Windows, Mac. Clearly the above functions weren't going to cut it for my needs. I needed something else.

I scoured the PHP documentation for a while. I knew the key was that the OS must manage the creation of the mutex, not the application. I knew that if the application managed the mutex, there will be race condition issues and other nastiness involved.

Then I found the solution I was looking for: fopen().

Every OS manages the file system and all access to files. However, the 'w' and 'r', 'a', and 'c' options of PHP automatically create a file if it doesn't exist. And then I found...'x'. It creates a file if it doesn't exist and will return false if it does. The '@' prefix suppresses errors and warnings. Since the OS manages the creation of the file and since file system access is universal, I had found my cross-platform solution (i.e. silver bullet). And WebMutex was born.

The first hurdle was deciding how to handle the lock files. I ended up opting for using an absolute path and filename and then appending '.lock'. I call @fopen($filename . ".lock", "xb") and check the result. If it is false (i.e. some other process beat this process to it), then I don't have the lock, sleep for a random amount of time, and try again. Releasing the lock was a matter of deleting the file. If Lock() is called multiple times, I just increment an internal counter instead of touching the file system again.

My next problem was that PHP scripts can only run for 'x' seconds before PHP forces the script to exit. PHP doesn't run cleanup code when it forcefully stops execution. So I had to come up with a way to delete the lock file. I eventually settled on creating another file using the fopen() 'x' option with '.stale' appended to the base filename. That way there aren't multiple processes deleting the lock file - potentially introducing a race condition that allows multiple locks to be obtained. Of course, it becomes possible to end up with both '.stale' and '.lock' files. However, that should be rather rare and requires manual intervention anyway.

The solution I came up with to the second problem isn't without its own problems though. If the file system is shared and there are two processes running on multiple machines and the timestamps of those machines aren't perfectly in sync, the lock file might get deleted prematurely.

The downside to this whole approach is performance. A file-system based solution isn't going to be nearly as fast as a native kernel solution. However, named mutexes aren't speed-demons to begin with. I'm after something that works not something that performs spectacularly.

However, the end result is actually a lot simpler than I originally thought it would be. It meets all my needs and addresses my concerns. And, perhaps more interestingly, this solution is easily portable to other languages too. The only requirement is being able to open a file for creation and return a failure condition if it already exists.

The source code to WebMutex is under the same license as WebCron (MIT or LGPL, your choice). You'll have to download WebCron to get at the source code for WebMutex.