Yesterday I spent about 10 hours at work, got in at 8 and left at 6. A major part of my day consisted in getting data from one system MC to the CRM system. We are building middle-ware that allows two way communication between said applications.
I got MC configured to spit out a CSV file that I am picking up with our CRM integration software and then from there I have to write this huge query to map out fields etc. Pretty simple stuff just a lot of work. Around 6 pm I decided I had enough and went home. I left my PC up and everything running with the expectation that it would be there in the morning.
I got in today around 8:30 and saw this message on my screen. Apparently Microsoft decided that their update was important enough to merited an automatic restart without asking for permission or having any regard whatsoever for the applications currently running. Needless to say I lost my HUGE query and a full day worth of work, the database crashed because I had an active connection to it and didn’t close it properly and to top it all of this happened across the company to almost every employee running XP.
To sum it up I have spent more than 4 hours trying to get everyone up and running, one of our servers that I had automatic updates running was also restarted overnight (no big deal)… except this server was a Domain Controller which pretty much broke every single computer in our system. We have managed to restore most of the functionality but we are still getting calls from people asking for their “open documents”, we have no choice but to tell them “you are out of luck sir, blame Microsoft”. Of-course we have taken now new measures to ensure that automatic updates are no longer automatic and that every system in the company has this setting turned off.
The messed up part of this is not that the updates made the computer restart it is the fact that a “Security Update” is supposed to protect the system from loosing data and getting attacked but by doing so it caused more problems than it helped. Out system is off the Internet , we have one proxy that talks to the world, the company rents private lines to do all the internal communications and there is no outside access. The only way in is trough Cisco VPN and that is highly restrictive. So there was no way that said vulnerability would have affected out system. However because Microsoft in its infinite wisdom decided to circumvent any logic, we ended up loosing several hundred of man hours.
We have many servers that run Linux and after this incident we may have many more, I just wish that someone out-there would make a decent windows emulator for Linux, then maybe Linux would take off and there would be no more restarting. I can’t understand how a company like Microsoft hasn’t figured out a way to apply a patch without taking the system offline. Do they realize that Linux allows you to hot-swap the kernel with the system running?