Who me ? Welcome to another Who, Me? confession of the Register readership and a reminder of unexpected side effects of software updates.
Today’s story of “Ralph” (not his name) takes us back almost two decades to the days when he was in charge of a selection of servers, that was when the machines virtual were everywhere.
He had a selection of kits in his books. An SQL server, an AS / 400 doing the accounting, and a Hewlett Packard server running Novell Netware and responsible for managing files and e-mail.
He also took care of the fleet of desktops, “having to eliminate those who spent a little too much time on some of the shady sites at the time.”
âI was,â he said modestly, âthe god of my domain.
The days passed without incident. Ralph has conscientiously ensured that live virus scans were performed on servers and desktops in order to keep the network perfectly clean. Updates have been deployed. His halo as the god of all things HE was burning hot.
Things inevitably changed. A seemingly harmless Novell update appeared and was installed. After hours, of course. It went off without a hitch and users were oblivious (“as they should be,” Ralph said).
A few days passed. The aforementioned virus scan has occurred. And hell broke loose.
The help desk was suddenly inundated. We have all been confronted with desktop PC crippling when IT, in their infinite wisdom, sees fit to run a virus scan. But it was more serious; no one could access the Novell box. The box that handled emails and files.
Ralph rushed to the console to find no open jobs, no file service, and many, many errors. How could that be? He checked the obvious and sure enough there was no free space on the disk and the best of Novell was not happy. But how? There had been no warning, no hint of impending doom. What had happened?
Naturally, the finger of blame fell on Novell. After all, an update had been installed a few days ago and now the world was seemingly over. However, after more than a day of research and long waits on the provider’s hotlines, Ralph was no closer to identifying the source of the jam.
Until he came across a thread from another user with a similar problem except that it was antivirus software. It turned out that he also had an update. One that didn’t play well with Novell. Not at all.
Due to the disagreement between the tools, the file server reported that the file size was 10 times larger than normal. The reader was indeed full, although it really was not.
âThe only way to fix this particular problem,â Ralph said, âwas a full rebuild and restore from backups, or to get the machine running so that the downloaded patch properly resolved the file size issue.â
Like all good administrators, he had free, unallocated disk space. In case.
Sacrificing his stash, he rebooted, applied the patch to fix the fix (very Microsoft, these days) andâ¦ all file sizes have been fixed and the files themselves can be accessed again.
âMail was flowing again and files were being served,â he said.
“I would like to say that the users were left oblivious, but alas, they were all very well aware of the day and a half outage caused by some software not playing in the sandbox with others.”
While the fixes have spread throughout all aspects of modern life (no, we’re also not sure why our microwave required a software update last weekend), the quality didn’t always fail. kept pace. Tell us about when your faith in vendors’ ability to deliver fixes without a blast radius got its first blow with an email to Who, Me? Â®