This month’s invitation from Marlon is sure to bring out the war stories as he asks us to share our most memorable SQL Server outage.
I enjoy a good war story.
But no, this isn’t the time Azure blindsided me, when service accounts were locked, or when I left a transaction open and halted a production system (easily fixed by closing SSMS 😅). Mine is a pivotal personal moment.
Early one morning I was headed into the city to beat the traffic and grab an early brew. What I didn’t expect heading into the office was the swift shoulder taps and messages that we had numerous services out of action.
A few weeks prior we’d migrated environments, and the impacted services were all on the same machine so it was likely server and not application specific. Hopping onto the server I see low CPU, memory available, services running. Oh but there’s that one full disk.
I know the migrated apps weren’t large. What’s even on that drive? Oh, logs. Oh. Logs.
No log backups had been set up. The disk had slowly filled. Nobody spotted it, no alerting in place, a silent killer. I got Ola’s script deployed, took a log backup, freed up the space, and averted the crisis before most of the business was even aware.
Memorable? Maybe not for a DBA, no.
But there’s the twist. I wasn’t part of the data function. The new environments were set up by our professional services DBAs. I was simply an app guy with an interest in data.
Diagnosing and resolving this cemented my passion for data platforms. Shortly after, when they decided to bring the DBA roles internally, I jumped at the chance to pivot – and was given the opportunity without hesitation, in part due to this minor outage.
Sometimes all you need is a bit of knowledge, initiative, and willingness to step up (and log backups). Whilst not the most complex outage, the incident changed the direction of my career. Not a bad way to start the morning.
Andy Brownsword 