[callout]It’s T-SQL Tuesday, the blog party that SQL Server expert Adam Machanic (blog|Twitter) started. This month’s episode is hosted by Kennie Nybo Pontoppidan (blog|Twitter) and the topic is “daily database-related WTF”. I’m sure you have had one of those horror stories – the ones that you’ve made and those done by others.[/callout]
I can guarantee you this. Because no one brags about their failures.
The year was 1999. A small business owner approached a young guy – fresh out of university with no database nor computer programming background – to write a software for their retail and consumer goods business. The business owner wanted a software that can be used as a point-of-sale system as well as managing inventory. The young guy declined, explaining that he didn’t have any background in computer programming nor database design. The business owner insisted: he fully trusted the young guy to deliver.
The young guy learned as much as he can about computer programming and database design. He read books, asked questions on forums and newsgroups. Tried sample codes. A lot of sleepless nights. After 24 days of banging on the keyboard writing code, he had a working prototype. He showed the business owner how it worked. It was very intuitive, easy to use and fully functional. He just got himself one happy customer.
The database had only one table…with 234 columns. It could have had about 17 tables if the design was properly normalized. What’s worse, all records were accessed using the RBAR database programming method. One “row by agonizing row” at a time.
The year was 2008. A conference room was packed with about four hundred plus people, mostly SQL Server database administrators. The topic: database recovery techniques. The speaker was showing a dozen different ways on how to recover from a database disaster – from somebody accidentally dropping a table to fixing a corrupted data page. The members of the audience were holding their breath. They just saw a disk volume disappear from underneath SQL Server, losing one of the database files in a filegroup. But the transactions kept going. Inserting records into the partitioned table, despite having one of the data files in the missing disk volume, kept going. Everyone in the room tried to guess what’s going to happen next. How will the database with a missing data file be recovered? Anticipation was building.
The speaker didn’t have a working database backup. How on earth can you recover that missing database file if you don’t have a working backup? One of the attendees even shouted, “well, that could be my production database right there.” And he was right.
The year was 2010. A consultant was working on designing and deploying System Center Operations Manager (OpsMgr) and System Center Configuration Manager (ConfigMgr) for a customer. The goal was to operationalize the enterprise software deployment process and systems monitoring. The servers were prepared, SQL Server was installed, OpsMgr and ConfigMgr configured. The project was in the stabilization phase waiting for the go-live schedule.
The consultant received an email alert. The SQL Server machine hosting both the OpsMgr and ConfigMgr databases was taken offline. And there’s no way the server is coming back online – ever. Because the system has still not reached “production phase,” only the databases had backups. It turned out that an image of a pre-built Windows XP Professional system was accidentally deployed specifically to the SQL Server machine. The process had a step to reformat the machine and install the Windows XP image. What’s worse, the deployment process was configured by the consultant himself.
In case you’re wondering, these three events have one thing common – ME.
I was the one who wrote the point-of-sale system and designed the database with a single table. I was the speaker who was right in the middle of a database disaster talking about, ironically, recovering from database disasters – in front of about four hundred plus people. I was the one who automated the deployment of the Windows XP image that took down the SQL Server machine.
But you probably didn’t notice that I called them events. They don’t define who I am.
I used to try to forget my failures. I was raised in a culture where failure is shameful. Until I realized that every failure contributed to where I am today. When I started seeing failure as a learning event – not something that defines who I am – I embraced it. Because it’s in the failures that I learn the most.
So, don’t be afraid to fail. Fail fast. Fail forward. Learn as much as you can. Just don’t make the same mistakes over and over again.
You still won’t find this in my resume. But I do talk about my failures. In fact, talking about my failures got me this wonderful prize.