Mistakes Engineers Learn

T-SQL Tuesday #92, Lessons learned the hard way

It’s T-SQL Tuesday, the blog party that SQL Server expert Adam Machanic (blog | Twitter) started. This month’s episode is hosted by Raul Gonzales (blog | Twitter) and the topic is about Lessons Learned the Hard Way 

The equation for fluid dynamics.

The concepts behind set theory.

The operators in relational algebra.

The mathematics behind materials strength.

The abstraction principle in computer programming.

If you’re an engineer or a computer professional, I’m sure you can relate. These are the things I’ve learned in engineering school. I learned how to use equations to solve a problem. To apply a specific principle to come up with a solution. I even have a diploma to prove it.

I went out to the real world applying the lessons that I’ve learned. I’ve written software, designed systems and fixed problems. I thought I was doing alright. I was wrong. I guess I’ve learned several mistakes along the way.

Mistake #1: Solution without Vision

I remember my very first enterprise-grade entity-relationship diagram for a national database that is supposed to store information for people with disabilities. I had all the forms that the organization used to capture the data. Armed with my legal pad and pen (hey, this was 2001), I started scribbling the design for the database. The design was translated into the database. My colleague and I then wrote the corresponding front-end web application. Everything was going well. The application was able to capture the data from all the printed forms that they have on file. Until it was time to demonstrate the solution to the management.

The solution has fallen short of the goal.

It turned out that the grand vision wasn’t just to create a national database. It was to assist policy makers in drafting legislation to benefit persons with disabilities. My colleague and I were so focused on writing code that we failed to see the bigger picture. It’s not that the solution didn’t work. It’s just missing the key ingredient that could help make a difference.

That mistake has taught me to always ask questions that go beyond the technical requirements. It also taught me to learn how to talk to decision makers and stakeholders.

Mistake #2: Technology without Empathy

We were about to roll out a custom high availability solution for a SQL Server 2000 database. I was the one responsible for designing the end-to-end solution. It involved several moving parts with some scripting to aid automation in case something went wrong. We had clearly defined SLAs and escalation procedures. Part of the escalation procedure was to get the operations engineers involved as the first response in case of an incident.

I was enjoying a long weekend with my family when I received a phone call. The voice on the other end sounded a bit frightened – I can hear his voice shaking. It was one of the operations engineers. The database has been offline for an hour (our SLA was 2 hours) and he’s been on the other line with the customer for half-an-hour. He was getting grilled and yelled at. I gave him instructions on how to resolve the problem and the database was back online after another 15 minutes.

I failed to realize that a solution is not supposed to showcase how good I am nor to pad keywords in my resume. A solution just has to work with the simplest design possible. More importantly, the people responsible for maintaining the solution should be well equipped to do so.

That mistake taught me to design with people in mind, not technology. It’s why I educate my customers every time I design and build a solution for them. And it’s the very reason I pursued this project.

Database engineers from the world’s 10th largest internet company will agree.

Mistake #3: Automatic without the Manual

No, this isn’t about your car.

I’m a big fan of automation. I’ve been working with PowerShell ever since the Monad days. I’ve written scripts, created workflows and wrote processes to automate repetitive tasks. It came to a point that I’ve literally forced myself out of a job – automation has taken over my day-to-day tasks. I did find another role within the same team.

It wasn’t always like this. I used to just write scripts. Anytime. Anywhere. In the bus, on the subway, across the atlantic. I’ve even deployed a full-blown SQL Server Availability Group on Microsoft Azure on board an Airbus A330. Talk about being on the cloud.

Write code. Test. It’s the shampoo algorithm. Engineers and computer professionals do this all the time.

The irony behind automation is that you have to really understand how manual labor works. Which means you have to do it over and over and over and over …  You get the point.

But the problem with automation is it forces us to think that it’s all about automation. In fact, I spent about 80% of the time in my previous job automating tasks. That was my biggest mistake. Because the more I automate things, the more often they fail. Then, I was back to square one.

That mistake taught me to start by defining processes – the manual. Process definition is key to successfully automating tasks. DevOps is a by-product of improved communication and collaboration between development and IT. In other words, better processes. Many engineers found it surprising when I told them I barely touched a computer when I was working as a data center engineer. That’s because I spent about 80% of the time defining and refining processes – a complete reverse of what I previously did. The streamlined process is what allowed our team to implement solid automation framework. Which made it possible for us to do disaster recovery exercises every quarter over a weekend despite running several hundred servers. And this was the year 2007, almost a decade ago.


I could list numerous other mistakes that I made – like that Windows XP image that I accidentally deployed on a SQL Server machine. Or that corrupted backup that I thought was useful. Maybe the audit script I ran that took out the production database. But when I think hard about all those other mistakes, they would fall under any of these three main mistakes.

It’s OK to fail. You just need to know the difference between a failure and a mistake.

Please note: I reserve the right to delete comments that are offensive or off-topic.

Leave a Reply

Your email address will not be published. Required fields are marked *

4 thoughts on “Mistakes Engineers Learn

  1. Excellent article…Thanks for sharing your experience 🙂 Your experience will help many like me in future.