Learning the hard way…
I can’t say that I have learned these things in the SQL world, because I am relatively new to the SQL world, having only been a SQL DBA for the last 3 and half years. Now, in my past lives as a Windows System Admin, a Linux Admin, and a Systems Analyst/DBA on AS/400’s I have learned quite a bit about things you shouldn’t do. Like for instance….
Are you sure?
Microsoft has this thing about asking the user if they are sure they want to do something. Are you sure if you want to delete this file? Do you really want to restart? Are you sure…? Believe me, this “feature” has saved me a few times. However, this “feature” is a Microsoft thing. It doesn’t exist in say, the Linux world. If you type the command line command of “init 6” to reboot the Linux server, there isn’t an are you sure box…Linux just reboots. I learned this the hard way when I was a Linux Admin at EDS and prematurely hit return on “init 6” before I had nicely shut down a couple of instances of MySQL. Let’s just say the DBAs at the time were not happy with me, as they shouldn’t have been.
What was my takeaway from that experience? Always…and I mean always, double-check the command before you execute it. If you’re not sure of the outcome ask someone else to double-check it, verify the command verbally or online, or test it on a dev/test environment first. Whatever you do, if you are unsure, DO NOT blindly run the command on a production server. If the server goes down, the query locks down the instance, or the update/delete query doesn’t have a where clause or not wrapped in a transaction….these kinds of things are going to make you have a bad day. And when you do screw up, because you will, OWN it. Don’t hide behind your mistake and don’t blame others.
Rebooting isn’t a fix
Another lesson I learned the hard way was rebooting isn’t a fix. It was a bad, lazy habitat that I was allowed to slip into the early days of my career. That’s not to say there are not times where a reboot is an appropriate action. However, if you have constant memory or CPU issues then you need to be doing a deeper dive into what’s wrong with your server. Again, back to my days at EDS, we had an application running on Linux that was hitting one of the MySQL databases hard. Instead of troubleshooting the app, the developers had us just reboot the server as needed. That is until management had enough of that stupidity and had us take it out of the app pool.
That’s not just a Linux thing either, when I was the DBA/Sys Admin for an AS/400 at a bank, a technician for the banking application suggested we reboot the AS/400 to solve an application problem. For those that don’t know, restarting a mainframe (the actual term is IPL) takes anywhere from 30 minutes to over an hour, sometimes two. It just isn’t done. You can bounce subsystems all day, but you just don’t IPL the system. In 7 years of working with 400s, I was only forced to IPL one once because an application locked up the 400. That just doesn’t happen.
I mention these things because Microsoft is seemingly making a huge investment in Linux as a hosting OS. DBAs, Developers, and Sys Admins that have spent their careers only in the Microsoft world are going to have to change their thinking a bit. Not only is Microsoft Linux, but they are also embracing the MacOS (ok, it’s just another Linux OS in a metal case) which for Microsoft is a huge about-face from the days of Steve Ballmer referring to Linux as cancer.
Being open-minded about your tools and how they are changing the landscape is going to be paramount in Microsoft careers. It’s always going to be paramount that with new tools, comes a new way of thinking with a new set of guidelines. Embracing these concepts can help staff from learning some lessons the hard way. However, also keep in mind, from our failures come our greatest lessons in what not to do. Oh and don’t forget to forgive yourself, or your team, for mistakes. It’s a fundamental part of learning.