What Makes a Good DevOps Engineer?

A few points from my experience hiring engineers and how to interview for these specific traits

What Makes a Good DevOps Engineer?
Photo by Gabriel Dalton / Unsplash

For the past five years or so I’ve been directly involved with finding, interviewing, and hiring DevOps engineers for various organizations, both large and small.

In that time I’ve read too many resumes and LinkedIn profiles to count, but from that vast ocean of data, a few trends have emerged that I would like to share with you and hopefully help you find and hire solid DevOps engineers.

Certifications Don’t Matter as Much As Some People Think

My first introduction to this was as an engineer for an MSP back in my early days. The small company I worked for hired a new engineer to fill the space left by an engineer who had just left the company.

While we were a mostly Windows shop and 90% of our clients had Windows-based infrastructures there were a number of Linux systems that we had to support, so it made sense that we hired an engineer with a certification for Red Hat. The only problem was that from the numerous questions he started asking it was clear he didn’t know anything about Linux. Situations like this are not isolated cases, either.

It would be impossible to count how many phone screens I’ve done where the candidate looks impressive on paper — multiple AWS certifications, Kubernetes certification, maybe a Cisco cert for good measure — but absolutely falls apart when questioned about basic stuff. Candidates will walk into an interview with a Kubernetes certification but won’t be able to answer a basic question like “what is a container?”.

As someone doing the interviews or actively looking to hire a member for one of my teams its frustrating to have to weed through the people who view certifications like simple merit badges to be won and displayed rather than what they should be: evidence of knowledge or experience of a particular subject that’s built on knowledge and experience of more fundamental subjects.

Getting a certification doesn’t mean someone necessarily understands the subject matter or has any kind of expertise or experience in the subject, it only means they know how to study for a test.

My philosophy on certifications is rather simple: if you have to study more than a few hours for a certification exam then you probably don’t deserve to pass the exam.

How To Interview Candidates With Certifications

Interviewing candidates with certifications is like doing research on Wikipedia — don’t take the information at face value, but instead look up the sources of that information.

If a candidate has a Kubernetes certification, ask about containers and how they work, container security best practices, and Linux cgroups or process isolation.

If someone has AWS certifications, you can ask them about how Linux machines work, basic TCP/IP networking (ie, TCP vs UDP, etc.), and various ways of scaling applications. Challenge them on what they say they know; if they fail the fundamentals then move on to the next candidate.

Critical Thinking is a Dying Skill

I’m going to blame our school system for this one; not too many people that come across my desk seem to be able to think critically about a problem. When faced with a problem they don’t know it seems people only want to have someone tell them what the solution is without figuring it out for themselves. This is rather problematic in my line of work, where thinking through a problem or a project is absolutely critical to success. The ability to look at a set of evidence, alarms, trouble tickets, or whatever your starting data is and formulate some kind of plausible hypothesis is a valuable and hard-to-find skill these days.

I hate hand-holding. I hate having to sit someone down and walk them through every single little step from point A to point B. It's not that I don’t like teaching people new things, I do, but at a certain point it transitions from teaching to coddling, and that point is where I stop being a gracious teacher and start getting annoyed.

At the very least, the bare minimum that anyone should be expected to do is to be able to Google something for themselves. I can’t even begin to count how many engineers I’ve had to ask me to help them with something only to find out they haven’t even tried solving the issue on their own first. There’s only so much I can teach someone if they don’t have the desire to learn in the first place, and I certainly can’t teach someone to think critically if they have no desire to do so.

How To Interview for Critical Thinking

Interviewing for critical thinking isn’t so straightforward, but there are ways to do it effectively.

Asking candidates about experiences where they had to troubleshoot large issues by themselves then have them walk you through what they did and how is one way.

Another way is what I like to dub the “architecture interview,” where I present the candidate with a problem and have them design a basic architecture to solve it, then I introduce new variables and new information and watch how they integrate that information into their solution.

Candidates who can integrate new information and be mentally flexible in the heat of the moment are the ones you want to look at further.

Strong Fundamentals Are the Most Important

I played basketball for well over ten years growing up, starting from Kindergarten all the way up through high school. (Not that I was any good or even close enough to going to college for it, but I was decent enough to make the bench.)

Those first five or so years of playing weren’t spent learning how to run plays, defensive schemes, or even just offensive concepts, instead, we learned the fundamentals of the game: dribbling, passing, shooting, defense. Our coaches taught us how to shoot the ball, where to put our hands, the physical mechanics of how the ball and the body interact.

We learned how to pass the ball and put it where we wanted. We learned how to play defense, what position to put our body in, where to be on the court, and how to rebound. It wasn’t until later on in high school that we started learning about running plays or different, more advanced defensive schemes, or how to break a full-court press.

We had to learn the basics and have a strong foundation before we could learn more. Everything that we learned built on the thing before it, and it's no different in technology.

Someone who wants to get into Kubernetes (I keep using this example, I know, but it's highly relevant today) that doesn’t understand how containers actually work or what a container is won’t really be able to understand how Kubernetes works.

Developers who want to make the next killer iOS or Android app but who don’t understand the ins and outs of how mobile devices work and the challenges that exist within the ecosystem will never succeed because they don’t understand things like I/O wait times, caching, or scheduling for their chosen platform.

How To Interview for Fundamentals

It doesn’t matter if you’re interviewing for a senior or junior role or if the candidate thinks they’re beyond these kinds of questions: always ask about the fundamentals. Make it policy. Make it standard. It doesn’t matter if you frame it like “hey, I know this is awkward, but I have to ask you these questions….” Ask them anyway. You might be surprised — that star candidate with the glowing resume and experience might not be as solid as you hoped, but you’ll never know unless you ask the “dumb” questions.

So what are kinds of questions am I talking about? Things like “how do I check disk usage in Linux,” “what port does HTTP typically communicate on,” “what is the difference between TCP and UDP,” and “if I put google.com into my browser and hit Enter, what happens.” You might be surprised. And the people who really know their stuff, the ones you want to hire, should have no problems answering these kinds of questions and should understand why you need to ask them.

Conclusion

So what exactly makes a good DevOps engineer? Well, it's a lot of things. But there are two things that are absolutely critical: critical thinking and good fundamentals.

Someone with those two traits will go far, and you can certainly do worse when hiring people for your organization.