Interviewing Software Quality Assurance engineer candidates

Introduction
I've been in the software industry for over 20 years. I've interviewed at lots of companies, from start ups in stealth mode to some of the biggest and most successful. I've also conducted lots of interviews, especially as I've become a more senior quality assurance engineer. Since I'm an individual contributor most of my experience is with interviewing candidates for individual contributor positions. I've never been the only employee who interviews candidates. I've always been part of a team that interviews candidates. In this article I offer some suggestions for how teams and individuals can conduct interviews for software quality assurance openings so they can find great candidates.

Before the interview, as a team
Whether the candidate talks to employees one at a time or in a group interview setting, the candidate will be evaulating the interview team, therefore your team should be prepared and organized.

The interview team can do a couple simple things to get the most out of the interview. Have a brief meeting  with all the employees who will talk with the candidate(s). Cover these items

• Most likely there is a job description for the opening but it's a good idea to have the hiring manager verbally describe what's most important for this opening. Doing this in front of the interview team gets everyone on the same page and allows people to ask the manager questions.
• If multiple employees will be talking with the candidate make sure everyone knows what others will be covering. It looks bad if 3 or more employees ask the candidate the same question. Plus, a smart candidate will be less likely to join to disorganized team so having a coordinated interview team is important for impressing the best candidates.
• If the candidate will be expected to program, make sure someone asks programming questions. Anyone can put Perl or Java on his or her resume. Make sure the candidate knows the programming language.  GIve the candidate code to review, ask basic questions for the languages your team uses. CareerGuru and tutorialspoint are two of the many web sites with programming interview questions.

Before the interview, as an individual

• Read the candidate's resume
• Know the areas you are going to cover and be ready with questions for candidate. Ideally you should have more questions than can be covered during your interview timeslot.

During the interview

• As you talk with the candidate you'll need to decide if you want to go broad or deep or a combination of both. Going broad means asking one or two questions on lots of areas. Be careful this doesn't turn into a conversation that simply verbalizes a lot of what's already on the candidate's resume. Going deep means asking more and more questions on a specific area. This gives you a chance to see how well the candidate can explain something and it gives the candidate a chance to show skills and knowledge. If you interview someone who is intelligent, excited and enthusiastic while answering your question, you have a good candidate.

Ideas on common types of questions
I'm sure you and your interview team will come up with an interview plan specific to your openings. I do have a few questions I've found useful for evaluating software quality assurance candidates at all levels. I'll describe the question and I'll describe what I'm looking for from the candidate and why.

The accuracy question
Quality Assurance engineers need to pay attention to the details. This is true when reviewing product specifications and when writing and running test cases. I've found a good way to determine a candidates attention to detail is to ask them to walk through up to three test cases. They don't run the test, they merely read the steps of the test case and tell me what they are thinking as they go through cases.

The first test case has a few steps in it about setting up some conditions, gathering data and then plots the results in a graph. The key in this test is the last step. It says to consider the test passes if the data plots close to the ideal results. The graph shows the ideal results and the output from the test. A good candidate will quickly point out that 'close' is a relative term and the test case needs to be more specific to determine pass or fail. A bad candidate will say something like 'The two lines look close so I guess the test passes'.

I have a second test case that is similar to the first one.  It shows a graph with actual and ideal sets of data points plotted. The actual data is nearly then same as the ideal, but not a perfect match. The test description says 'This test passes if the actual data mimics the expected data'.  'mimic', like 'close', is too vague for a test case and a good quality assurance engineer should see this.  I'm looking for the candidate to point out the poor wording of the test case.

The last test case is very specific in its pass criteria. It shows the output data and the target data in a table and states very clearly the test passes if the actual data is within 0.05 of the target data. All candidates get this one right. For a few of the 'bad' candidates this test has actually caused them to realize the mistake they made with the first two test cases. One candidate went back and revised his answer to the first two questions and commented on the vagueness of the test case description. I've found this process to be very helpful in evaluating how a candidate approaches testing.

The toaster question
Your boss walks into your office and sets a toaster on your desk. He says test this and give me a report tomorrow. What do you do?

I usually give the candidate about five minutes to talk about this. The most important thing I want to hear from candidates is to ask for a specification. Most all candidates talk about safety testing and longevity testing and functional testing. All that's good stuff, but everyone should get those. What most candidates don't do is ask for a specification. How can you test a product if you don't know what the product is supposed to do, even with something 'obvious' like a toaster?

The kitchen question
The next day your boss comes into your office. You give him your report. Then he says for tomorrow he wants you to come up with a test plan for a complete kitchen. What do you do ?

Again, I'm looking for the candidate to ask for a spec, a description or floor plan. Most candidates don't do this. Additionally, I'm looking for the candidate to talk more about workflow type testing. Testing the toaster was about testing one item. The kitchen test is about testing a system. Does the candidate recognize this? A good candidate will ask questions. Is it a restaurant kitchen or a residential kitchen? will kids be in the kitchen? and so on. I may ask 'If you only had a day to test this kitchen what would you focus on ?" "Why?"

I've played with the idea of having a poorly layed out kitchen floor plan (e.g, dishwasher far from the sink). If a candidate asks for one I offer this and see if the candidate calls out the problem(s).

Conclusion
Whether you use these questions or come up with your own, I encourage you to come up with a plan that will help you determine the best candidates for your Quality Assurance openings. You and your company will probably make a significant investment in a new employee so making an up front effort to help you land the best candidate is a wise investment.