Interviewing can be super stressful, even under the best of circumstances or the best intentions. I really want my hiring practices to be as collaborative as possible, and trying to reduce the stress of interviewing is part of that. A few weeks ago I wrote about updating our technical assessment with the promise of an upcoming Part 2 about the changes I made to panel interviews - Reader, this is that Part 2 I know you’ve been waiting for!

Last year, my skip-level (one of our VPs of Engineering) was trying out some changes to our engineering interview process. He was doing a proof-of-concept on having candidates submit written answers to behavioral questions with an interview panel to follow-up instead of just doing the traditional face-to-face panels. There were a couple of challenges that he was trying to solve for with this experiment:

• The situation where an interview ends and interviewers feel like they weren’t able to get enough detail from the candidate’s answers
• Attempting to reduce interviewer bias, especially around how biases can affect our impression of candidates, the questions we ask, and how we ask them

Based on both open feedback and scaled-response questions, candidate feedback was mostly positive. Candidates felt that the written format reduced pressure while still giving them a good opportunity to present their qualifications for the role, and thought that the interviewers did a good job of using their responses in the follow-up panels. One candidate called out that non-native English speakers could find the written process more difficult, and might introduce additional interviewer bias.

I love that he was thinking about ways to make our hiring process more inclusive! During my previous job search, I’d had the opportunity for a written interview response and I agree that it helped take off some of the pressure. But also - it was a lot of work. There were about 10 questions, and no context around what they were looking for in an answer. I spent hours on the questions, way longer than an actual interview would have taken. Was this response long enough? Was that one too long? Did I include enough context? Did I add unnecessary details? What tone is coming through? How does this response compare to the one before or after it? In the end, I really ended up trading off one type of pressure for another. I still appreciate the alternative, but that experience means it’s not at the top of my own list to try out as a hiring manager.

But there was another option I wanted to try - sending the panel questions to candidates in advance.

I talked with my manager and skip-level to get their feedback, and also brought it up as a discussion in a leadership Slack community that I’m active in. I really liked this idea, but I wanted to hear perspectives from other folks, especially those with more experience. There were some interesting pros and cons that came out of these conversations.

Cons

• There was a worry that people might over-prepare and be falsely verbose, which would mask their actual communication style
• The idea that people might “cheat the system” and cover up gaps or negatives that would come out in the traditional panel format
• Advanced questions tend to favor people with the privilege of more time - the same downside of take-home technical assessments

Pros

• It allows people to “prime their brain” and get their thoughts organized around what they want to say
• It helps the interview process better match the working environment and expectations (e.g. we don’t have pop quizzes on the job!)
• Reduces the stress of “unknown unknowns” by allowing candidates to narrow their areas of focus
• Interview sessions can be more of a collaborative conversation instead of having to follow up with more targeted questions due to verbal wandering

In conversations about the pros and cons and what the process might look like, I realized that my motivation behind this was a little different than other people’s approach. A lot of the discussions were around how sending questions in advance would improve or prevent our ability to hire good candidates - but that wasn’t what was driving this process change. I’ve never worked at a company that sent the interview questions in advance and we still hired good people, so I’m pretty confident that we can find and hire good candidates without this. My motivation was purely to reduce the pressure and stress of interviewing.

Iterations

I’ve used this updated process for a few rounds of interviewing now, and overall I was really happy with the changes! Candidates seemed less nervous on the video calls, and we got positive comments about reducing anxiety and feeling better prepared to give thoughtful answers when we asked for feedback about the process. From the interviewing side, I noticed that we were having better conversations with candidates! Previously, we’d often find ourselves having to ask follow-up questions because we weren’t getting enough information to evaluate the candidate well, but now we were getting more meaningful responses that allowed us to have a more conversational approach and find out more about them.

But there were also some experiences that highlighted the gaps in this new process. During an interview, we had a candidate who seemed to be reading long-form answers from their computer in response to questions - it seemed like they had over-prepared by writing out all of their answers in advance, and this did make it hard to get a read on their communication style. It actually reduced the conversational format and made follow-up questions difficult because they were so focused on what they had written down that they weren’t always able to extend those answers.

There were mixed reactions to this, but I realized that it wasn’t the candidate’s fault for over-preparing and sticking to a script - it was mine, for not setting context and expectations. So I’ve recently added a blurb to the emails we send to candidates, which adds context to the behavioral panel, the technical panel, and the final wrap-up panel.

This sets expectations appropriately for candidates - it tells them who’s in each interview so they can ask relevant questions based on their audience. It tells them why we’re sending the questions in advance, and how we expect them to use that information. It lets them prep a bit for their technical interview and get familiar with the environment they’ll be working in for that session. And hopefully, it helps ease some of the stresses and pressure that people feel when going through the process of job searching and interviewing.

Are you looking to change up your interview process so it’s more inclusive and collaborative? Or have you already made changes that were successful? I’d love to hear your experiences!