Why OpsLevel Offers Interview Feedback

Over the last few months at OpsLevel, we’ve changed how we tell interview candidates we’re not moving forward.  Instead of a templated response, we now provide specific and actionable feedback for each candidate about where they fell short in our interview process and how they can improve.  The results have been spectacular.

Why most companies don’t provide interview feedback

When we first started interviewing and having to reject some candidates, we used the same type of emails sent by most companies.  They were kind, but vacant.  Here’s an example:

 Thanks so much for applying to be a Software Developer at OpsLevel and for taking the time to solve the take home challenge and speaking with me today.  We received a large number of applications and while we were really impressed by your skills and you clearly show the potential of doing some great work, we sadly can’t move forward with your application at this point.

🤦

At the very least, we replied to everyone we interviewed to let them know we weren’t moving forward.  However, as a candidate, this e-mail offers nothing as to why we were not moving forward or what they could have done differently to be at the top of the applicant pool.

Historically, there are two reasons companies don’t give feedback to candidates:

• It’s faster not to.
  It takes time and cognitive cycles to offer cogent feedback.  ‍
• They don’t want to be sued.
  In Canada and the US, it is illegal to discriminate against candidates based on protected traits (e.g., gender, age, race, religion, sexual orientation, etc.).  To avoid the possibility of anything said being construed as even remotely discriminatory, many companies flat out ban giving feedback.  

Why OpsLevel now provides interview feedback

When we started OpsLevel, we did not dedicate much thought to how we reject candidates.  We had experience conducting interviews at our previous companies and simply maintained the same processes we had used there.  Empty emails that told a candidate we weren’t moving forwards, but without any reasoning as to why.

Feedback is an important aspect of professional growth and a cornerstone of our own company culture.  We realized that extending feedback to candidates would allow us to be truer to our own values:

• Empathy
  Putting ourselves in candidates’ shoes, it sucks to spend hours interviewing with a company and then receive zero feedback.  
• ‍Growth Mindset
  We want everyone to grow, not just OpsLevelers.  How can people grow if they’re not given feedback?  
• ‍Intellectual Honesty
  Offering feedback directly to candidates forces us to objectively explain why we’re not moving forward.  This helps eliminate bias.  

We did evaluate the downside risk that offering candidate feedback could one day mean we face a lawsuit.  However, we weighed that downside against the upside that providing feedback is overall helpful to both candidates and OpsLevel.  What’s more, the downside risk can be hedged by the fact that technical feedback should inherently be objective.

Example Feedback

When we now reject candidates, we offer them both positive feedback about what went well in their interview and constructive feedback about where they could improve.  Below are some examples of both.

Positive Feedback

 👍 Communication
We really like how you described each step as you were coding.

 👍 Data structure fundamentals and complexity
You were able to talk us through the tradeoffs of using different types of data structures and chose the data structure that best suited your solution. You were also able to keep complexity in mind and made it a priority when asked to implement new features.

Constructive Feedback

 🙁 Communication
We were hoping to see you verbalize more why you were taking a certain approach.  For example, when determining the vertical win conditions, you tried a few different techniques, but did not explain what you were trying or why it was useful.  This led to a bit of thrash where you repeatedly went down a path that did not work.  Related, you seemed intent on trying to use a hashmap at various points in the interview, but without explaining why it would be appropriate.

 🙁 Proficiency
For this slot, we allow candidates to select the language / framework they’re most comfortable with.  You picked Javascript, but we found that you relied quite heavily on Google for rudimentary tasks and did not demonstrate fluency with its standard library or idioms.

 😐 System Design
You got to a working solution with a nice data model and well articulated interfaces.  That said, we were hoping to see more depth overall in your explanations and descriptions of how workflows are executed.  For example, you started with db polling and then were able to suggest Celery, but there wasn’t much depth beyond that.  Similarly, you mentioned Redis, but we were unclear if that was as a backing store for Celery or as a separate storage for locks.

The impact of candidate feedback

Feedback isn’t only useful for candidates, it’s useful for OpsLevel as well.  Providing feedback ensures everyone performing interviews at OpsLevel is standardized, objective, and impartial in their assessments.  Furthermore, some candidates we let down enjoyed their interview and feedback so much that they referred other new candidates to us, which helps our hiring pipeline.

Since we started offering feedback to candidates, many replied with their thoughts.  Here is some of what they’ve said:

 First off, I just want to say I’m so impressed by the entire interview process. The feedback is so valuable to me! I thought overall it was pretty honest. Admittedly, I did feel that maybe I wasn’t the right fit for this role as I do not have that depth of experience as a developer. I also definitely should have done better at some questions … Again, thank you for your time and feedback!

 Thank you so much for taking the time to provide such a thorough response, it’s so helpful. I really appreciate the feedback and will be sure to work on those points. Thanks again for taking the time to talk with me, and all the best filling the position.

 I completely understand your decision. Thank you for the feedback - it is much appreciated!

 Thank you very much for the feedback, not many people take the time to give one or even to send an email to say that it is not a match and I really appreciate that. I totally understand you being selective, this is a quality in my eyes and re-reading my code the day after I was not very satisfied either.

 Thank you so much for your feedback, I will definitely work on these things in future assessments.

 Thank you for the thorough feedback and kind words. I will definitely continue to hone my craft. I said this to Alberto, and want to reiterate that your take home assignment and follow up interview were really enjoyable compared to a lot of the interviews I’ve done recently. The concept of building on top of something you’re familiar with in a familiar environment really takes a lot of the stress out of the interview process.

 Thanks for taking the time to respond and provide feedback! Not many companies take the time to do this, and I will use this to strengthen areas where I need improvement.

 I appreciate the opportunity and the feedback! The experience interviewing was great and points out some things I need to work on. Best of luck to you guys as well.

 I really appreciate the thorough feedback, I do take this to heart and use it as a learning opportunity moving forward.

 Thanks for the response. While I’m disappointed that I’m not what OpsLevel is currently looking for, I appreciate the feedback and can understand where my skill gaps are. Thanks for the interview, it was a very enjoyable experience!

 Thanks for letting me know of the decision and the feedback is certainly appreciated. Will use it to address the deficits in my skills/knowledge. It was great interviewing with your team on the interview chain for arranging the interviews. Wish OpsLevel the best.

 Thanks Ken for your kind words, and Beto for your time yesterday as well. I really appreciate the feedback and transparency throughout the entire process, it’s rare!

If you interview other people, consider taking the time to write feedback or advocating that your company change its policies.  The impact is tremendous.

If you’re interested in interviewing with us and helping engineering teams adopt service ownership and DevOps best practices, OpsLevel is hiring.

Thank you to Djoume Salvetti for reviewing drafts of this article.