At one place there was a bug squash interview like this. Rough idea was we wrote a very small version of a system we had in our app (some data-syncing-then-displaying service), with a handful of bugs and a very simple feature request.
It was very helpful for sanity checking if a person was able to be in front of a computer. There's a bit of a challenge because I think there's a pretty low ceiling of performance (strace on a Python program is fun, but our bugs are not that deep!), but we could use it to also talk about new features on this mini-system and whatnot.
General feeling on it was "this is a great filter", because ultimately we needed people to just pass above a minimum skill bar, not be maximally good at coding. And having the exercise really helped to communicate the day-to-day (which can be tedious!)
I've done the equivalent of this by asking them to describe an interesting bug they've encountered in past lives; how it came up, how they hunted it, how they fixed it. By listening to them describe the work, asking questions, and following their thought processes, you can come to a fairly good hire/no from this single walk-through.
I know, it's short and humane, so not a good fit for current Sillycon Valley culture.
> It’s easy for the candidate to self-assess their own progress.⊕ When the candidate isn’t doing well on it, they probably already knew it without needing to be told as much by the recruiter. This is a much better candidate experience than the whiplash of thinking you solved a question perfectly only to realize that the interviewer was looking for something else entirely.
Not to say that I think this is a bad type of question overall, but IMO, this is an anti-feature. The candidate does not need to accurately self-assess their performance on the day. They need to have their confidence preserved so they don’t tilt and tank the signal for the entire interview round.
I wonder if this is an improvement over a conventional white boarding question.
It seems to me that debugging in particular is often dependent on the developer realizing some edge case or scenario where things circumstances line up to produce a bug. In that case, wouldn't a "bug squash" end up being similar to a "gotcha style" white boarding question.
The author, quite correctly, says that the interview should be scored not on whether the candidate can immediately vomit up a solution but on whether they can demonstrate their an organized thought process. However, isn't that true of white board questions as well?
Overall, it seems to me that the real problem with conventional white board questions is when the interviewer focuses on whether the candidate gets the right answer rather than on whether they demonstrate the ability to problem solve. While it sounds like the author is a good interviewer who focuses on assessing the candidate's thought process, it's not clear to me that giving debugging questions is actually what causes that.
I've only had one "find the bug" interview, and it was awful:
- Didn't set you up with a way to reliably run/test the code, nor a step-through-debugger which, jeez is it 1980? Like setting up a repo in a language you aren't familiar with (say getting your py-env exactly right) can take a whole hour if it's not your main language.
- Didn't have standardized questions, which is hugely problematic (some bugs are 2 orders of magnitude harder than others)
- It also just seems like there's a huge random element, like am I debugging code written by somebody who thinks at the same level of abstraction as me? Did they write comments I understand?
My favourite thing to do for "coding" interviews is to give the candidate a piece of absolutely awful python code that a friend of mine came up with for use for interviews.
There are code smells, side effects, errors, confusing syntax, a whole slew of things in it. I give them the code, tell them I'll help with every bit of python syntax (and really emphasise that I don't mark people down _at all_ for any lack of familiarity with python / python syntax), and ask them to work through the code and do a code review.
There's some python specific quirks that I largely consider bonus points at best, but anyone with any appreciable familiarity with at least one language should be able to spot a number of things. As they call stuff out, I'll dig in (or not) as seems appropriate.
So far it seems to be working out great for me as an interviewer. Candidates seem to be way less stressed than they are with a "whiteboard coding" exercise. Good discussions are showing me their development skills, etc.
I have an interview question:
step 1: the candidate is shown the specification for a method and the results of running the test suite on an obfuscated version of the method. All tests pass. The test suite is minimal, and test coverage is abysmal.
step 2: the candidate is asked to come up with more test cases, based on the specification. The code is run against the updated test suite - most new tests will fail because the method's implementation has several known bugs.
step 3: The un-obfuscated source is provided and the candidate is asked to correct any bugs they discover.
step 4: the changed source is run against a full test suite.
I like this because the candidate's ability to think of test cases, debug, and fix existing code are all tested for.
One of my favorite interviews was like this, but it was a long time ago and they printed out the code and told me there was a problem. Here's a pencil. What's wrong? Most were a function call or two. In one case I remember someone was doing a loop that incremented the index of a SQL query in code and using each result, instead of querying the set and looping through it in the code.
The fun part was it was a discussion. Two people, paper and pencil, talking about the code. And the examples were from bugs they'd already squashed in a code base they inherited. It was a project that I ended up working on that had... quite a few interesting bugs like that.
I somewhat disagree that this is reflective of a person's everyday work. Usually we're only ever working on at most handful of codebases at a time, and getting oriented in a completely new application (including getting it to build and run on your machine) doesn't happen every day.
I've done interviews like this and one major pitfall is the start-up time. It's possible to spend an unreasonable amount of time debugging cold-start issues with getting the repo set up, dependencies installed, and the app to build while the interview slips away. These things are representative of the first week on a new team or project, maybe. Figuring out why bundler can't seem to download the dependency in the gemfile isn't my idea of a good use of interview time.
I'm not too proud to admit I took the general concept from https://sadservers.com/ and turned it into something I could use for interactive debugging interviews.
I had golden images with some scenarios I wrote myself, and the images automatically shared a tmux session over http so I could follow along without requiring the candidates to screen share.
I did have to ask for IPs so I could ensure the machines were just available to me and the candidate, but it was otherwise pretty seamless!
Though now that https://sadservers.com/ has a paid service it might be worth looking into.
I was given a similar interview problem and thought it was one of the better interviews I've had. In my case the interviewer pulled up a web app and said this particular page is loading too slow, how would you approach the problem? We went from opening dev tools to look at the requests driving the page all the way through database optimizations and every layer in between.
This kind of interview didn't feel like a gotcha and was much much closer to real world work than the toy and/or algorithm problems that I have encountered. More companies should adopt these types of interview approaches.
I had a bug squash interview today. I found it nice, but also frustrating.
It was nice because I didn't need to practice and I knew exactly how to debug the thing.
It was frustrating because my personal laptop is from 7 years ago (from college), is slow, and the dependencies and editor don't work out of the box for an a new repo. Additionally, I'd prefer to use IntelliJ like I do at work but again, that's too heavy for my computer to handle so I resort to vscode and have to figure out how to use it. So then the interview becomes debugging my environment instead of debugging the problem. Maybe that's a useful signal, but it's not really bug squashing anymore then.
So overall, it was still requiring learning but there was not a very good way to test in advance (how do you test all possible repo structures?)
This is one of the interview questions at Stripe, and I agree that it was the most "fun" part of the interview (It is still pretty stressful giving an interview, but magnitude times much less so than converting a BST to doubly linked list, and converting it back )
If you can’t sit down with someone, have a simple tech conversation with them and be able to tell if they know what they are doing… if you need to “give someone a test”… the problem is likely not the candidate.
Triplebyte (one of the best hiring experiences I ever had) gave me this during their initial interview.
They dropped an archive of a medium-ish codebase to my machine, that had failed unit tests. My task was to simply fix the code so the tests passed.
Not only did I feel engaged with the interview because I could speak aloud as I debugged, I also found it fun!
Had an interview question like this where the interviewers were very impressed. Said it was the fastest and cleanest they'd seen anyone do it, and in an unfamiliar framework no less. Unfortunately it wasn't enough for me to get the job. I had a couple of answers in the next section they didn't quite like.
I agree. Great question. Academic LeetCode exercises are really not what I consider useful in evaluating candidates.
However, I'm biased. I'm really, really good at finding and fixing bugs, and pretty much stink at LeetCode, so take my support with a grain of salt.
One problem is that, if the exercise gets known, expect an underground economy of solution cheats. Same with LeetCode, but that's sort of expected. Bug fix solutions hit harder.
> You’ll need at least one question per language that you want to allow people to interview in. ... “Was this performance poor because they’re bad at debugging, or just unfamiliar with this language’s tools?”
Not necessarily a con. Different languages/tech stacks have very different tools. I.e. debugging a GC issue is different than debugging a network issue is different than debugging an embedded issue. If you chose a language for a very good reason, then it's not a con to filter out those who aren't familiar with that language enough to debug an issue in it
I did a bug squash interview when I was interviewing for new grad positions and it was one of my favorite interviews (both as an interviewee and an interviewer)
- People were allowed to use their favorite IDEs - so you could see how proficient some people were
- Great engineers are really really good at debugging - it was great to see how people debugged and it helped me pick up a few things as well
- People couldn't leetcode their way out of this
Although I like this interview question, it has a big downside. You make the candidate spend quite a lot of effort while you as an interviewer don’t learn much. The answer to “Can this candidate find and fix this bug?” is not necessarily equal to “Is this candidate a good expansion of the team. What do they bring that we need?”
This sounds like a lot of fun to me. A lot of people dislike debugging, but I always enjoyed it. It's a bit of a murder mystery. You not only need to understand what's happening several layers beyond the initial problem, but why the code may have been written the way it is even when you find the issue.
I had a very similar intro experience to dev. I started with some friends as a hobby, and my first paid jobs were when I was hired to do a bunch of small tasks in different projects. Add some animations in this Ember project, fix some bugs in this Wordpress project, add a very specific feature in this Angular project, etc. I probably did a terrible job at that time and I'll always be grateful, but it also taught me A LOT about jumping into a foreign codebase and start working on it almost right away, especially since I was paid hourly and I was super-aware I was paid to fix the problems and not to learn the code. IMHO what this article explains is one of the most valuable bits I retain from that time.
I've never done a bug squash interview. I have done PR review interviews that are similar - given a PR that technically runs but has tons of problems, then see how many of the problems the candidate identities. I really liked that interview.
I was thinking, wouldn't it be fairer to do the test in an online code sandbox where being able to build and install isn't an issue? Then I thought, isn't letting the candidate use their own machine and debugging tools about as biased as hiring a limo driver based on how fast they can drive their own car? It does tell you who's a real "car guy" or "Unix guy", it just has a slight disparate impact by screening for people with money and hobbies. I mean, I think you should be allowed to hire these people because they will be better at the job, but hopefully they could also show that in a sandbox test.
I've written some interview questions before and can confirm that having people find and fix simple issues in code is a surprisingly great way to weed out people who have no clue what they're doing.
> Cheating effectively is indistinguishable from debugging skill. Even with knowledge of the exact bug ahead of time, if you just open the file with the bug, barf out the code to fix it, and run the tests, you’re going to fail the interview.
Did you mean "distinguishable" here, by chance? The first sentence seems to contradict the second.
A similar exercise but much easier to do in an interview is to give a short piece of code that does something a bit non-trivial and ask them to write (paper or whiteboard) test cases for it. Can then either run the test cases and see if they can find (all) the bug(s), or simply review vs a known list. Classic example in Myers The Art Of Software Testing is a method that takes 3 numbers as the length of the sides of a triangle and prints out whether it is scalene, isosceles, or equilateral. Many people miss a lot of corner cases.
This is indeed a good interview challenge, and as a candidate I’ve enjoyed taking it the couple of times I have encountered it.
I also like the related challenge of, “add a feature that does X” to a codebase.
In computer security, this sort of interview often takes the form of find the vulnerability in this app.
I like it. It feels like much more directly measuring skills than most interview questions.
How much time do you allot for this thing? Cloning the repo, dependency installs, server/DB setups, possible build steps, configuring the IDE to the project ... even if the whole thing is containerized, just going from nothing to running code in an IDE alone could be expected to take 20-30 minutes of the interview.
Compare and contrast https://sockpuppet.org/blog/2015/03/06/the-hiring-post/
I ended up doing this for an entire week, once.
I was hired and presented with a system that didn't work. I knew nothing whatsoever about their proprietary system and wasn't even given access to their codebase. I wondered why they were being so spectacularly unhelpful with onboarding tasks. At the end of the week, I found out - the whole thing had been a test to see how well I would do, and needless to say I failed miserably.
They told me I was a bad fit, and I had to agree, but probably not for the same reasons they were thinking. I really dodged a bullet with that one.
I like this idea but I'm curious how others would implement it. Does the developer have to write the failing test themselves? Are they instructed to do it first, or is that behaviour worthy of bonus points? (Most write a passing test afterwards, which is worse).
> I’ve seen candidates wield their text editor like it was an extension of their fingertips.
So am I allowed to install my own text editor and other tools (Emacs with my own config that requires some build time)? Or is it on my own laptop (I don't have one)? It seems unfair if some candidates get to use their favourite tools and others don't.
This is essentially what I do with candidates. I ask them to pair with me on adding features to a codebase, and also to offer a code review to a PR. 100% success rate on candidates thus far.
I agree. The best interview experience that I had was to be sat in front of a small app that had lots issues, both big and small, and also a lot an not-really-issues-but-still like inconsistent code styles and names, and code comments and code that weren't aligned. This was a great starting point for both me and the interviewer. The interviewer had a list of things to ask into about the code, so that we didn't get stuck.
Having done nearly 100 technical interviews (with multiple questions each), the bug squash question gave us the biggest signal on candidates.
Our question was far simpler: it was a simple class (java + python variants, no fancy syntax) and ask them to describe what it does, then find the bug, and finally ask them what they would change.
It reflects a true test of what the day to day is, and whether or not the candidate would succeed in the role.
Thought it was going to be about squashing a bug or releasing it outside your house, which would tell me a lot more about a potential employee than this.
"Work Sample Test"
Our best received interview question was in this style. Pick something your team fixed or did, distilled down do a 15min thing a teammate could do. Give the candidate 3x the time.
For us, it was a db value not being set as expected during a cron. The candidate got our forged bug reports, cloned a repo, ran the script, verified the unexpected db state, and off to the races
I’ve always liked the Gilded Rose refactoring[1] which is similar but about improvements/feature requests instead.
[1] https://github.com/emilybache/GildedRose-Refactoring-Kata/bl...
I could be wrong but given how fast things change in this business, this seems to test very micro level skills not the learning and adaptability skills.
What if a guy could, given the business or technical issue the code solves, write something faster than he could fix others code?
May be the job is about heads down coding so this bug fix test is appropriate..
I am quite skeptical of item 6 in this list, but otherwise it looks pretty good:
> Cheating effectively is indistinguishable from debugging skill.
If you know the code or problem ahead of time and how to fix it, you can certainly be convincing in your walk-through of how to do it.
> It’s fun because fixing self-contained, reproducible bugs like this is what so many of us enjoy the most about software engineering in the first place.
Do really "many of us" enjoy that? In the "day-to-day business", not interviews.
”what we’re observing is X, but we see Y instead.”
What?
Anyway, I’ve found these kind of interview questions rarely helpful because the scenario is either overly simplistic, or it’s some obscure bug they identified (which they only caught because it caused an outage, i.e. they also didn’t “solve” this when they wrote it).
A similarly poor interview question happens in IT operations when the interviewer asks super specific details about some CVE from 5 years ago that he’s particularly proud of pulling an all nighter to patch a bunch of servers.
it's kind of strange to me that ad-hoc debugging is considered such a valuable skill. I thought it's mostly a juniors' perspective. I typically set up extended loggers and logger methods, write everything to the some formatted file, and have a diffable, persistent, provable, multithread-friendly and versioned bug demonstration, which scales well to bugs of any complexity.
(I once found some 10 bugs in a pretty old and tested bond calculation engine while migrating it to the cloud, which nobody could initially believe.)
My kingdom for this approach to take the place of (some/all) leetcodes.
> I’ve seen candidates drop into strace to debug a Ruby program
strace is great. 99% of the time, your program is looking for a missing file.
A pity it's so hard to use in modern macOS due to OS integrity protection.
Kind of like the "install this software" as part of the interview process.
"My hourly rating for figuring out your bug is $X".
There is academic research on how to interview. Most comments here give no indication the commenter is even aware it exists much less what it says.
I'm aware this exists, but I'll admit to not having read it directly. (HR gives me a list of questions I'm allowed to ask in an interview and they tell me those questions are based on research but I only have their word they are)
I like this approach. We did something similar, but for a position as support engineer, where coding ability was one of the requirements, though not the only one.
They were given a piece of code on a toy, easy to read, english-like pseudo-language, and were asked to examine it and explain what the output to a specific input would be.
They were told that we would answer any question, except what the answer was or what the code was doing, and encouraged them to explain their reasoning as they went ahead.
We wanted to asses the candidates ability to problem solve, communicate, how would they react under preassure, and how they approached the issue.
We were not necessarily looking for perfect answers, we rather rated the candidates willingness to ask for help when stuck, how well they communicated while doing the task, whether they tried different approaches, their ability to grasp what an unknown piece of code was doing and so on... all important properties for a role supporting mission critical solutions with near to zero down-time requirements.
Our reasoning was that technical knowledge is easy to acquire over time, but problem solving skills, how candidates tackle difficult and unknown challanges were more important for the role.
In my current role I try to design interview questions like that, looking both for the candidates current skills as well as their future potential.
Not easy, but IMHO more rewarding for us and them in the long run.
It's easy to write this question. I can cut a branch with gnarly UI bugs in it, no sweat.
Unrelated: why there are no stories about interview questions for managers?
Would love for others to share non-technical questions that can filter bad candidates!
Ah the humble 2 point bug ticket
>Cheating is indistinguishable from debugging skill. Even with knowledge of the exact bug ahead of time, if you just open the file with the bug, barf out the code to fix it, and run the tests, you’re going to fail the interview.
Umm hopefully you don't fail for this. I've definitely known and hired devs that can do this with surprising frequency. Just because the interviewer may have taken hours to find the bug doesn't mean someone else must.
The only downside I see is that I need to share my screen with a person I don't know. Don't get me wrong, but I normally have access to some crypto on my laptop, banking notifications turned on for my 2FAs, private messages I'm getting from my spouse, maybe some borderline NSFW chatgpt dialogs, screenshots of my ideas, emails, private notes, personal projects, whole bunch of tabs I am not willing to close or share, etc.
I don't have issues to show the screen to a friend of mine. Or showing the screen of a work computer to a colleague. However, demoing the whole screen is violation of privacy. Please don't do that, unless you grant VNC access to a machine with a set of popular tools, code editors, etc.
How about reverse bug squash? Interviewer shares the screen with all his/her private info, notes, tabs, development environment settings, shell command history (with maybe autosuggestions turned on), and the interviewee needs to guide the interviewer towards finding the bug?
I like this approach far, far more than coding tests!
> It’s fun. It’s fun in the same way an escape room is fun. It’s fun because of the dopamine you get when the test suite blinks green.
Keep in mind that for lots of people in a job interview setting (even excellent candidates) this is not fun, this is stressful. It may be fun on the job or at home, but the stress of a job interview both eliminates any "fun" aspect and makes even simple tasks more difficult. Just mentioning that to encourage some amount of empathy and understanding for the candidate.