By Scott Berkum
Royal Winchester is a very smart guy. He recently offered me this theory on interviews.
The theory goes as follows: Interviewing is mostly bullshit.
Most of us make instinctive judgments on factors we don’t understand in the first five minutes, and spend the rest of the time, and the time discussing with other interviewers, back-filling logical reasons to support an intuitive response we’re largely in denial of.
Not everyone does this of course, and not all the time, but the theory suggests it’s true much of the time, or is a significant factor in interviews.
There are some contributing hypotheses to the theory:
- Few are mature enough to sort out their biases. Very few people posses the self awareness to realize why they instinctively like or do not like someone they’ve just met. And even fewer, especially among the business/engineering crowd, feel comfortable with their feelings. It’s considered unacceptable to say ‘the guy did well but I didn’t like him for reasons I can’t explain’. It’s much easier to hide that feeling inside unfair judgments, using whatever flavor of corporate jargon can be found in the official hiring criteria (Lacked intellectual horsepower, couldn’t deal with ambiguity, didn’t know the secret handshake, etc.)
- Talking about doing is not doing. Most interviewers focus on trying to extract a prediction about someone’s ability through having them talk about their ability. This is ridiculous. Could you evaluate an NFL running back by asking them questions about how they run? (e.g. “I run really really fast”, “Great, you’re hired.”). Better interviewers work hard to put candidates in problems and situations like the real ones they’ll face, and watch. They collaborate on real problems during the interview, as that’s what much of work is. Over time they’re able to calibrate what it means for a candidate to do well, given real problems, in an hour. But this requires skill and patience few interviewers have. And even when they do, the candidates are in an awkward and artificially stressful environment that does not approximate real work well, unless the interviewer is diligent on compensating for these issues (tip: hiring candidates for a trial project is often a better use of everyone’s time. Get a sample of them actually doing the job).
- Interviews work better as a filter. The job interview loop is more effective at eliminating bad candidates than identifying good ones. The bet is by the end only good candidates remain, but that’s not true. Like bacteria responding to antibiotics, strains of bad candidates that are immune to your process survive as well, and are hard to distinguish from good ones. The process can be prone to false negatives too (people who get rejected but would have thrived).
- Recommendations are underestimated. Since interviews are mostly bullshit, it makes sense to put more weight on a recommendation from a trusted person (not necessarily the names on the candidates resume) who has worked with the candidate somewhere else. They have first hand experience on the millions of things that can only be witnessed outside of the interview room. If you trust them, and they trust the candidate, that may have more predictive ability than 60 awkward minutes in your office.
- No one else saw what happened. Interviewers are free to lie and distort, intentionally or not. All interviewers are free to invent pet theories on which questions work best, or how good they are an extracting the value of a candidate. They are the only record of what they asked, how they asked it, and how the candidate performed. If they have bad habits that bias the candidate, no one will ever know, as the candidate has almost no ability to report on the interviewer. Every interview is a cat and mouse trapped in the room, and the mouse is motivated to do whatever it can to survive the cat, no matter how cruel or unfair the cat is.
- We never go back a year later and evaluate. The hiring loop at nearly all companies is broken, as there is no feedback loop. No one forces you to go back 6 months or 2 years later and see: how many of the hire decisions you made worked out well, and how many of the people you rejected kicked ass at other companies with similiar cultures and needs. With no data, the value of any interview process is guesswork, not rigor.
Despite my affinity for this theory, I believe groups that take interviewing seriously, and leaders who reward interviewers for putting more time and careful thought into interviews, end up with better teams. The choice to hire someone is the most important decision you make that month, or year and the wise know this. At a start-up it can make or break the company. And the more seriously people take the process, even if it’s flawed, the higher the odds they’ll recognize the natural shortcomings above and invest in minimizing them.
While I don’t think all interviews are a crapshoot, I agree with the Winchester theory – in most interviews, most of the time, it’s mostly bullshit as a tool in truly evaluating how well a person would perform in the job, even if the people doing the interviews don’t intend it to be.