What is the chance we get it right when we pick a CEO, given 10% can do work and we can identify with 80% accuracy?

• A. 8%
• B. 18%
• C. 26%
• D. 31%

Answer: (D)

This question shows how a test’s accuracy combines with how rare the skill is in the population. Even a reasonably accurate test can yield a low probability that someone flagged as capable is actually capable when the true-capable group is small.

Think through 100 people: 10 can do the work, 90 cannot. With 80% accuracy, 8 of the truly capable are correctly identified as capable, while 20% of the non-capable are incorrectly flagged as capable, which is 18 people. So there are 26 people your test would call capable, but only 8 of them actually can do the work.

The chance that a person flagged as capable is truly capable is 8 out of 26, which is 8/26 ≈ 0.3077, about 30.8%, i.e., roughly 31%.

So, even with an 80% accurate identification, the probability of hiring a truly capable CEO in this scenario is about 31%.