In their 1978 paper Performance Sampling in Social Matches, researchers March and March discussed the implications of performance sampling for understanding careers in organizations. They came to some interesting conclusions with implications for those of us working in organizations.
Considerable evidence exists documenting that individuals confronted with problems requiring the estimation of proportions act as though sample size were substantially irrelevant to the reliability of their estimates. We do this in hiring all the time. Yet we know that sample size matters.
On how this cognitive bias affects hiring, March and March write have some good insights.
False Record Effect
A group of managers of identical (moderate) ability will show considerable variation in their performance records in the short run. Some will be found at one end of the distribution and will be viewed as outstanding; others will be at the other end and will be viewed as ineffective. The longer a manager stays in a job, the less the probable difference between the observed record of performance and actual ability. Time on the job increased the expected sample of observations, reduced expected sampling error, and thus reduced the chance that the manager (of moderate ability) will either be promoted or exit.
Within a group of managers of varying abilities, the faster the rate of promotion, the less likely it is to be justified. Performance records are produced by a combination of underlying ability and sampling variation. Managers who have good records are more likely to have high ability than managers who have poor records, but the reliability of the differentiation is small when records are short.
On the average, new managers will be a disappointment. The performance records by which managers are evaluated are subject to sampling error. Since a manager is promoted to a new job on the basis of a good previous record, the proportion of promoted managers whose past records are better than their abilities will be greater than the proportion whose past records are poorer. As a result, on the average, managers will do less well in their new jobs than they did in their old ones, and observers will come to believe that higher level jobs are more difficult than lower level ones, even if they are not.
…The present results reinforce the idea that indistinguishability among managers is a joint property of the individuals being evaluated and the process by which they are evaluated. Performance sampling models show how careers may be the consequences of erroneous interpretations of variations in performance produced by equivalent managers. But they also indicate that the same pattern of careers could be the consequence of unreliable evaluation of managers who do, in fact, differ, or of managers who do, in fact, learn over the course of their experience.
But hold on a second before you stop promoting new managers (who, by definition, have a limited sample size).
I'm not sure that sample size alone is the right way to think about this.
Consider two people: Manager A and Manager B who are up for promotion. Manager A has 10 years of experience and is an “all-star” (that is great performance with little variation in observations). Manager B, on the other hand, has only 5 years of experience but has shown a lot of variance in performance.
If you had to hire someone you'd likely pick A. But it's important not to misinterpret the results of March and March and dig a little deeper.
What if we add one more variable to our two managers.
Manager A's job has been “easy” whereas Manager B took a very “tough” assignment.
With this in mind, it seems reasonable to conclude that Manager B's variance in performance could be explained by the difficulty of their task. This could also explain the lack of variance in Manager A's performance.
Some jobs are tougher than others.
If you don't factor in degree-of-difficulty you're missing something big and sending a message to your workforce that discourages people from taking difficult assignments.
The importance of measuring performance over a meaningful sample size is the key to distinguishing between luck and skill. When in doubt go with the person that's excelled with more variance in difficulty.