Long before he won the Nobel Prize for Economics, Daniel Kahneman was a psychology officer for the Israeli Defence Force. He got a lot out of his time in the army too, including inspiration for the illusion of validity.
Kahneman invented this concept after evaluating officer candidates in a "leaderless group challenge," according to his book Thinking, Fast And Slow.
This was a drill where 8 candidates who didn't know each other, with no markings of rank, were instructed to carry themselves and a long log over a six-foot wall without touching the wall. The test required ingenuity and teamwork (getting the eighth man across typically involved him jumping at the log held at an angle by the seven men on the other side) and often resulted in failure.
In the process, the test was supposed to reveal who was a good leader.
Kahneman and his colleagues felt like it was very effective test: "We were completely confident in our evaluations and felt that what we had seen pointed directly to the future."
Unfortunately, it didn't work:
The evidence that we could not forecast success accurately was overwhelming. Every few months we had a feedback session in which we learned how the cadets were doing at the officer-training school and could compare our assessments against the opinions of commanders who had been monitoring them for some time. The story was always the same: our ability to predict performance at the school was negligible. Our forecasts were better than blind guesses, but not by much.
Despite the failure of the test, the IDF continued using it. What's more, Kahneman and his colleagues continued to feel confident with each prediction that they were getting it right.
This hypocrisy inspired a breakthrough:
What happened was remarkable. The global evidence of our previous failure should have shaken our confidence in our judgments of the candidates, but it did not. It should also have caused us to moderate our predictions, but it did not. We knew as a general fact that our predictions were little better than random guesses, but we continued to feel and act as if each of our specific predictions was valid. I was reminded of the Müller-Lyer illusion, in which we know the lines are of equal length yet still see them as being different. I was so struck by the analogy that I coined a term for our experience: the illusion of validity.
Wikipedia describes the illusion of validity as "a cognitive bias described by Amos Tversky and Daniel Kahneman in which consistent evidence persistently leads to confident predictions even after the predictive value of the evidence has been discredited."
Kahneman's book also discusses the cognitive failures of pundits, financial advisors, amateur investors, and pretty much everyone.
Don't miss: 57 Behavioral Biases That Make Us Think Irrationally >