Expert Political Judgment and Hard Facts

A couple of books out right now that bring some understanding, if not comfort, about the world of media and leadership that we frequently lament in this blogosphere of ours. Both deal with responses to complexity and the failure of simplistics (people and ideas) to manage it effectively. Almost by definition, this conclusion shouldn't require major efforts with exhaustively research backing them up, but you know us. In fact, despite the work here, it's very likely that we'll continue on our merry, simplistic way. Still, these works are worth the review.

Philip Tetlock's Expert Political Judgment: How Good Is It? How Can We Know? tells us basically what we want to hear. Thomas Friedman is a moron. Well, Professor Tetlock is too scholarly to say those words exactly, but Tom is likely the first person you'll think of when you read his descriptions of "experts" so wedded to their monolithic view of the world that they can't adjust when the world shifts lanes. However, because they are so forceful and dedicated to their beliefs, able to put forward "solutions" authoritatively, they dominate discourse and search for policy options. Tetlock calls these individuals "hedgehogs," honoring Isaiah Berlin's classic formulation of people who know "one big thing" and stick to it come hell or high water. These are opposed to Berlin's "foxes," who know lots of little things, take the world in far less black and white, are able to deal better with contingencies and complexity, and thus can zig with the world's zigs and zag with . . . well, you get it.

The thing about Tetlock's analysis is that, rather than simply assert what is to me a comforting view of who does better in understanding reality, he proves it true. His methodology, fully and almost deadeningly explained in an appendix for anyone wanting to copy it, was basically to take some known "experts" categorized by their hedgehogness or foxness, give them scenarios about current situations and ask for predictions, complete with rationales. Then he checked for accuracy and asked for explanations about correct and incorrect predictions. Foxes were more likely to consider alternatives and to be open to being wrong and changing situations; hedgehogs were generally able to convince themselves that their beloved theories were still correct, it was reality that was wrong.

Sound familiar?

Tetlock concludes that more complex the situations observed or policy problems of concern, the better foxes will be at dealing with them because of their openness to contrary information and willingness to deviate from initial plans. However, in stable environments without much change going on, hedgehogs who have managed to glom onto that stability effectively will outproduce foxes who will still tend to search for the nuances and ambiguities. Since the former situations dominate the latter right now, hedgehogs get the worse of it. But, because they sound so confident and can explain away their inaccuracies so convincingly, all in ready-made bumper sticker simplicity, they suit our brain-challenged media much more than people who might actually get us out of our messes.

Dealing effectively with a complex world with flows of varying stability is what Tetlock calls "a precarious balancing act." It requires being able "to monitor our own thought processes for telltale signs of excessive closed- or open-mindedness and to strike a reflective equilibrium faithful to our conceptions of the norms of fair intellectual play." He says of the best analysts in his experiments, "if I had to bet on the best long-term predictor of good judgment among the observers in this book, it would be their commitment--their soul-searching Socratic commitment--to thinking about how they think."

Which is, coincidentally, the central message of Jeffrey Pfeffer and Robert I. Sutton's new Hard Facts, Dangerous Half-Truths, & Total Nonsense: Profiting from Evidence-Based Management. (Why they didn't just call it Good Nonsense and Bad, I'm not sure.) Pfeffer is one of the few management theorists I have ever paid much attention to, with really good past works on power and leadership, because he gets the whole "fox"/"hedgehog" thing while the bulk of that literature, at least the popularly available kind, deals with half-truths gussied up as the one true answer. Which are, coincidentally, the focus of the authors' concern.

Way too many decisions are influenced by way too many slogans describing accurately only a few situations, but that have been inflated into guides for all situations. They list and dismantle six--"work is fundamentally different from the rest of life," "the best organizations have the best people," "financial incentives should drive company performance," "strategy is destiny," "change or die," and "great leaders control their companies." It's not that there's NO truth to these missives, it's just that the evidence for them in most situations is weak, nonexistent, or actually the opposite of reality. Real reality, they assert, is too complex for simplistic theories to handle well, and real harm can be done by those who insist on sticking with them come hell or high water. Pfeffer and Sutton tell us that wisdom is "knowing what you know and what you don't know and finding a midpoint between overconfidence and insecurity." The best leaders, like the best "experts," they say, "are lifelong students because they are curious and driven to keep learning what works best for their companies." Tetlock will be glad to be confirmed.

Thus, their mission is to emphasize the importance of basing decisions and management on, get this wild advice, actual evidence. While this sounds fundamental, it's surprisingly rare, as they demonstrate with many cases, as well as extolling the virtues of those companies that practice "evidence-based" management. They don't get nearly as much into government policy, but I can tell you from unfortunate experience in criminal justice, public education, and higher education that very little public decision-making has very much evidence in support of it. Georgi Bushnev and his "my way" advisors were no surprise to me. Ideology and idiocy dominate, and evidence is only good as a weapon in the politics of it all. As Wendy Kaminer said (I think) in It's All the Rage, "Knowledge isn't power in criminal justice debates. Knowledge is irrelevant." The price of that for companies with hedgehog leaders can be very high in their changing market environments. It's also high for nations with hedgehog leaders in changing power environments. Confusion and failure will be the likely outcomes for the companies and nations. (Again, not exactly their words, but the book's relevance is greater than just for business.)

Don't worry, though. Our media will produce the experts needed to explain it all to us and show us the way.