Authors: Ian Ayres
Justin didn't stop there. The future belongs to the Super Cruncher who can work back and forth and back again between his intuitions and numbers. The graph led Justin to hypothesize about point shaving. His hypothesizing led him to look for further tests that could confirm or disconfirm his hypothesis. He dug further in and found that if you looked at the score five minutes before the end of the game, there was no shortfall. The favored team was right on track to beat the spread 50 percent of the time. It was just in the last five minutes that the shortfall appeared. This isn't proof positive, but it does make a stronger circumstantial caseâafter all, that's the safest time for a bribed player to let up a bit, secure in the knowledge that it won't lead to his team losing.
The future belongs to people like Wolfers who are comfortable with both intuition and numbers. This “new way to be smart” is also for the consumers of Super Crunching. Increasingly, it will be useful for people like Anna to be able to quantify their intuitions. It is also important to be able to restate other people's Super Crunching results in terms that internally make intuitive sense.
One of the very coolest things about Super Crunching is that it not only predicts but also simultaneously tells you how accurate its prediction is. The standard deviation of the prediction is the crucial measure of accuracy. Indeed, the 2SD rule is the key to understanding whether a prediction is so accurate that Super Crunchers say it is “statistically significant.” When statisticians say that a result is statistically significant, they are really just saying that some prediction is more than two standard deviations away from some other number. For example, when Wolfers says that the shortfall in favored teams covering the spread is statistically significant, he means that their 47 percent probability of covering is more than two standard deviations below the 50 percent probability he would predict if it was really a fair bet.
The designation of “statistical significance” is taken by a lot of people to be some highly technical determination. Yet it has a very intuitive explanation. There is less than a 5 percent chance that a random variable will be more than two standard deviations away from its expected mean (this is just the flip side of the 2SD rule). If an estimate is more than two standard deviations away from some other number, we say this is a statistically significant difference because it is highly unlikely (i.e., there is less than a 5 percent probability) that the estimated difference happened by chance. So just by knowing the 2SD rule, you know a lot about why “statistical significance” is pretty intuitive.
In this chapter, I hope to give you an idea for what it feels like to toggle back and forth between intuitions and numbers. I'm going to do it by introducing you to two valuable quantitative tools for the man or woman of the future. Reading this will not train you enough to be a full-fledged Super Cruncher. Yet learning and playing with these tools will put you well on the road to the wonderful dialectic of combining intuitions and statistics, experience and estimates. You've already started to learn how to use the first toolâthe intuitive measure of dispersion, the standard deviation. One of the first steps is to see if you can communicate what you know to someone else.
A World of Information in a Single Number
When I taught at Stanford Law School, professors were required to award grades that had a 3.2 mean. Students would still obsess about how professors graded, but instead of focusing on the professor's mean grade, they'd obsess about how variable the grades were around the mandatory mean. Innumerable students and professors would engage in inane conversations where students would ask if a professor was a “spreader” or “clumper.” Good students would want to avoid clumpers so that they would have a better chance at getting an A, while bad students hated the spreaders who handed out more As but also more Fs.
The problem was that many of the students and many of the professors had no way to express the degree of variability in professors' grading habits. And it's not just the legal community. As a nation, we lack a vocabulary of dispersion. We don't know how to express what we intuitively know about the variability of a distribution of numbers.
The 2SD rule could help give us this vocabulary. A professor who said that her standard deviation was .2 could have conveyed a lot of information with a single number. The problem is that very few people in the U.S. today understand what this means. But you should know and be able to explain to others that only about 2.5 percent of the professor's grades are above 3.6.
It's amazing the economy of information that can be conveyed in just a few words. We all know that investing in the stock market is risky, but just how risky is risky? Once again, standard deviations and the 2SD rule come to our rescue. Super Crunching regression tells us that the predicted return next year of a diversified portfolio of New York Stock Exchange stocks is 10 percent, but that the standard deviation is 20 percent. Just knowing these two numbers reveals an incredible amount.
Suddenly we know that there's a 95 percent chance that the return on this portfolio will be between minus 30 percent and positive 50 percent. If you invest $100, there's a 95 percent chance that you'll end the year with somewhere between $70 and $150. The actual returns on the stock market aren't perfectly normal, but they are close enough for us to learn an awful lot from just two numbers, the mean and the standard deviation.
Indeed, once you know the mean and standard deviation of a normal distribution, you know everything there is to know about the distribution. Statisticians call these two values “summary statistics” because they summarize all the information contained in the entire bell curve. Armed with a mean and a standard deviation, we can not only apply the 2SD rule; we can also figure out the chance that a variable will fall within any given range of values. Want to know the chance that the stock market will go down this coming year? Well, if the expected return is 10 percent and the standard deviation is 20 percent, you're really asking for the chance that the return will fall more than one half of a standard deviation below the mean. Turns out the answer (which takes about thirty seconds to calculate in Excel) is 31 percent.
Exploiting this ability to figure out the probability that some variable will be above or below a particular value pays even greater dividends in political polls.
Probabilistic Leader of the Pack
The current newspaper conventions on how to report polling data are all screwed up. Newspaper articles tend to say something like: “In a Quinnipiac poll of 1,243 likely voters, Calvin holds a 52 percent to 48 percent advantage over Hobbes for the Senate seat. The poll's margin of error is plus or minus two percentage points.”
How many people understand what the margin of error really means? Do you? Before going on, write down what you think is the chance that most people in the state really support Calvin.
It should come as no surprise that the margin of error is related to the font of all statistical wisdom, the 2SD rule. The margin of error is nothing more than two standard deviations. So if the newspaper tells you that the margin of error is two percentage points, that means that one standard deviation is one percentage point. We want to know what proportion of people in the entire state population of likely voters support Calvin and Hobbes, but the sample proportions by chance might be misrepresentative of the population proportions. The standard deviation measure tells us how far the sample predictions might stray by chance from the true population proportions that we care about.
So once again we can apply our friend, the 2SD rule. We start with the sample proportion that supports Calvin, 52 percent, and then construct a range of numbers by adding on and subtracting off the margin of error (which is two standard deviations). That's 52 percent plus or minus 2 percent. So using the 2SD rule we can say, “There is a 95 percent chance that somewhere between 50 percent and 54 percent of likely voters support Calvin.” Printing something like this would provide a lot more information than the cryptic margin of error disclaimer.
Even this 95 percent characterization fails, however, to emphasize an even more basic result: the probability that Calvin is actually leading. For this example, it's pretty easy to figure out. Since there is a 95 percent chance that Calvin's true support in the state is between 50 percent and 54 percent, there is a 5 percent chance that his true support is in one of the two tails of the bell curveâeither above 54 percent or below 50 percent. And since the two tails of the bell curve are equal in size, there is just a 2.5 percent chance that Calvin's statewide support is less than 50 percent. That means there's about a 97.5 percent chance that Calvin is leading.
Reporters are massively misinformed when it comes to figuring out the probability of leading. If Laverne is leading Shirley 51 percent to 49 percent with a margin of error of 2 percent, news articles will say that the race is “a statistical dead heat.” Balderdash, I say. Laverne's polling result is a full standard deviation above 50 percent. (Remember, the margin of error is two standard deviations, so in this example one standard deviation is 1 percent.) Crunching these numbers in Excel tells us in a few seconds that there is an 84 percent chance that Laverne currently leads in the polls. If something doesn't change, she is your likely winner.
In many polls, there are undecideds and third-party candidates, so the proportions of the two leading candidates often add up to less than 100 percent. But the probability of leading tells you just what it saysâthe probable leader of the pack.
People have a much easier time understanding proportions and probabilities than they do standard deviations and margins of error. The beauty of the 2SD rule is that it provides a bridge for translating one into the other. Instead of reporting the margin of error, reporters should start telling people something that they intuitively understand, the “probability of leading.” Standard deviations are our friends, and they can be used to tell even the uninitiated about things that we really do care about.
Working Backwards
But wait, there's more. The stock and survey examples show that if you know the mean and standard deviation, you can work forward to calculate a proportion or probability that tells people something interesting about the underlying process. Yet sometimes it's useful to work backward, starting with a probability and then estimating the implicit standard deviation that would give rise to that result. Lawrence Summers got into a lot of trouble for doing just this.
On January 14, 2005, the president of Harvard University, Lawrence Summers, touched off a firestorm of criticism when he spoke at a conference on the scarcity of women professors in science and math. A slew of newspaper articles characterized his remarks as suggesting that women are “somehow innately deficient in mathematics.” The
New York Times
in 2007 characterized Summers's remarks as claiming that “a lack of intrinsic aptitude could help explain why fewer women than men reach the top ranks of science and math in universities.” The article (like many others) suggested that the subsequent furor over Summers's speech contributed to his resignation in 2006 (and the decision to replace him with the first female president in the university's 371-year history).
Summers's speech did in fact suggest that there might be innate differences in the intelligence of men and women. But he didn't argue that the average intelligence of women was any less than that of men. He focused instead on the possibility that the intelligence of men is more variable than that of women. He explicitly worked backwards from observed proportions to implicit standard deviations. Here's what Summers said:
I did a very crude calculation, which I'm sure was wrong and certainly was unsubtle, twenty different ways. I lookedâ¦at the evidence on the sex ratios in the top 5 percent of twelfth graders [in science and math]. If you look at thoseâthey're all over the map [but] one woman for every two men would be a high-end estimate [for the relative prevalence of women]. From that, you can back out a difference in the implied standard deviations that works out to be about 20 percent.
Summers doesn't say it, but his calculation assumes what researchers have in fact found: there is no pronounced difference in the
average
math or science scores for male and female twelfth graders. But in a variety of different studies, researchers have found a difference in the tails of the distribution. In particular, Summers focused in on the tendency for there to be two men for every one woman when you looked at the top 5 percent of math and science achievement among twelfth graders. Summers worked backwards to figure out what kind of a difference in standard deviations would give rise to this sex difference in the tails. His core claim, indeed his only claim, of innate difference was that the standard deviation of men's intelligence might be 20 percent greater than that of women.
Summers in the speech was careful to point out that his calculation was “crude” and “unsubtle.” But Summers is no dummy. He is the youngest person ever to be voted tenure at Harvard. He won the prestigious John Bates Clark award for the best U.S. economist under forty. Two of the three greatest American economists of the twentieth century, Kenneth Arrow and Paul Samuelson, are his uncles, and like them Summers in his early forties was headed straight for a Nobel Prize. He definitely understands standard deviations. But after almost dying from Hodgkin's disease, Summers chose a different path. Like Paul Gertler of Progresa fame, he became chief economist for the World Bank and eventually went on to be secretary of the treasury at the end of the Clinton administration. He is almost always the smartest person in the room (and his critics say he knows it).
Being smart, however, does not mean that everything you say is right. Summers's back-of-the-envelope empiricism doesn't definitely resolve the question of whether women have less variable intelligence. For example, lots of other factors could have influenced the math and science scores of twelfth graders besides innate ability. Yet there have been subsequent studies suggesting that the IQ scores of women are in fact less variable than those of men.