How We Know What Isn't So (10 page)

The most direct evidence for this claim comes from a study in which people who had bet on professional football games provided tape-recorded accounts of their thoughts about the outcomes of their bets. (Their thoughts were recorded in the guise of keeping a record for themselves to help them make additional bets later in the season.) An analysis of their comments indicated that they spent more time discussing their
losses
than their wins. Furthermore, the kind of comments made about wins and losses were quite different. The bettors tended to make “undoing” comments about their losses—comments to the effect that the outcome would have been different if not for some anomalous or “fluke” element (“… it was just luck. Their quarterback got hurt during the game and that probably led to their defeat.”). In contrast, they tended to make “bolstering” comments about their wins—comments indicating that the outcome either should have been as it was, or should have been even more extreme in the same direction (“I don’t think you can put the blame on losing the quarterback. He is an exceptional quarterback, but so is their backup”). By carefully scrutinizing and explaining away their losses, while accepting their successes at face value, gamblers do indeed rewrite their personal histories of success and failure. Losses are often counted, not as losses, but as “near wins.”

One consequence of the greater amount of time the bettors spent scrutinizing their losses is particularly noteworthy: They remembered their losses better than their wins when tested three weeks later. This contradicts everyday intuition as well as a good deal of psychological theorizing that would have us believe that people remain confident in the possibility of future success by selectively remembering their successes and forgetting their failures.
⁸

The studies of gambling and of capital punishment demonstrate that we do not generally treat information at variance with our beliefs as lightly as is sometimes thought, although such information
is
dealt with in such a way that it has relatively little impact on our beliefs. Rather than simply ignoring contradictory information, we often examine it particularly closely. The end product of this intense scrutiny is that the contradictory information is either considered too flawed to be relevant, or is redefined into a less damaging category. Opponents of the death penalty come to view evidence supporting the deterrent efficacy of capital punishment as hopelessly deficient and uninformative. Gamblers come to see negative outcomes not as losses that signal the difficulty of ever coming out ahead, but as near-wins that call for just a little strategic fine-tuning.

Gamblers and partisans of the capital punishment debate are not the only ones who fail to treat supportive and antagonistic information evenhandedly. Scientists have been known to do the same when evaluating evidence relevant to their fields. The methodological critiques and publication recommendations of peer reviewers, for example, have been shown to be greatly affected by whether the results of a study support or oppose the reviewer’s own theoretical orientation.
⁹
Every experimental psychologist I know is much more likely to run an additional experiment if the results of an initial study refute a favored hypothesis than if the results support it. More vividly, the history of scientific attempts to relate brain size or body shape to intelligence, personality, and (often by implication) “social worth” is riddled with examples of investigators vigorously challenging and reinterpreting unanticipated results while glossing over similar flaws and ambiguities in more comfortable findings. The French craniologist Paul Broca could not accept that the German brains he examined were on average 100 grams heavier than his sample of French brains. As a consequence, he adjusted the weights of the two brain samples to take into account extraneous factors such as overall body size that are related to brain weight. However, Broca never made a similar adjustment for his much-discussed difference in the brain sizes of men and women.
¹⁰
The “criminal anthropologist” Cesare Lombroso supported his thesis about the primitive and animalistic nature of criminals and “lower races” by citing numerous examples of their insensitivity to pain—examples that he construed as courage and bravery when exhibited by a privileged European.
¹¹

Although the history of science contains numerous examples of an investigator’s expectations clouding his or her vision and judgment, the most serious of these abuses are overcome by the discipline’s insistence on replicability and the public presentation of results. Findings that rest on a shaky foundation tend not to survive in the intellectual marketplace. To a lesser extent, the same is true with regard to beliefs formed in everyday life: Some of our most erroneous beliefs are weeded out by the corrective influence of our peers and society at large (although see Chapter 7 for a discussion of the limits of this phenomenon in everyday life). The biggest difference between the world of science and everyday life in protecting against erroneous beliefs is that scientists utilize a set of formal procedures to guard against the sources of bias and error discussed in this book—a set of procedures of which the average person is insufficiently aware, and has not adequately adopted in daily life. Scientists employ relatively simple statistical tools to guard against the misperception of random sequences discussed in Chapter 2. They utilize control groups and random sampling to avoid drawing inferences from incomplete and unrepresentative data (Chapter 3). They use “blind” observers as one way of eliminating the influence of the biased evaluation processes discussed in this chapter.
*

But perhaps the most fundamental safeguard of the scientific enterprise is the requirement that the meaning of various outcomes be precisely specified (in advance if possible) and objectively determined. If a scientist sets out to test the ability of subliminal self-help tapes to improve the productivity of salespeople, he or she would doubtless focus on actual sales volume, and would ignore the claims of enhanced confidence, improved poise, and increased energy from those who were exposed to the tapes. (If such testimonials were to be used at all, it would be as suggestions for further hypotheses that would themselves be subjected to rigid test.) This kind of precise specification of what constitutes “success” and “failure” is something we rarely do in everyday life, and consequently our preconceptions often lead us to interpret the meaning of various outcomes in ways that favor our initial expectations. If we are interested in informally testing the effectiveness of vitamin C with the data of our own experience, it may be wise to specify in advance that “success” or “improvement” be defined as a reduction in the number of days with a cold. If not, we run the risk of reading too much into every moment’s respite from post-nasal drip or any temporary reduction in our fever-induced nagging of loved ones.

To stretch this idea a bit further (and pursue a theme introduced earlier), the methods of science protect an investigator from juggling the meaning of different results by deliberately making the investigator rigid and “unintelligent” in the same way that computers are rigid and unintelligent. Experimental results, like the input to a computer, must fall into certain pre-specified slots according to pre-specified rules or they are not processed at all. As scientists, we willingly sacrifice some “intelligence” and flexibility for the benefit of objectivity.

This is not to suggest, of course, that all of science is such a rigid, constrained process. A distinction must be made between the processes involved in generating versus testing ideas; between what philosophers of science have referred to as the “context of discovery” and the “context of justification.” In the context of discovery, “anything goes” in science as in everyday life; it is in the context of justification that scientists become more conservative. As Sir Peter Medawar has noted, science works “… in a rapid reciprocation of guesswork and checkwork, proposal and disposal, conjecture and refutation.”
¹²
Flashes of inspiration are followed by rigorous test. When asked on a talk show to explain the secret of his success, two-time Nobel Laureate Linus Pauling once replied that “… you need to have a lot of ideas, and then you have to throw away the bad ones.” Much of the scientific enterprise can be construed as the use of formal procedures for determining when to throw out bad ideas, a set of procedures that we might be well advised to adopt in our everyday lives. We humans seem to be extremely good at generating ideas, theories, and explanations that have the ring of plausibility.
¹³
We may be relatively deficient, however, in evaluating and testing our ideas once they are formed. One of the biggest impediments to doing so is our failure to realize that when we do not precisely specify the kind of evidence that will count as support for our position, we can end up “detecting” too much evidence for our preconceptions.

Another way of stating this is that our expectations can often be confirmed by any of a set of “multiple endpoints” after the fact, some of which we would not be willing to accept as criteria for success beforehand.
¹⁴
When a psychic predicts that “a famous politician will die this year,” it is important to specify then and there the range of events that will constitute a success. Otherwise, we are likely to be overly impressed by various tenuous connections between the prediction and any of a number of subsequent events. Suppose Armand Hammer dies within the year: Is that a successful prediction? (He is an industrialist rather than a politician, but he has served as this country’s ambassador-without-portfolio to Moscow for several generations.) Or suppose the President is shot in an unsuccessful assassination attempt: Does that count? Without specifying the meaning of all possible outcomes, the test is no longer objective, and we run the risk that our initial hypotheses will receive apparent support too easily.

The problem of multiple endpoints is most severe when the subject under investigation is inherently fuzzy and hard to define. For instance, suppose someone claims that day care during infancy hinders “personal adjustment” in later life. Well, what is “personal adjustment” and how does one measure it? The number of friends during adolescence? Academic success? Happiness with chosen career? It is at times such as these, when the meaning of the phenomenon under investigation is unclear, that our preconceptions have their greatest effect. Any measure of personal adjustment that supports our initial beliefs is likely to be seized upon as the “true” test. In contrast, if someone were to claim that day care during infancy hinders subsequent “scholastic achievement,” there is less flexibility in how it should be defined (although some remains) and therefore less latitude for our preconceptions to exert an effect.

An interesting analogue of the problem of multiple endpoints is what could be called the problem of “variable windows.” The essence of a number of beliefs is that certain events tend to happen within some (unspecified) period of time. The belief that things “happen in threes” is a perfect example: Many people believe that events like plane crashes, serial-killing sprees, or birth announcements tend to occur in triplets. It is almost certainly the case, however, that these beliefs are mere superstitions that stem from the tendency to allow the occurrence of the third event in the triplet to
define
the period of time that constitutes their “happening together.” If three plane crashes occur in a month, then the period of time that counts as their happening together is one month. If the third plane crash does not happen for another month, the relevant period of time is stretched to two months. By allowing the window of opportunity to be sufficiently flexible, such beliefs can
only
be confirmed.

People often comment on the resemblance between a newborn baby and one or both of the parents. “He has his mother’s eyes.” “She sure has that Gilovich nose.” Interestingly, these same observations are often made when the child, unknown to the observer, has been adopted. Even when there is no genetic connection between parent and child, it is still possible to detect, from the vast number of possible features, a few striking similarities.

This phenomenon illustrates a particularly common result of the problem of multiple endpoints that gives rise to a specific class of erroneous belief. Certain beliefs or suppositions imply a similarity between two entities: A child should look like his or her parents, identical twins should behave alike, or a personality description ought to resemble the person it describes. However, if the two entities are sufficiently complex, then mapping one onto the other will almost certainly produce a number of points of overlap, and the expectation will appear to be confirmed.

One of the best examples of this phenomenon is the “Barnum effect,” named after circus entrepreneur P. T. Barnum because it was he who said “there’s a sucker born every minute.” The Barnum effect refers to the tendency for people to accept as uncannily descriptive of themselves the same generally worded assessment, as long as they believe it was written specifically for them on the basis of some “diagnostic” instrument such as a horoscope or personality inventory. Consider the following description: