How We Know What Isn't So (31 page)

A set of important mental habits we also need to develop are those that can help to overcome the drawbacks associated with one of our most remarkable skills—the facility with which we can explain a vast range of outcomes in terms of our pre-existing theories and beliefs. Because of our talent for
ad hoc
explanation, even quite unexpected and damaging outcomes can be seen to be consistent with our original convictions. Our beliefs thus appear to receive too much support from equivocal evidence, and they are too seldom discredited by truly antagonistic results. To compensate, we need to develop the habit of employing one of several “consider the opposite” strategies. We can learn to ask ourselves, for example, “Suppose the exact opposite had occurred. Would I consider
that
outcome to be supportive of my belief as well?” Alternatively, we can ask, “How would someone who does not believe the way I do explain this result?”, or, more generally, “What alternative theory could account for it?” By asking these questions, we become aware that the link between evidence and belief is not so tight as it might first appear. These strategies thus help to guard against premature acceptance of doubtful propositions, and they encourage us to figure out (and try to obtain) the evidence necessary to truly test a belief’s validity.

There are a number of other, less general habits of mind that are helpful in warding off many of the sources of erroneous beliefs discussed in earlier chapters. Guidelines for dealing with the uncertainties and distortions of secondhand information were offered in Chapter 6. To these it is important to add that we would be well advised to consider the possibility that information that comes to us from others may be more remote than it first appears. That which is described as secondhand is often thirdhand, that which is passed off as thirdhand is often even more distant, and so on. Events described to us by a trustworthy source may nonetheless have originated with someone less credible. We therefore should be more skeptical than we seem to be about evidence presented to us secondhand. We should become accustomed to asking ourselves where the information originated, and how much distortion—deliberate or otherwise—is likely to have been introduced along the way.

Chapter 7 suggests that we should question whether our beliefs are really as widely shared as they appear. The absence of explicit disagreement should not automatically be taken as evidence of agreement. Chapter 2 calls for an awareness of the human tendency to impute order to any complex set of stimuli and an understanding of when and where statistical regression is likely to occur. An appreciation of both phenomena should encourage us to consider the “just chance” hypothesis and to be less prone to rush to judgment and intervention.

Many of these essential habits of mind, particularly the most general ones for dealing with incomplete and unrepresentative evidence, were originally developed as part of the scientific enterprise. For instance, the idea that what one observes under one set of conditions can only be evaluated with reference to what would have happened under slightly different circumstances is embodied in the scientist’s use of the control group. Procedures for distinguishing random from ordered phenomena were developed not long ago in the field of statistics. Statistical regression was discovered through the study of genetic inheritance. And so on.

It stands to reason, then, that greater familiarity with the scientific enterprise should help to promote the habits of mind necessary to think clearly about evidence and to steer clear of dubious beliefs. Involvement in the process and concepts of science not only teaches these habits of mind directly, it also provides experience with problems, phenomena, and strategies from which they can sometimes be intuited, or at least more deeply understood. Also, one who participates in the scientific enterprise receives valuable exposure to uncertainty and doubt. Because science tries to stretch the limits of what is known, the scientist is constantly thrust against a barrier of ignorance. The more science one learns, the more one becomes aware of what is
not
known, and the provisional nature of much of what is. All of this contributes to a healthy skepticism toward claims about how things are or should be. This general intellectual outlook, this awareness of how hard it can be to really know something with certainty, while humbling, is an important side benefit of participating in the scientific enterprise.

A number of authors have recently written about the woeful state of science and mathematics education in the United States and its role in producing a citizenry that is not at all critically minded. Sometimes the concern is with whether the voting public will be able to develop well-informed opinions about the increasingly complex issues that are part of our technological world; at other times the concern is about the state of our more abstract abilities to reason effectively. One can only agree with the general argument that generating more interest in the scientific enterprise would be helpful in these regards.

There is an intriguing twist to this general contention, however. A set of recent studies suggests—albeit only tentatively at this point—that a particular kind of science education may be especially effective in developing the habits of mind necessary for thinking clearly about the evidence of everyday experience. The logic that motivated these studies was quite simple: Exposure to the “probabilistic” sciences may be more effective than experience with the “deterministic” sciences in teaching people how to evaluate adequately the kind of messy, probabilistic phenomena that are often encountered in everyday life. Probabilistic sciences are those such as psychology and economics that deal mainly with phenomena that are not perfectly predictable, and with causes that are generally neither necessary nor sufficient. The death of a spouse, for instance, is associated with a deterioration of health in the bereaved, but not for all widows and widowers, and ill health often descends for other reasons. Thus, bereavement is neither a necessary nor sufficient cause of ill health. Likewise, attractive people are generally responded to more favorably than the unattractive, but not all beautiful people are well liked, and good looks are not a requirement for winning another person’s esteem or affection.

Deterministic sciences, on the other hand, are those such as chemistry and many branches of physics that typically deal with much tidier phenomena for which the causal connections are more often necessary and sufficient. To increase the gravitational attraction of two objects of given mass, it is both necessary and sufficient to move them closer together. It is with respect to the uncertain phenomena studied by the probabilistic sciences that ideas like statistical regression, sample bias, and the importance of control groups are particularly germane. Familiarity with these fields, then, should best facilitate the habits of mind necessary to evaluate properly the evidence of everyday experience.

To test this idea, a group of psychologists administered a test of statistical and methodological reasoning to students receiving graduate training in psychology, chemistry, medicine, and law.
¹
As a control procedure to assess any differences in general learning across the four disciplines, the students were also administered the verbal reasoning subtest of the Graduate Record Examination (GRE). The design of this study was both cross-sectional and longitudinal: First- and third-year graduate students in each field were compared to one another to assess the effects of their graduate training, and the first-year students were reassessed two years later, with their later performance compared to their original.

Two forms of the test were developed to permit retesting in the longitudinal component of the study. The questions were designed to assess the sophistication of the students’ statistical and methodological reasoning in both scientific and everyday-life contexts. For example, in one of the scientific questions, the subjects were told about a hypothetical teaching experiment and were asked what could be expected to happen to students in the control condition of that experiment who had initially received relatively high or low grades. The question was meant to elicit whether the respondents would exhibit any recognition of the regression principle by stating that students with high initial grades could be expected to receive somewhat lower grades subsequently, and those with low initial grades could be expected to receive somewhat higher grades. In an everyday-life problem, the subjects were told about a mayor who boasted of the 12% reduction in crime that had taken place during his administration. They were then asked about the kinds of evidence they thought would be necessary to evaluate the validity and import of the mayor’s claim. This question was concerned with whether they would recognize the importance of control-group data by, say, wanting to examine crime rates during the same period in cities of similar size and close geographical location.

The results for both cross-sectional and longitudinal assessments were clear-cut and pointed to the relative effectiveness of social science training in teaching statistical and methodological reasoning. There were no initial differences in test scores across the four disciplines. However, two years of training in psychology led to a 70% increase in test scores, whereas a similar period of training in chemistry or law produced no improvement whatsoever. Medical training also improved statistical and methodological reasoning, with two years of medical school producing a 25% improvement in test scores. Graduate education in the four different disciplines did not produce any reliable differences in improvement on the verbal reasoning sub-test of the GRE (gains ranged from 4% to 17%). The investigators concluded:

It seems, then, that social scientists may have a special opportunity to impart some wisdom about how to properly evaluate the evidence of everyday experience. The authors of the study just described argue that there are certain formal properties of the subject matter of social science (e.g., considerable irregularity and uncertainty, the relative lack of necessary and sufficient causal relations) that make it particularly effective for teaching some important principles of sound reasoning. There are also a number of pragmatic characteristics of social science that add to its effectiveness in this regard. Part of the popularity of college courses in such areas as personality and social psychology derives from the fact that they deal with phenomena that students have encountered and thought about in their everyday lives. Some of the material conflicts with students’ pre-existing beliefs and thus provides much more than the usual incentive to engage in critical analysis, to suggest alternative explanations, and to consider the adequacy of both existing data and other potentially informative evidence. The student is thus encouraged to engage his or her analytic faculties with unusual intensity because the very nature of the material invites it. The complexity of the phenomena, the difficulty of untangling correlated variables, and the relative scarcity of truly decisive experiments compel all but the most disengaged students to dig deeper and think harder. The general principles of scientific inference are straightforward and easy to teach. What is difficult is to teach how and when to apply them. In this respect many branches of the social sciences have an advantage. Many of these fields are concerned with the highly visible processes and phenomena of everyday life in which nearly everyone can take an interest—the best ways to influence other people, the causes of people’s attraction to one another, or the causes and correlates of happiness and well-being. Thus by their very nature, many of the social sciences provide helpful practice in thinking clearly and rigorously about the phenomena of everyday life.

Social scientists suffer from physics envy. From the beginning, they have felt like poor relations among the sciences, unable to match the natural scientists’ cumulative achievements, explanatory power, and predictive precision. There is indeed much to admire about the progress made in the “hard” sciences—progress that the social sciences will likely never match. Nevertheless, it is important to acknowledge that there is a special benefit from studying the messy, complex phenomena that constitute the subject matter of the social sciences. Dealing with such irregular, uncertain phenomena has led to a number of methodological innovations. Social scientists are generally more familiar than those in other fields with how easy it is to be misled by the evidence of everyday experience, and they are more aware of the methodological controls that are necessary before one can draw firm conclusions from a set of data. This may be one reason why fewer psychologists believe in the existence of ESP than their colleagues in the natural sciences or the humanities.
²

As a consequence, what social scientists might best offer both their students and the general public is their methodological sophistication, their way of looking at the world, the habits of mind that they promote—process more than content. In fits and starts, social science has advanced human knowledge a great deal over the years. Nevertheless, much of what we think we have learned will certainly change over the next 50 or 100 years. How we go about our business, on the other hand, and the methods we employ to advance our knowledge, will be largely the same. An awareness of how and when to question and a recognition of what it takes to truly know something are among the most important elements of what constitutes an educated person. Social scientists, I believe, may be in the best position to instill them.