How to Read a Paper: The Basics of Evidence-Based Medicine (18 page)

7.
Become master of the hanging comparative (‘better’—but better than what?).

8.
Invert the standard hierarchy of evidence so that anecdote takes precedence over randomised trials and meta-analyses.

9.
Name at least three local opinion leaders who use the drug, and offer ‘starter packs’ for the doctor to try.

10.
Present a ‘cost-effectiveness’ analysis which shows that your product, even though more expensive than its competitor, ‘actually works out cheaper’ (see section ‘The great guidelines debate’).

Before you agree to meet a rep (or a patient armed with material from a newspaper article or DTCA website), remind yourself of some basic rules of research design. As sections ‘Cohort studies’ and ‘Cross-sectional surveys’ argued, questions about the benefits of therapy should ideally be addressed with randomised controlled trials. But preliminary questions about pharmacokinetics (i.e. how the drug behaves while it is getting to its site of action), particularly those relating to bioavailability, require a straight dosing experiment in healthy (and, if ethical and practicable, sick) volunteers.

Common (and probably trivial) adverse drug reactions may be picked up, and their incidence quantified, in the randomised controlled trials undertaken to demonstrate the drug's efficacy. But rare (and usually more serious) adverse drug reactions require both pharmacovigilance surveys (collection of data prospectively on patients receiving a newly licensed drug) and case–control studies (see section ‘Cohort studies’) to establish association. Ideally, individual rechallenge experiments (where the patient who has had a reaction considered to be caused by the drug is given the drug again in carefully supervised circumstances) should be performed to establish causation [8].

Pharmaceutical reps do not tell nearly as many lies as they used to (drug marketing has become an altogether more sophisticated science), but as Goldacre [9] has shown in his book ‘Bad Pharma’, they still provide information that is at best selective and at worst overtly biased. It often helps their case, for example, to present the results of uncontrolled trials and express them in terms of before-and-after differences in a particular outcome measure. Reference to section ‘Cross-sectional surveys’ and the literature on placebo effects [10] [11] should remind you why uncontrolled before-and-after studies are the stuff of teenage magazines, not hard science.

Dr Herxheimer, who edited the
Drug and Therapeutics Bulletin
for many years, undertook a survey of ‘references’ cited in advertisements for pharmaceutical products in the leading UK medical journals. He tells me that a high proportion of such references cite ‘data on file’, and many more refer to publications written, edited and published entirely by the industry. Evidence from these sources has sometimes (although by no means invariably) been shown to be of lower scientific quality than that which appears in independent, peer-reviewed journals. And let's face it, if you worked for a drug company that had made a major scientific breakthrough you would probably submit your findings to a publication such as the
Lancet
or the
New England Journal of Medicine
before publishing them in-house. In other words, you don't need to ‘trash’ papers about drug trials
because
of where they have been published, but you do need to look closely at the methods and statistical analysis of such trials.

Making decisions about therapy

Sackett and colleagues [8], in their book ‘
Clinical epidemiology—a basic science for clinical medicine
’, argue that before starting a patient on a drug, the doctor should:

a.
identify
for this patient
the ultimate objective of treatment (cure, prevention of recurrence, limitation of functional disability, prevention of later complications, reassurance, palliation, symptomatic relief, etc.);

b.
select the
most appropriate
treatment using all available evidence (this includes addressing the question of whether the patient needs to take any drug at all);

c.
specify the
treatment target
(how will you know when to stop treatment, change its intensity or switch to some other treatment?).

For example, in the treatment of high blood pressure, the doctor might decide that:

a.
the
ultimate objective of treatment
is to prevent (further) target organ damage to brain, eye, heart, kidney, and so on (and thereby prevent death);

b.
the
choice of specific treatment
is between the various classes of antihypertensive drugs selected on the basis of randomised, placebo-controlled and comparative trials—as well as between non-drug treatments such as salt restriction; and

c.
the
treatment target
might be a Phase V diastolic blood pressure (right arm, sitting) of less than 90 mmHg, or as close to that as tolerable in the face of drug side effects.

If these three steps are not followed (as is often the case—e.g. in terminal care), therapeutic chaos can result. In a veiled slight on surrogate endpoints, Sackett and his team remind us that the choice of specific therapy should be determined by evidence of what
does
work, and not on what
seems
to work or
ought
to work. ‘Today’s therapy', they warn, ‘when derived from biologic facts or uncontrolled clinical experience, may become tomorrow’s bad joke' [8].

Surrogate endpoints

I have not included this section solely because it is a particular hobby horse of mine. If you are a practising (and non-academic) clinician, your main contact with published papers may well be through what gets fed to you by a ‘drug rep’. The pharmaceutical industry is a slick player at the surrogate endpoint game, and I make no apology for labouring the point that such outcome measures must be evaluated very carefully.

I will define a surrogate endpoint as ‘
a variable which is relatively easily measured and which predicts a rare or distant outcome of either a toxic stimulus (e.g. pollutant) or a therapeutic intervention (e.g. drug, surgical procedure, piece of advice), but which is not itself a direct measure of either harm or clinical benefit
’. The growing interest in surrogate endpoints in medical research reflects two important features of their use.

• They can considerably reduce the
  sample size
  ,
  duration
  and, therefore,
  cost
  , of clinical trials.
  
• They can allow treatments to be assessed in situations where the use of primary outcomes would be excessively
  invasive
  or
  unethical
  .
  

In the evaluation of pharmaceutical products, commonly used surrogate endpoints include

• pharmacokinetic measurements (e.g. concentration–time curves of a drug or its active metabolite in the bloodstream);
  
• in vitro
  (i.e. laboratory) measures such as the mean inhibitory concentration (MIC) of an antimicrobial against a bacterial culture on agar;
  
• macroscopic appearance of tissues (e.g. gastric erosion seen at endoscopy);
  
• change in levels of (alleged) ‘biological markers of disease’ (e.g. microalbuminuria in the measurement of diabetic kidney disease);
  
• radiological appearance (e.g. shadowing on a chest X-ray—or in a more contemporary setting, functional magnetic resonance imaging).
  

Surrogate endpoints have a number of drawbacks. First, a change in the surrogate endpoint does not itself answer the essential preliminary questions: ‘what is the objective of treatment in this patient?’ and ‘what, according to valid and reliable research studies, is the best available treatment for this condition?’. Second, the surrogate endpoint may not closely reflect the treatment target—in other words, it may not be valid or reliable. Third, the use of a surrogate endpoint has the same limitations as the use of any other
single
measure of the success or failure of therapy—it ignores all the other measures! Over-reliance on a single surrogate endpoint as a measure of therapeutic success usually reflects a narrow or naïve clinical perspective.

Finally, surrogate endpoints are often developed in animal models of disease because changes in a specific variable can be measured under controlled conditions in a well-defined population. However, extrapolation of these findings to human disease is liable to be invalid [12].

• In animal studies, the population being studied has fairly uniform biological characteristics and may be genetically inbred.
  
• Both the tissue and the disease being studied may vary in important characteristics (e.g. susceptibility to the pathogen, rate of cell replication) from the parallel condition in human subjects.
  
• The animals are kept in a controlled environment, which minimises the influence of lifestyle variables (e.g. diet, exercise, stress) and concomitant medication.
  
• Giving high doses of chemicals to experimental animals may distort the usual metabolic pathways and thereby give misleading results. Animal species best suited to serve as a surrogate for humans vary for different chemicals.
  

The ideal features of a surrogate endpoint are shown in Box 6.2. If the ‘rep’ who is trying to persuade you about the value of the drug cannot justify the endpoints used, you should challenge him or her to produce additional evidence.

Box 6.2 Ideal features of a surrogate endpoint

1.
The surrogate endpoint should be reliable, reproducible, clinically available, easily quantifiable, affordable and exhibit a ‘dose–response’ effect (i.e. the higher the level of the surrogate endpoint, the greater the probability of disease).

2.
It should be a true predictor of disease (or risk of disease) and not merely express exposure to a covariable. The relationship between the surrogate endpoint and the disease should have a biologically plausible explanation.

3.
It should be sensitive—that is, a ‘positive’ result in the surrogate endpoint should pick up all or most patients at increased risk of adverse outcome.

4.
It should be specific—that is, a ‘negative’ result should exclude all or most of those without increased risk of adverse outcome.

5.
There should be a precise cut-off between normal and abnormal values.

6.
It should have an acceptable positive predictive value—that is, a ‘positive’ result should always or usually mean that the patient thus identified is at increased risk of adverse outcome (see section ‘Ten questions to ask about a paper describing a complex intervention’).

7.
It should have an acceptable negative predictive value—that is, a ‘negative’ result should always or usually mean that the patient thus identified is not at increased risk of adverse outcome (see section ‘Ten questions to ask about a paper describing a complex intervention’).

8.
It should be amenable to quality control monitoring.

9.
Changes in the surrogate endpoint should rapidly and accurately reflect the response to therapy—in particular, levels should normalise in states of remission or cure.

If you are interested in pursuing some real examples of surrogate endpoints that led to misleading practices and recommendations, try these.

• The use of ECG findings instead of clinical outcomes (syncope, death) in deciding the efficacy and safety of anti-arrhythmia drugs [13];
  
• The use of X-ray findings instead of clinical outcomes (pain, loss of function) to monitor the progression of osteoarthritis and the efficacy of disease-modifying drugs [14];
  
• The use of albuminuria instead of the overall clinical benefit–harm balance to evaluate the usefulness of dual renin–angiotensin blockade in hypertension [15] [16]. In this example, the intervention was based on a hypothetical argument that blocking the renin–angiotensin pathway at two separate stages would be doubly effective, and the surrogate marker confirmed that this seemed to be the case—but the combination was also doubly effective at producing the potentially fatal side effect of hypokalaemia!
  

It would be unsporting to suggest that the pharmaceutical industry always develops surrogate endpoints with the deliberate intention of misleading the licensing authorities and health professionals. Surrogate endpoints, as I argued in section ‘“Evidence” and marketing’, have both ethical and economic imperatives. However, the industry does have a vested interest in overstating its case on the strength of these endpoints [9], so use caution when you read a paper whose findings are not based on ‘hard patient-relevant outcomes’.

Surrogate endpoints are only one of many ways in which industry-sponsored trials may give a misleading impression of the efficacy of a drug. Other subtle (and not so subtle) influences on research design—such as framing the question in a particular way or selective reporting of findings—have been described in a recent Cochrane review of how industry-sponsored trials tend to favour industry products [17].

What information to expect in a paper describing a randomised controlled trial: the CONSORT statement