Read How to Read a Paper: The Basics of Evidence-Based Medicine Online
Authors: Trisha Greenhalgh
Drug trials are an example of a ‘simple intervention’—that is, an intervention that is well demarcated (i.e. it's easy to say what the intervention comprises) and lends itself to an ‘intervention on’ versus ‘intervention off’ research design. In Chapters 3 and 4, I gave some preliminary advice on assessing the methodological quality of research studies. Here's some more detail. In 1996, an international working group produced a standard checklist, known as
Consolidated Standards of Reporting Trials
(
CONSORT
), for reporting randomised controlled trials in medical journals, and this has now been updated several times, the latest in 2010 [18]. Without doubt, the use of such checklists has increased the quality and consistency of reporting of trials in the medical literature [19]. A checklist based on the CONSORT statement is reproduced in
Table 6.1
. Please do not try to learn this table off by heart (I certainly couldn't reproduce it myself from memory), but do refer to it if you are asked to critically appraise a paper to which it applies—or if you are planning on doing a randomised trial yourself.
Table 6.1
Checklist for a randomised controlled trial based on the CONSORT statement (see Reference [17])
| Title/abstract | Do the title and abstract say how participants were allocated to interventions (e.g., ‘random allocation’, ‘randomised’ or ‘randomly assigned’)? |
| Introduction | Is the scientific background and rationale for the study adequately explained? |
| Methods | |
| Objectives | Were the specific objectives and/or hypothesis to be tested stated explicitly? |
| Participants and setting | Does the paper state the eligibility criteria for participants and the settings and locations where the data were collected? |
| Interventions | Does the paper give precise details of the intervention(s) and the control intervention(s) and how and when they were administered? |
| Outcomes | Have the primary and secondary outcome measures been clearly defined? When applicable, have the methods used to enhance the quality of measurements (e.g. multiple observations, training of assessors) been set out? |
| Sample size | How was sample size determined? When applicable, were any interim analyses and/or rules for stopping the study early explained and justified? |
| Blinding (masking) | Does the paper state whether or not participants, those administering the interventions and those assessing the outcomes were blinded to group assignment? How was the success of blinding assessed? |
| Statistical methods | Were the statistical methods used to compare groups for primary and secondary outcome(s) and any subgroup analyses, appropriate? |
| Details of randomisation | |
| Sequence generation | Was the method used to generate the random allocation sequence, including details of any restrictions (e.g., blocking, stratification), clearly described? |
| Allocation concealment | Was the method used to implement the random allocation sequence (e.g., numbered containers or central telephone), stated, and was it made clear whether the sequence was concealed until interventions were assigned? |
| Implementation | Does the paper say who generated the allocation sequence, who enrolled participants and who assigned participants to their groups? |
| Results | |
| Flow diagram | Is a clear diagram included showing the flow of participants through the trial? This should report, for each group, the numbers of participants randomly assigned, receiving intended treatment, completing the study protocol and analysed for the primary outcome. |
| Protocol deviations | Are all deviations from the original study protocol explained and justified? |
| Recruitment dates | Have the authors given the date range during which participants were recruited to the study? |
| Baseline data | Are the baseline demographic and clinical characteristics of each group described? |
| Numbers analysed | Is the number of participants (denominator) in each group included in each analysis, and is the analysis by ‘intention-to-treat’? |
| Outcomes and estimation | For each primary and secondary outcome, is there a summary of results for each group, and the estimated effect size and its precision (e.g., 95% confidence interval)? |
| Ancillary analyses | Are all additional analyses described and justified, including subgroup analyses, both pre-specified and exploratory? |
| Adverse events | Have the authors reported and discussed all important adverse events? |
| Discussion | |
| Interpretation | Is the interpretation of the results justified, taking into account study hypotheses, sources of potential bias or imprecision and the dangers of multiple comparisons? |
| Generalisability | Have the authors made defensible estimate of the generalisability (external validity) of the trial findings? |
Incidentally, one important way to reduce bias in drug marketing is to ensure that every trial that is
begun
is also written up and
published
[20]. Otherwise, the drug industry (or anyone else with a vested interest) could withhold publication of any trial that did not support their own belief in the efficacy and/or cost-effectiveness of a particular product. Goldacre [9] covers the topic of compulsory trial registration at inception (and the reluctance of some drug companies to comply with it) in his book.
Getting worthwhile evidence out of a pharmaceutical representative
Any doctor who has ever given an audience to a ‘rep’ who is selling a non-steroidal anti-inflammatory drug will recognise the gastric erosion example. The question to ask him or her is not ‘what is the incidence of gastric erosion on your drug?’, but ‘what is the incidence of potentially life-threatening gastric bleeding?’. Other questions to ask ‘drug reps’, based on an early article in the
Drug and Therapeutics Bulletin
[21], are listed here. For more sophisticated advice on how to debunk sponsored clinical trial reports that attempt to blind you with statistics, see Montori and colleagues' helpful Users' Guide [22] and (more tangentially but worth noting) Goldacre's blockbuster on the corporate tricks of ‘big pharma’ [9].
1.
See representatives only by appointment. Choose to see only those whose product interests you and confine the interview to that product.
2.
Take charge of the interview. Do not hear out a rehearsed sales routine but ask directly for the information.
3.
Request independent published evidence from reputable peer-reviewed journals.
4.
Do not look at promotional brochures, which often contain unpublished material, misleading graphs and selective quotations.
5.
Ignore anecdotal ‘evidence’ such as the fact that a medical celebrity is prescribing the product.
6.
Using the ‘STEP’ acronym, ask for evidence in four specific areas:
7.
Evaluate the evidence stringently, paying particular attention to the power (sample size) and methodological quality of clinical trials and the use of surrogate endpoints. Apply the CONSORT checklist (
Table 6.1
). Do not accept theoretical arguments in the drug's favour (e.g. ‘longer half-life’) without direct evidence that this translates into clinical benefit.
8.
Do not accept the newness of a product as an argument for changing to it. Indeed, there are good scientific arguments for doing the opposite.
9.
Decline to try the product via starter packs or by participating in small-scale, uncontrolled ‘research’ studies.
10.
Record in writing the content of the interview and return to these notes if the rep requests another audience.
References
1
Godlee F. Doctors and the drug industry. BMJ 2008; 336 doi:
http://dx.doi.org/10.1136/bmj.39444.472708.47
.
2
Hollon MF. Direct-to-consumer advertising.
JAMA: The Journal of the American Medical Association
2005;
293
(16):2030–3.
3
Liang BA, Mackey T. Direct-to-consumer advertising with interactive internet media global regulation and public health issues.
JAMA: The Journal of the American Medical Association
2011;
305
(8):824–5.
4
Kaphingst KA, Dejong W, Rudd RE, et al.
A content analysis of direct-to-consumer television prescription drug advertisements. Journal of Health Communication: International Perspectives
. 2004;
9
(6):515–528.
5
Brody H. The company we keep: why physicians should refuse to see pharmaceutical representatives.
The Annals of Family Medicine
2005;
3
(1):82–5.
6
O'brien M, Rogers S, Jamtvedt G, et al. Educational outreach visits: effects on professional practice and health care outcomes.
Cochrane Database of Systematic Reviews (Online)
2007;
4
(4):1–6.
7
Kravitz RL, Epstein RM, Feldman MD, et al. Influence of patients' requests for direct-to-consumer advertised antidepressants.
JAMA: The Journal of the American Medical Association
2005;
293
(16):1995–2002.
8
Sackett DL, Haynes RB, Tugwell P.
Clinical epidemiology: a basic science for clinical medicine
. Boston, USA: Little, Brown and Company, 1985.
9
Goldacre B.
Bad pharma: how drug companies mislead doctors and harm patients
. London, Fourth Estate: Random House Digital Inc., 2013.
10
Rajagopal S. The placebo effect.
Psychiatric Bulletin
2006;
30
(5):185–8.
11
Price DD, Finniss DG, Benedetti F. A comprehensive review of the placebo effect: recent advances and current thought.
Annual Review of Psychology
2008;
59
:565–90.
12
Gøtzsche PC, Liberati A, Torri V, et al. Beware of surrogate outcome measures.
International Journal of Technology Assessment in Health Care
1996;
12
(02):238–46.
13
Connolly SJ. Use and misuse of surrogate outcomes in arrhythmia trials.
Circulation
2006;
113
(6):764–6.
14
Guermazi A, Hayashi D, Roemer FW, et al. Osteoarthritis: a review of strengths and weaknesses of different imaging options.
Rheumatic Diseases Clinics of North America
2013;
39
(3):567–91.
15
Messerli FH, Staessen JA, Zannad F. Of fads, fashion, surrogate endpoints and dual RAS blockade.
European Heart Journal
2010;
31
(18):2205–8.
16
Harel Z, Gilbert C, Wald R, et al. The effect of combination treatment with aliskiren and blockers of the renin–angiotensin system on hyperkalaemia and acute kidney injury: systematic review and meta-analysis.
BMJ: British Medical Journal
2012;
344
:e42.