| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
COMMENTARY |
Evan Mayo-Wilson is with the Centre for Evidence-Based Intervention, Department of Social Policy and Social Work, University of Oxford, Oxford, England.
Correspondence: Requests for reprints should be sent to Evan Mayo-Wilson, Centre for Evidence-Based Intervention, Department of Social Policy and Social Work, Barnett House, 32 Wellington Square, Oxford OX1 2ER, UK (e-mail: Evan.Mayo-Wilson{at}socres.ox.ac.uk).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONIMPLEMENTATION DATAEXISTING GUIDELINESPROPOSALSCONCLUSIONReferences |
|---|
Randomized controlled trials of public health interventions are often complex: practitioners may not deliver interventions as researchers intended, participants may not initiate interventions and may not behave as expected, and interventions and their effects may vary with environmental and social context.
Reports of randomized controlled trials can be misleading when they omit information about the implementation of interventions, yet such data are frequently absent in trial reports, even in journals that endorse current reporting guidelines.
Particularly for complex interventions, the Consolidated Standards of Reporting Trials (CONSORT) statement does not include all types of information needed to understand the results of randomized controlled trials. CONSORT should be expanded to include more information about the implementation of interventions in all trial arms.
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONIMPLEMENTATION DATAEXISTING GUIDELINESPROPOSALSCONCLUSIONReferences |
|---|
Trial reporting has improved since the introduction of guidelines that emphasize transparent reporting of methods and results1; however, evidence demonstrates that trial reports continue to lack information about the implementation of interventions—their actual delivery by practitioners and uptake by participants.
Implementation data increase the external validity of trials and aid the application of results by practitioners.2,3 Policymakers, administrators, and researchers need these data to assess the generalizability of findings, to synthesize literature,4 to design future trials, to determine the feasibility of interventions,5 and to develop treatment guidelines.6 The importance of implementation data is emphasized in the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) statement,7 a guide for reporting nonrandomized controlled trials that complements the Consolidated Standards of Reporting Trials (CONSORT) statement,8 a guide for reporting randomized controlled trials. Implementation data are needed to understand the results and implications of both randomized and nonrandomized trials, but unlike TREND, CONSORT gives little attention to practitioner actions and participant experiences.
On the basis of previous research findings, I propose that CONSORT be expanded to encourage the inclusion of implementation data in reports of randomized controlled trials.
|
IMPLEMENTATION DATA |
|---|
|
TOPABSTRACTINTRODUCTIONIMPLEMENTATION DATAEXISTING GUIDELINESPROPOSALSCONCLUSIONReferences |
|---|
One review of 539 studies published in the Journal of Applied Behavior Analysis between 1968 and 1980 found that among the surveyed studies presenting operational definitions, only an average of 16% (range 3%–34%) also performed some check on the accuracy of the implementation of the independent variable.11 A similar review of school-based studies found that 64 of 181 (35%) operationally defined the intervention and 45 (25%) monitored or measured its implementation.12 In a review of studies involving people with learning disabilities, 12 of 65 (18%) measured implementation of the independent variable.13 A review of 148 studies on parent training research published in 18 journals between 1975 and 1990 found that almost all reports failed to examine differences between program design and implementation.14
In a broader review, fewer than 6% of 359 psychosocial trials included a treatment manual, implementer supervision, and an adherence check; 55% did not report using any of these methods to promote and verify implementation.15 An analysis of 162 prevention studies found that 39 (24%) reported a method for verifying intervention delivery,16 and reviews of the 1990 editions of Behavior Therapy and the Journal of Consulting and Clinical Psychology found that 9 of 25 (36%) and 7 of 22 (32%) articles, respectively, assessed treatment delivery directly.17
In 2005, the National Institutes of Health Behavior Change Consortium published one of the most comprehensive analyses of implementation data from 342 health behavior intervention studies; 71% of studies reported theoretical models, whereas only 27% reported mechanisms to monitor adherence.18
Recently, systematic reviews have been used to highlight the omission of implementation information in trial reports.19–21 For example, a review about smoking cessation concluded that studies should describe "the intervention in sufficient detail for its replication even if the detail requires a separate paper."22(p10) A review of interventions to promote smoke detectors identified what later authors labeled "systematic deficiencies in the literature in reporting context, methods, and details of implementation."23(p150)
Recurrent omissions of implementation data may prevent readers from acting on the results of trials. Worse, results can be misleading when implementation data are not considered. A review of tap water for wound cleansing concluded that tap water might be as effective as sterile water or sterile saline water for preventing infection and promoting healing24; however, most trials took place in settings with sanitary tap water. The results applied only to similar settings.25
|
EXISTING GUIDELINES |
|---|
|
TOPABSTRACTINTRODUCTIONIMPLEMENTATION DATAEXISTING GUIDELINESPROPOSALSCONCLUSIONReferences |
|---|
The Transparent Reporting of Evaluations with Nonrandomized Designs statement complements CONSORT and strongly emphasizes the importance of implementation data2: "Sufficient detail and clarity in the report allow readers to understand the conduct and findings of the intervention study and how the study was different from or similar to other studies in the field."8(p361) The same logic surely applies to the reporting of randomized trials.
Implementation data may not be collected for practical and scientific reasons (e.g., monitoring adherence might confound a trial); however, information about implementation is generally undervalued. Researchers may exclude implementation information because it does not seem important or to give positive impressions of interventions that encounter problems in delivery or compliance. Journal editors may not demand implementation data because of space restrictions. Furthermore, funding bodies neglect mixed-methods research about putting interventions into practice.29 Expanding CONSORT would signal the importance of implementation information, expose its frequent omission, and encourage its measurement and reporting.
Guidelines in Action
A review of a 2006 series of reports of the Women’s Health Initiative Randomized Controlled Dietary Modification Trial30–32 shows that reports of well-conducted trials in journals endorsing CONSORT could be improved by including data about the implementation of interventions.
The Women’s Health Initiative trial tracked nearly 49 000 women for more than 8 years to investigate the impact of "18 group sessions in the first year and quarterly maintenance sessions thereafter"30(p631) on cardiovascular disease, breast cancer, and colorectal cancer. Although the trial tested a behavioral intervention, reports implied that the study was designed to "directly address the health effects of a low-fat eating pattern"31(p644); an early paper said "the intervention is a dietary pattern."33(pS95) The trial allowed substantial variation across sites,34 and some aspects of delivery were monitored,35 but the reports neither included nor referenced information about the actual delivery of the intervention by program staff.
The titles of these reports ("Low-fat dietary pattern and. . .") and popular media accounts of them(e.g., 36) unintentionally confused diet (what people actually eat) and a diet (a behavioral modification program). In the first year of the study, 68.6% of women in the intervention group did not reduce their fat consumption to the target level (20% of total energy intake); in the sixth year, 85.6% exceeded their target, reducing their fat consumption to less than 20% of their total energy intake.31 One abstract reported that "women in the comparison group continued their usual eating pattern,"31(p643) but in the first year of the study, women in the control group reduced both their energy and fat intakes.30 Furthermore, self-reported food intake was inconsistent with changes in weight for both groups.37
Information about participants33 and recruitment38 has been published elsewhere, but participant uptake (e.g., attendance at sessions) was mentioned only in a definition of dropout and as a statistical variable in an analysis comparing women who attended specified numbers of sessions. Reports considered the impact of compliance on statistical power, but reports did not consider why the intervention failed to produce expected behavioral changes.
Although reports suggest that the Women’s Health Initiative trial was well designed and internally valid, more implementation data would increase the utility of its results. Implementation data would help readers understand the trial and help health professionals design and improve other dietary interventions.
|
PROPOSALS |
|---|
|
TOPABSTRACTINTRODUCTIONIMPLEMENTATION DATAEXISTING GUIDELINESPROPOSALSCONCLUSIONReferences |
|---|
In reports of randomized trials, authors often report what they intended rather than what actually happened. Information about intentional and unintentional deviations in the delivery of interventions by practitioners, as well as information about deviations because of external factors, helps readers understand and apply the results of randomized trials.
The Transparent Reporting of Evaluations with Nonrandomized Designs statement includes incentives used as part of the intervention (items 4 and 21) and asks for "discussion of the success of and barriers to implementing the intervention"8(p365) (item 20). This information is similarly important in reports of randomized trials.
Adding such items to CONSORT would encourage authors to include, as far as possible, information needed to replicate trials as they happened, including delivery of nonspecific treatment components and receipt of interventions outside trial protocols.
|
CONCLUSION |
|---|
|
TOPABSTRACTINTRODUCTIONIMPLEMENTATION DATAEXISTING GUIDELINESPROPOSALSCONCLUSIONReferences |
|---|
Critics of evidence-based practice are right to argue that many researchers value statistical outcomes at the expense of other types of data; limited reporting of qualitative and descriptive data prevents researchers, practitioners, and policymakers from using the application of the results of randomized controlled trials appropriately.
CONSORT has helped improve the quality of trial reports, yet it remains true that "in many clinical trials we must still guess what treatment was actually tested."17(p1) Implementation data are essential to users of trial reports. Although it may be impossible to include complete data in all printed reports, implementation data could be included in extended online journals, in referenced papers or Web sites, or in trial registries. CONSORT should ask researchers to include or reference implementation data in reports of randomized controlled trials or to justify their absence.
|
Acknowledgments |
|---|
|
Footnotes |
|---|
Accepted for publication October 2, 2006.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONIMPLEMENTATION DATAEXISTING GUIDELINESPROPOSALSCONCLUSIONReferences |
|---|
2. Bellg A, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol. 2004;23:443–451.[CrossRef][Web of Science][Medline]
3. Resnick B, Bellg A, Borrelli B, et al. Examples of implementation and evaluation of treatment fidelity in the BCC studies: where we are and where we need to go. Ann Behav Med. 2005; 29(Suppl 2):46–54.[CrossRef][Web of Science][Medline]
4. Herbert RD, Bo K. Analysis of quality of interventions in systematic reviews. BMJ. 2005;331:507–509.
5. Dusenbury L, Brannigan R, Falco M, Hansen W. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18: 237–256.
6. Jackson N, Waters E, for the Guidelines for Systematic Reviews of Health Promotion and Public Health Interventions Taskforce. Guidelines for Systematic Reviews of Health Promotion and Public Health Interventions. Version 1.2. Victoria, Australia: Deakin University; 2005. Available at: http://www.vichealth.vic.gov.au/cochrane/activities/Guidelines%20for%20HPPH%20reviews.pdf. Accessed January 14, 2007.
7. Moher D, Schulz KF, Altman DG, for the CONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Ann Intern Med. 2001;134: 657–662.
8. Des Jarlais D, Lyles C, Crepaz N, for the TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health. 2004;94: 361–366.
9. Billingsley F, White O, Munson R. Procedural reliability: a rationale and an example. Behav Assess. 1980;2: 229–241.[Web of Science]
10. Bond G, Evans L, Salyers M, Williams J, Kim HW. Measurement of fidelity in psychiatric rehabilitation. Ment Health Serv Res. 2000;2:75–87.[CrossRef][Medline]
11. Peterson L, Homer A, Wonderlich S. The integrity of independent variables in behavior analysis. J Appl Behav Anal. 1982;15:477–492.[CrossRef][Web of Science][Medline]
12. Gresham F, Gansle K, Noell G, Cohen S, Rosenblum S. Treatment integrity of school-based behavioral intervention studies: 1980–1990. School Psychol Rev. 1993;22:254–272.
13. Gresham F, MacMillan D, Beebe-Frankenberger M, Bocian K. Treatment integrity in learning disabilities intervention research: do we really know how treatments are implemented? Learn Disabil Res Pract. 2000;15:198–205.[CrossRef]
14. Rogers-Wiese MR. A critical review of parent training research. Psychol Sch. 1992;29:229–236.[CrossRef][Web of Science]
15. Moncher F, Prinz R. Treatment fidelity in outcome studies. Clin Psychol Rev. 1991;11:247–266.[CrossRef][Web of Science]
16. Dane A, Schneider B. Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clin Psychol Rev. 1998; 18:23–45.[CrossRef][Web of Science][Medline]
17. Lichstein K, Riedel B, Grieve R. Fair tests of clinical trials: a treatment implementation model. Adv Behav Res Ther. 1994;16:1–29.[CrossRef]
18. Borelli B, Sepinwall D, Ernst D, et al. A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. J Consult Clin Psychol. 2005;73:852–860.[CrossRef][Web of Science][Medline]
19. Jackson N, Waters E. Criteria for the systematic review of health promotion and public health interventions. Health Promot Int. 2005;20:367–374.
20. Montgomery P, Gardner F, Operario D, Mayo-Wilson E, Tamayo S, Underhill K. The Oxford Implementation Reporting Index: the development of an indicator of treatment fidelity in systematic reviews of psychosocial interventions. Paper presented at: XIII Cochrane Colloquium; October 22–26, 2005; Melbourne, Australia.
21. Underhill K, Operario D, Montgomery P. Reporting deficiencies in trials of abstinence-only programmes for HIV prevention. AIDS. 2007;21(2): 266–268.[Web of Science][Medline]
22. Lumley J, Oliver S, Chamberlain C, Oakley L. Interventions for promoting smoking cessation during pregnancy. Cochrane Database Syst Rev. 2004;(4): CD001055.
23. Arai L, Roen K, Roberts H, Popay J. It might work in Oklahoma but will it work in Oakhampton? context and implementation in the effectiveness literature on domestic smoke detectors. Inj Prev. 2005;11:148–151.
24. Fernandez R, Griffiths R, Ussia C. Water for wound cleansing. Cochrane Database Syst Rev. 2002;(4): CD003861.
25. Tharyan P. Doing the right reviews: how do we ensure our reviews bring about answers to important questions? Paper presented at: XIII Cochrane Colloquium; October 24, 2005; Melbourne, Australia.
26. Mills E, Wu P, Gagnier J, Devereaux P. The quality of randomized trial reporting in leading medical journals since the revised CONSORT statement. Contemp Clin Trials. 2005; 26:480–487.[CrossRef][Web of Science][Medline]
27. Egger M, Jüni P, Bartlett C, for the CONSORT Group. Value of flow diagrams in reports of randomized controlled trials. JAMA. 2001;285: 1996–1999.
28. Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134:663–694.
29. Sanders D, Haines A. Implementation research is needed to achieve international health goals. PLoS Med. 2006;3(6):e186.[CrossRef][Medline]
30. Prentice R, Caan B, Chlebowski R, et al. Low-fat dietary pattern and risk of invasive breast cancer: the Women’s Health Initiative Randomized Controlled Dietary Modification Trial. JAMA. 2006;295:629–642.
31. Beresford S, Johnson K, Ritenbaugh C, et al. Low-fat dietary pattern and risk of colorectal cancer: the Women’s Health Initiative Randomized Controlled Dietary Modification Trial. JAMA. 2006;295:643–654.
32. Howard B, Van Horn L, Hsia J, et al. Low-fat dietary pattern and risk of cardiovascular disease: the Women’s Health Initiative Randomized Controlled Dietary Modification Trial. JAMA. 2006;295:655–666.
33. Ritenbaugh C, Patterson R, Chlebowski R, et al. The Women’s Health Initiative Dietary Modification trial: overview and baseline characteristics of participants. Ann Epidemiol. 2003;13(9):S87–S97.[CrossRef][Web of Science][Medline]
34. Tinker L, Burrows E, Henry H, Patterson R, Rupp J, Van Horn L. The Women’s Health Initiative: overview of the nutrition components. In: Krummel D, Etherton P, eds. Nutrition in Women’s Health. Gaithersburg, Md: Aspen Publishers Inc; 1996:510–542.
35. Anderson GL, Manson J, Wallace R, et al. Implementation of the Women’s Health Initiative study design. Ann Epidemiol. 2003;13(9):S5–S17.[CrossRef][Web of Science][Medline]
36. Kolata G. Low-fat diet does not cut health risks, study finds. New York Times. February 16, 2006; A1.
37. Greene PJ. Low-fat diet and weight change in postmenopausal women. JAMA. 2006;296:394.
38. Hays J, Hunt J, Hubbell F, et al. The Women’s Health Initiative recruitment methods and results. Ann Epidemiol. 2003;13(9):S18–S77.[CrossRef][Web of Science][Medline]
This article has been cited by other articles:
|
|
 |
|
  S. L. Robb and J. S. Carpenter A Review of Music-based Intervention Reporting in Pediatrics J Health Psychol, May 1, 2009; 14(4): 490 - 501. [Abstract] [PDF]
|
|
|
|
 |
|
 P. Glasziou, E. Meats, C. Heneghan, and S. Shepperd What is missing from descriptions of treatment in trials and reviews? BMJ, June 28, 2008; 336(7659): 1472 - 1474. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
