|
|
||||||||
COMMENTARY |
Don C. Des Jarlais is with the Baron Edmond de Rothschild Chemical Dependency Institute, Beth Israel Medical Center, New York City. Cynthia Lyles and Nicole Crepaz are with the Centers for Disease Control and Prevention, Atlanta, Ga. TREND Group members are listed below in the Acknowledgments.
Correspondence: Requests for reprints should be sent to Don C. Des Jarlais, PhD, Beth Israel Medical Center, Chemical Dependency Institute, First Ave at 16 St, New York, NY 10003 (e-mail: dcdesjarla{at}aol.com).
| ABSTRACT |
|---|
|
|
|---|
Developing an evidence base for making public health decisions will require using data from evaluation studies with randomized and nonrandomized designs. Assessing individual studies and using studies in quantitative research syntheses require transparent reporting of the study, with sufficient detail and clarity to readily see differences and similarities among studies in the same area. The Consolidated Standards of Reporting Trials (CONSORT) statement provides guidelines for transparent reporting of randomized clinical trials.
We present the initial version of the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) statement. These guidelines emphasize the reporting of theories used and descriptions of intervention and comparison conditions, research design, and methods of adjusting for possible biases in evaluation studies that use nonrandomized designs.
OVER THE PAST SEVERAL decades, a strong movement toward evidence-based medicine has emerged.13 In the context of evidence-based medicine, clinical decisions are based on the best available scientific data rather than on customary practices or the personal beliefs of the health care provider. There is now a parallel movement toward evidence-based public health practices.4,5 The movement is intended to utilize the best available scientific knowledge as the foundation for public healthrelated decisionmaking.
In the context of evidence-based medicine, the randomized controlled trial (RCT) is usually considered of greatest evidentiary value for assessing the efficacy of interventions. Indeed, the preference for this design is sufficiently strong that when empirical evidence from RCTs is available, "weaker" designs are often considered to be of little or no evidentiary value. In this issue, Victora et al.6 make a strong argument that evidence-based public health will necessarily involve the use of research designs other than RCTs. Most important, they argue that RCTs are often not practical or not ethical for evaluating many public health interventions and discuss methods for drawing causal inferences from nonrandomized evaluation designs ("plausibility" and "adequacy" designs in their terminology).
Also in this issue, Donner and Klar,7 Murray et al.,8 and Varnell et al.9 provide overviews of the benefits and pitfalls of the group-randomized trial, which, in some situations, may be a reasonable alternative to the RCT. There are also a wide variety of nonrandomized evaluation designs that can contribute important data on the efficacy or effectiveness of interventions, such as quasi-experimental designs,10 nonrandomized trials, and natural experiments. Including these types of designs in developing evidence-based recommendations can provide a more integrated picture of the existing evidence and could help to strengthen public health practice. Excluding data collected under such designs would undoubtedly bias the evidence base toward interventions that are "easier" to evaluate but not necessarily more effective or cost-effective.
If nonrandomized designs are to be systematically used in building evidence-based public health practices, it will be necessary to improve the reporting quality of these types of studies. The transparency, or clarity, in the reporting of individual studies is key. Sufficient detail and clarity in the report allow readers to understand the conduct and findings of the intervention study and how the study was different from or similar to other studies in the field.
Furthermore, evidence-based practice may often rely on meta-analyses of large numbers of studies, some of which may report negative results. Metaanalysis requires full reporting of methods and outcomes to enable assessment of comparability of different studies. Inadequate, or nontransparent, reporting may make it difficult to understand the variables that affect intervention outcomes and the central elements in intervention success or failure over multiple studies.
In recent years, efforts have been made to improve the quality of reporting of RCTs. The Consolidated Standards of Reporting Trials (CONSORT) statement11 provides a 22-item checklist and subject flow chart for the transparent reporting of RCTs. This statement has been adopted as a framework for the reporting of RCTs by a large number of medical, clinical, and psychological journals (153, according to http://www.consort-statement.org, as of September 16, 2003). Use of the CONSORT statement has improved the quality of RCT reports over the past several years.12 There is yet, however, no agreed-upon framework for the transparent reporting of nonrandomized research evaluations.
The HIV/AIDS Prevention Research Synthesis (PRS) team of the Centers for Disease Control and Prevention (CDC) has been synthesizing evidence from HIV behavioral intervention studies involving RCT and nonrandomized designs. The PRS team found that many study reports failed to include critical information (e.g., intervention timing and dosage, effect size data) necessary for research syntheses.1317 To improve their ability to synthesize HIV behavioral prevention research, the PRS team convened the CDCs Journal Editors Meeting in Atlanta, Ga, on July 2425, 2003; this meeting was attended by editors and representatives of 18 journals that publish HIV behavioral intervention studies (a complete list of the journals is available from the authors and at http://www.TREND-statement.org). The main goals of the meeting were to (1) communicate the usefulness and importance of adequate reporting standards, (2) reach consensus on reporting standards for behavioral interventions, (3) develop a checklist of reporting standards to guide authors and journal reviewers, and (4) develop strategies to disseminate the resulting reporting standards.
The discussions at the meeting broadened to include standardized reporting of behavioral and public health interventions in general, rather than focusing only on HIV behavioral interventions. There was strong consensus at the meeting in regard to more standardized and transparent reporting of research evaluations using other than randomized designs, particularly those with some form of comparison group. This agreement was reached with the realization that additional input would be needed from a wide variety of researchers, other journal editors, and practitioners in the public health field before the adoption of a final set of reporting standards.
Table 1
presents a proposed checklistthe Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) checklistfor reporting standards of behavioral and public health intervention evaluations involving nonrandomized designs. The TREND checklist is meant to be consistent with the CONSORT checklist for the reporting of RCTs. Items presented in boldface type in the table are particularly relevant to behavioral and public health intervention studies, whether or not randomized designs are used. Thus, we would suggest that they be used to expand the information requested by CONSORT for RCTs of behavioral and public health interventions. Some of the items (8, 10, and 15) presented in the proposed TREND checklist are not relevant to RCTs and, thus, not included in the CONSORT checklist, but they are extremely relevant to nonrandomized designs. We also refer readers to CONSORT elaboration reports11,18 that provide rationales and examples for items in Table 1
that are shared with the CONSORT checklist.
|
It is important to note that the TREND checklist is not intended to serve as a criterion for evaluating papers for publication. Rather, it is intended to improve the quality of data reporting in peer-reviewed publications so that the conduct and the findings of research are transparent. As the volume of public health literature is consistently expanding, research synthesis becomes an important tool for creating a cumulative body of knowledge and making evidence-based recommendations of effective interventions. Reporting standards will help ensure that fewer intervention trials with nonrandomized designs are missing information critical for research synthesis and that comparable information across studies can be more easily consolidated and translated into generalizable knowledge and practice.
We recognize several challenges in promoting and disseminating reporting standards for nonrandomized intervention evaluations. Most important, the TREND checklist is only a suggested set of guidelines and should be considered a work in progress. It is highly likely that improvements will be necessary; moreover, adaptations may be needed to refine the standards for specific fields of intervention research, and additional specifications for specific types of nonrandomized evaluation designs are likely to be needed. Furthermore, page limitations in many journals create strong pressure toward shorter rather than longer articles. Some alternatives were recommended in the CDCs Journal Editors Meeting to resolve the space issue, such as having additional information from a published study provided on a journals Web site or on an authors Web site or having authors send additional information to relevant research synthesis groups or a central repository.
Finally, the process has so far involved only CDC scientists and journal editors in a single meeting along with the preparation of this commentary. Although many of the journal editors included are notable researchers in the fields of HIV, public health, and drug abuse prevention, we realize that successful promotion and dissemination of these guidelines must involve an ongoing dialogue and must be extended to a large number of other researchers, methodologists, and statisticians across various healthrelated research fields.
In an effort to initiate this dialogue, we invite all editors, reviewers, authors, and readers to provide comments and feedback to help us revise the standards. Comments can be sent to TREND{at}cdc.gov, and the TREND group will periodically revise the guidelines accordingly. Also, journals are encouraged to endorse this effort by publishing editorials or commentaries on the TREND statement or by referencing it in their publication guidelines for authors and reviewers. To increase accessibility and ease of use, the revised versions of the TREND statement will be posted on an open access Web site (http://www.TREND-statement.org).
If the movement toward evidence-based public health is to succeed, it will be necessary to improve our ability to synthesize research on public health interventions. As Victora and colleagues note, this will include using data from intervention evaluations that do not involve randomized designs. The TREND statement presented here is proposed as a first step toward developing standardized and transparent reporting for nonrandomized intervention research evaluations in public healthrelated fields.
| Acknowledgments |
|---|
We also thank the following Prevention Research Synthesis team members for their assistance in the meeting preparation and contribution to the data reporting presentation: Julia Britton, Tanesha Griffin, Angela Kim, Mary Mullins, Paola Marrero-Gonzalez, Sima Rama, R. Thomas Sherba, and Sekhar Thadiparthi.
| Footnotes |
|---|
Accepted for publication October 26, 2003.
| References |
|---|
|
|
|---|
2. US Cochrane Center. The Cochrane Collaboration. Available at: http://www.cochrane.us. Accessed January 5, 2004.
3. Evidence-Based Medicine Resource Center, New York Academy of Medicine. Home page. Available at: http://www.ebmny.org/thecentr2.html. Accessed January 5, 2004.
4. Agency for Healthcare Research and Quality. AHRQs evidence-based practice centers. Available at: http://www.ahrq.gov/clinic/epc. Accessed January 5, 2004.
5. Centers for Disease Control and Prevention. Guide to community preventive services. Available at: http://www.thecommunityguide.org. Accessed January 5, 2004.
6. Victora CG, Habicht J-P, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94:400405.
7. Donner A, Klar N. Pitfalls of and controversies in cluster randomization trials. Am J Public Health. 2004;94:416422.
8. Murray DM, Varnell SP, Blitstein JL. Design and analysis of grouprandomized trials: a review of recent methodological developments. Am J Public Health. 2004;94:423432.
9. Varnell et al. Design and Analysis of Group-Randomized Trials: A Review of Recent Practices. Am J Public Health. 2004;94:393399.
10. Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Boston, Mass: Houghton Mifflin Co; 1963.
11. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Lancet. 2001;357:11911194.[ISI][Medline]
12. Moher D, Jones A, Lepage L. Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001;285:19921995.
13. Semaan S, Kay L, Strouse D, et al. A profile of U.S.-based trials of behavioral and social interventions for HIV risk-reduction. J Acquir Immune Defic Syndr. 2002;30(suppl 1):S30S50.
14. Semaan S, Des Jarlais DC, Sogolow E, et al. A meta-analysis of the effect of HIV prevention interventions on the sex behaviors of drug users in the United States. J Acquir Immune Defic Syndr. 2002;30(suppl 1):S73S93.
15. Johnson W, Hedges L, Ramirez G, et al. HIV prevention research for men who have sex with men: a systematic review and meta-analysis. J Acquir Immune Defic Syndr. 2002;30(suppl 1):S118S129.
16. Mullen PD, Ramirez G, Strouse D, Hedges L, Sogolow E. Meta-analysis of the effects of behavior HIV prevention interventions on the sexual risk behavior of sexually experienced adolescents in U.S. trials. J Acquir Immune Defic Syndr. 2002;30(suppl 1):S94S105.
17. Neumann MS, Johnson WD, Semaan S, et al. Review and meta-analysis of HIV prevention intervention research for heterosexual adult populations in the United States. J Acquir Immune Defic Syndr. 2002;30(suppl 1):S106S117.
18. Altman DG, Schultz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134:663694.
This article has been cited by other articles:
![]() |
C. R. Hollin, J. McGuire, J. C. Hounsome, R. M. Hatcher, C. A.L. Bilby, and E. J. Palmer Cognitive Skills Behavior Programs for Offenders in the Community: A Reconviction Analysis Criminal Justice and Behavior, March 1, 2008; 35(3): 269 - 283. [Abstract] [PDF] |
||||
![]() |
R. Armstrong, E. Waters, L. Moore, E. Riggs, L. G. Cuervo, P. Lumbiganon, and P. Hawe Improving the reporting of public health intervention research: advancing TREND and CONSORT J. Public Health Med., March 1, 2008; 30(1): 103 - 109. [Abstract] [Full Text] [PDF] |
||||
![]() |
C. R. Hollin Evaluating offending behaviour programmes: Does only randomization glister? JCriminology and Criminal Justice, February 1, 2008; 8(1): 89 - 106. [Abstract] [PDF] |
||||
![]() |
J. M. Raad, S. Bellinger, E. McCormick, M. C. Roberts, and R. G. Steele Brief Report: Reporting Practices of Methodological Information in Four Journals of Pediatric and Child Psychology J. Pediatr. Psychol., December 27, 2007; (2007) jsm130v1. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. A. Reed, D. A. Cook, T. J. Beckman, R. B. Levine, D. E. Kern, and S. M. Wright Association Between Funding and Quality of Published Medical Education Research JAMA, September 5, 2007; 298(9): 1002 - 1009. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. W Taylor, K. A McAuley, W. Barbezat, A. Strong, S. M Williams, and J. I Mann APPLE Project: 2-y findings of a community-based obesity prevention program in primary school age children Am. J. Clinical Nutrition, September 1, 2007; 86(3): 735 - 742. [Abstract] [Full Text] [PDF] |
||||
![]() |
The Communication Evaluation Expert Panel, J. Abbatangelo-Gray, G. E. Cole, and M. G. Kennedy Guidance for Evaluating Mass Communication Health Initiatives: Summary of an Expert Panel Discussion Sponsored by the Centers for Disease Control and Prevention Eval Health Prof, September 1, 2007; 30(3): 229 - 253. [Abstract] [PDF] |
||||
![]() |
D. L. Weeks The Regression Effect as a Neglected Source of Bias in Nonrandomized Intervention Trials and Systematic Reviews of Observational Studies Eval Health Prof, September 1, 2007; 30(3): 254 - 265. [Abstract] [PDF] |
||||
![]() |
A. E Pettifor, C. MacPhail, S. Bertozzi, and H. V Rees Challenge of evaluating a national HIV prevention programme: the case of loveLife, South Africa Sex. Transm. Inf., August 1, 2007; 83(suppl_1): i70 - i74. [Abstract] [Full Text] [PDF] |
||||
![]() |
W. Hollingworth and J. G. Jarvik Technology Assessment in Radiology: Putting the Evidence in Evidence-based Radiology Radiology, July 1, 2007; 244(1): 31 - 38. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. Salmon, M. L. Booth, P. Phongsavan, N. Murphy, and A. Timperio Promoting Physical Activity Participation among Children and Adolescents Epidemiol. Rev., June 7, 2007; (2007) mxm010v1. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. H. Robinson, J. R. Levin, G. D. Thomas, K. A. Pituch, and S. Vaughn The Incidence of "Causal" Statements in Teaching-and-Learning Research Journals American Educational Research Journal, June 1, 2007; 44(2): 400 - 413. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. P. Stone, B. S. Cooper, C. C. Kibbler, B. D. Cookson, J. A. Roberts, G. F. Medley, G. Duckworth, R. Lai, S. Ebrahim, E. M. Brown, et al. The ORION statement: guidelines for transparent reporting of Outbreak Reports and Intervention studies Of Nosocomial infection J. Antimicrob. Chemother., May 1, 2007; 59(5): 833 - 840. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. L. Hess, J. S. Reingold, J. Jones, M. A. Fellman, P. Knowles, J. E. Ravenell, S. Kim, J. Raju, E. Ruger, S. Clark, et al. Barbershops as Hypertension Detection, Referral, and Follow-Up Centers for Black Men Hypertension, May 1, 2007; 49(5): 1040 - 1046. [Abstract] [Full Text] [PDF] |
||||
![]() |
E. Mayo-Wilson Reporting Implementation in Randomized Trials: Proposed Additions to the Consolidated Standards of Reporting Trials Statement Am J Public Health, April 1, 2007; 97(4): 630 - 633. [Abstract] [Full Text] [PDF] |
||||
![]() |
C. M. Lyles, L. S. Kay, N. Crepaz, J. H. Herbst, W. F. Passin, A. S. Kim, S. M. Rama, S. Thadiparthi, J. B. DeLuca, M. M. Mullins, et al. Best-Evidence Interventions: Findings From a Systematic Review of HIV Behavioral Interventions for US Populations at High Risk, 2000-2004 Am J Public Health, January 1, 2007; 97(1): 133 - 143. [Abstract] [Full Text] [PDF] |
||||
![]() |
E. Wood, M. W. Tyndall, J. S. Montaner, and T. Kerr Summary of findings from the evaluation of a pilot medically supervised safer injecting facility. Can. Med. Assoc. J., November 21, 2006; 175(11): 1399 - 1404. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. E. Fey Commentary on "Making Evidence-Based Decisions About Child Language Intervention in Schools" by Gillam and Gillam Lang Speech Hear Serv Sch, October 1, 2006; 37(4): 316 - 319. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. E. Glasgow, L. M. Klesges, D. A. Dzewaltowski, P. A. Estabrooks, and T. M. Vogt Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues Health Educ. Res., October 1, 2006; 21(5): 688 - 694. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. Lang Documenting research in scientific articles: guidelines for authors: reporting research designs and activities. Chest, October 1, 2006; 130(4): 1263 - 1268. [Full Text] [PDF] |
||||
![]() |
D. B. Abrams Applying transdisciplinary research strategies to understanding and eliminating health disparities. Health Educ Behav, August 1, 2006; 33(4): 515 - 531. [Abstract] [PDF] |
||||
![]() |
K. Stronks and J. P. Mackenbach Evaluating the effect of policies and interventions to address inequalities in health: lessons from a Dutch programme Eur J Public Health, August 1, 2006; 16(4): 346 - 353. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. Viotti, C. Vigliano, B. Lococo, G. Bertocchi, M. Petti, M. G. Alvarez, M. Postan, and A. Armenti Long-Term Cardiac Outcomes of Treating Chronic Chagas Disease with Benznidazole versus No Treatment: A Nonrandomized Trial Ann Intern Med, May 16, 2006; 144(10): 724 - 734. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. Young, J Vidanapathirana, M. Abramson, A Forbes, C Fairley, D. Operario, and N. Crepaz Cochrane Column Int. J. Epidemiol., April 1, 2006; 35(2): 233 - 236. [Full Text] [PDF] |
||||
![]() |
L. W. Green and R. E. Glasgow Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof, March 1, 2006; 29(1): 126 - 153. [Abstract] [PDF] |
||||
![]() |
L. M. Morton, J. Cahill, and P. Hartge Reporting Participation in Epidemiologic Studies: A Survey of Practice Am. J. Epidemiol., February 1, 2006; 163(3): 197 - 203. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Nebot Health promotion evaluation and the principle of prevention J. Epidemiol. Community Health, January 1, 2006; 60(1): 5 - 6. [Full Text] [PDF] |
||||
![]() |
N. Jackson, E. Waters, and for the Guidelines for Systematic Reviews in Healt Criteria for the systematic review of health promotion and public health interventions Health Promot. Int., December 1, 2005; 20(4): 367 - 374. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. C. Des Jarlais, C. M. Lyles, and N. Crepaz RE: "QUALITY OF REPORTING OF OBSERVATIONAL LONGITUDINAL RESEARCH" Am. J. Epidemiol., November 15, 2005; 162(10): 1032 - 1032. [Full Text] [PDF] |
||||
![]() |
S. M. Chang, S. L. Reynolds, N. Butowski, K. R. Lamborn, J. C. Buckner, R. S. Kaplan, and D. D. Bigner GNOSIS: Guidelines for neuro-oncology: Standards for investigational studies--reporting of phase 1 and phase 2 clinical trials Neuro-oncol, October 1, 2005; 7(4): 425 - 434. [Abstract] [PDF] |
||||
![]() |
F Davidoff and P Batalden Toward stronger evidence on quality improvement. Draft publication guidelines: the beginning of a consensus project Qual. Saf. Health Care, October 1, 2005; 14(5): 319 - 325. [Abstract] [Full Text] [PDF] |
||||
![]() |
W. Hollingworth Radiology Cost and Outcomes Studies: Standard Practice and Emerging Methods Am. J. Roentgenol., October 1, 2005; 185(4): 833 - 839. [Full Text] [PDF] |
||||
![]() |
W. N. Hait Updated Methods for Reporting Clinical Trials Clin. Cancer Res., October 1, 2005; 11(19): 6753 - 6754. [Full Text] [PDF] |
||||
![]() |
A J Munro The conventional wisdom and the activities of the middle range Br. J. Radiol., May 1, 2005; 78(929): 381 - 383. [Full Text] [PDF] |
||||
![]() |
M. A Ciliberto, H. Sandige, M. J Ndekha, P. Ashorn, A. Briend, H. M Ciliberto, and M. J Manary Comparison of home-based therapy with ready-to-use therapeutic food with standard therapy in the treatment of malnourished Malawian children: a controlled, clinical effectiveness trial Am. J. Clinical Nutrition, April 1, 2005; 81(4): 864 - 870. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. A. Johnson, D. T. Eurich, E. L. Toth, R. Z. Lewanczuk, T. Lee, and S. R. Majumdar Generalizability and Persistence of a Multifaceted Intervention for Improving Quality of Care for Rural Patients With Type 2 Diabetes Diabetes Care, April 1, 2005; 28(4): 783 - 788. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. Tooth, R. Ware, C. Bain, D. M. Purdie, and A. Dobson Quality of Reporting of Observational Longitudinal Research Am. J. Epidemiol., February 1, 2005; 161(3): 280 - 288. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. M. Bernhardt Communication at the Core of Effective Public Health Am J Public Health, December 1, 2004; 94(12): 2051 - 2053. [Full Text] [PDF] |
||||
![]() |
D. A. Dzewaltowski, P. A. Estabrooks, L. M. Klesges, and R. E. Glasgow TREND: AN IMPORTANT STEP, BUT NOT ENOUGH Am J Public Health, September 1, 2004; 94(9): 1474 - 1474. [Full Text] [PDF] |
||||
![]() |
B. Kirkwood Making public health interventions more evidence based BMJ, April 24, 2004; 328(7446): 966 - 967. [Full Text] [PDF] |
||||
| |||||||||||||||||||||||||||||||||||