|
 |
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
© 2007 American Public Health Association
DOI: 10.2105/AJPH.2005.081711
TOWARD A PREPARED FUTURE FOR PUBLIC HEALTH |
Bernard C. K. Choi is with the Centre for Chronic Disease Prevention and Control, Public Health Agency of Canada, Ottawa, Ontario; Department of Public Health Sciences, University of Toronto, Toronto, Ontario; and the Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa. John Frank is with the Institute of Population and Public Health, Canadian Institutes of Health Research, Toronto; Department of Public Health Sciences, University of Toronto, Toronto; and Institute for Work and Health, Toronto. Jennifer S. Mindell is with the Department of Epidemiology and Public Health, University College, London, England. Anna Orlova is with the Division of Health Sciences Informatics, Johns Hopkins School of Medicine; Public Health Data Standards Consortium; and the Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Md. Vivian Lin is with the School of Public Health, La Trobe University, Bundoora, Australia. Alain D. M. G. Vaillancourt is with the Office of Public Health Practice, Public Health Agency of Canada, Ottawa. Pekka Puska is with the National Public Health Institute, Helsinki, Finland. Tikki Pang is with Research Policy and Cooperation, World Health Organization, Geneva, Switzerland. Harvey A. Skinner is with the Faculty of Health, York University, Toronto. Marsha Marsh is with the Environmental Science Center, US Environmental Protection Agency, Fort Meade, Md. Ali H. Mokdad is with the Behavioral Surveillance Branch, National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention, Atlanta, Ga. Shun-Zhang Yu is with the School of Public Health, Fudan University, Shanghai, China; and the Shanghai Preventive Medicine Association, Shanghai. M. Cristina Lindner is with the Hospital de Clínicas, Universidad de la República del Uruguay, Montevideo. Gregory Sherman is with the Office of Public Health Practice, Public Health Agency of Canada, Ottawa. Sandhi M. Barreto is with the Federal University of Minas Gerais, Belo Horizonte, Brazil. Lawrence W. Green is with the Department of Epidemiology and Biostatistics, University of California, San Francisco. Lawrence W. Svenson is with the Public Health Surveillance and Environmental Health Branch, Alberta Ministry of Health and Wellness, Calgary; Department of Public Health Sciences, University of Alberta, Calgary; and the Department of Community Health Sciences, University of Calgary, Calgary. Peter Sainsbury is with Population Health, Sydney South West Area Health Service, Camperdown, Australia; and the School of Public Health, University of Sydney, Sydney, Australia. Yongping Yan is with the Department of Epidemiology, Fourth Military Medical University, Xian, China. Zuo-Feng Zhang is with the Department of Epidemiology, University of California School of Public Health, Los Angeles. Juan C. Zevallos is with the National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention, Atlanta. Suzanne C. Ho is with the Department of Community and Family Medicine, and the School of Public Health, Chinese University of Hong Kong, Hong Kong, China. Ligia M. de Salazar is with the Public Health Evaluation Centre, University of Valle, Cali, Colombia.
Correspondence: Requests for reprints should be sent to Bernard C. K. Choi, Centre for Chronic Disease Prevention and Control, Public Health Agency of Canada, Government of Canada, AL#6701A, Ottawa, Ontario K1A 1B4, Canada (e-mail: Bernard_ Choi{at}phac-aspc.gc.ca).
ABSTRACT
In public health, the generation, management, and transfer of knowledge all need major improvement. Problems in generating knowledge include an imbalance in research funding, publication bias, unnecessary studies, adherence to fashion, and undue interest in novel and immediate issues.
Impaired generation of knowledge, combined with a dated and inadequate process for managing knowledge and an inefficient system for transferring knowledge, mean a distorted body of evidence available for decisionmaking in public health. This article hopes to stimulate discussion by proposing a Global Registry of Anticipated Public Health Studies.
This prospective, comprehensive system for tracking research in public health could help enhance collaboration and improve efficiency. Practical problems must be discussed before such a vision can be further developed.
IN 2004, THE INTERNATIONAL Committee of Medical Journal Editors announced that, from July 2005, researchers submitting articles to 11 medical journals would be asked to report the full results of clinical trials, both positive and negative, and that the journals would not publish studies unless they had been included in a public registry at their inception.1 This policy should guard against selective reporting of trials and the distortion of the body of evidence available for clinical decisionmaking.2 The International Committee of Medical Journal Editors has now specified the minimum registration data set.3,4
We believe that similar considerations and standards are needed for nontrial public health studies. Public health studies are studies related to the efforts organized by society to protect, promote, and restore the people’s health,5 many of which are observational and nonexperimental, or involve "natural experiments."6 We examined the current problems in generating, managing, and transferring knowledge in public health and have described a vision for a future Global Registry of Anticipated Public Health Studies (GRAPHS).
Knowledge generation and use are a critical foundation of effective public health programs and policies, but many problems are evident in current practice. Because this article covers a broad scope, it can discuss neither all of the issues nor each issue in sufficient depth. Also, the proposed vision is not meant to address all of the numerous practical issues but to generate public discussion. We are describing a vision to encourage possible solutions to important problems; we are not making a concrete proposal. For the purpose of this article, the knowledge cycle is divided into 3 stages: (1) knowledge generation (also known as knowledge acquisition or creation)7; (2) knowledge management (exploitation and development of the knowledge assets)8; and (3) knowledge transfer (also known as knowledge exchange, dissemination, access, brokering, or translation).7
THE NEED
Knowledge Generation
The first problem in generating knowledge is the imbalance in the funding of health care versus health protection, which is mirrored in the funding of medical versus public health research. For example, in the United States, less than 1% of the US $1.9 trillion annually spent on health care is spent on health protection and prevention.9 In the United Kingdom, although there is a government commitment to improving population health and prevention,10 only 2% of cancer research funding11 and 0.4% of public health research12 are provided for prevention. The problem is exacerbated because few major journals are interested in publishing public health research.6 Thus, the Guide to Community Preventive Services,13 a major US effort to synthesize current evidence supporting health promotion and prevention programs, reveals significant gaps (lack of studies of reasonable quality) in the current knowledge base when reviewing evidence to support health promotion programs.14 Furthermore, these knowledge gaps are not random and are found to be greater for interventions developed by the public sector versus those funded by private interests, for interventions that are more complex, longer term, and focused on upstream versus downstream causes of ill health, and in the developing versus the developed world.15
The second problem is the accumulation of false-positive information, or the "false-positive research cycle."16 It begins with publication of a false-positive study (possibly because of bias or chance), which then leads to new studies on the same topic (hot topic bias). At the conventional significance level of 0.05, 100 studies where the null hypothesis is true will, on average, yield 5 false-positive studies. Because positive results are more likely to be submitted to scientific journals and to be accepted by editors, a cycle can be generated that may convince readers of something that is not true.16
The third problem is continuing research themes when issues are largely settled, or "circular epidemiology."17 True-positive (or true-negative) studies are repeated needlessly, wasting resources that could have been used to study other more pressing problems. For example, between 1987 and 2002, 64 studies were published on the effectiveness of aprotinin (to reduce perioperative blood loss). Because the effectiveness was clearly established after the 12th study in 1992, what followed were essentially 52 studies that did not really advance science.18
The fourth problem is the funding of areas that are in fashion at the expense of others less in vogue but more important. Once a certain volume of research has been conducted in an area, a group of experts is formed whose vested interest in the topic influences future decisions on funding and publication.
The fifth problem is nonpublication of unpopular or even politically incorrect studies, or "cul-de-sac epidemiology."19 Findings are shunned by the medical community and the media because they are deemed inappropriate (e.g., modern obstetric anesthesia as a possible risk factor for autism). The original studies are not replicated, even by the original investigators, and they are rarely quoted after publication.19
The sixth problem relates to a novelty effect in research. Scientific journals, researchers, the media, and the public show a particular interest in "new" risk factors. Even very shaky evidence of possible new risk factors draws media and public attention that helps cast doubt on the known risk factors. Thus, researchers still go about searching for new risk factors for coronary heart disease after most are already known, as traditional risk factors for coronary heart disease have been found to explain 75%20 to 90%21 of new cases.
The seventh problem is the preoccupation of health researchers and practitioners with immediate but not necessarily the most important health issues.22 Thus, to accentuate the first problem, the search for effective treatment often overshadows prevention, even though approximately half of all deaths in the United States in 2000 were driven by behavioral and social risk factors, with 40% attributable to tobacco, poor diet and physical inactivity, or excessive drinking of alcoholic beverages, all of them potentially preventable.23
The eighth problem involves sacrificing external validity for internal validity. The problem arises when control measures used to ensure greater internal validity create artificial and unnatural circumstances that limit external validity (generalizability). Thus, an efficacious intervention in a controlled environment fails to work when used in normal practice circumstances.24
Furthermore, many epidemiological biases affect the design, data collection, and analysis of public health research.25,26 At least 48 types of questionnaire biases have been identified.27 Other issues with public health research include the underfunding of upstream research addressing disease prevention, the quality of research, and the importance of research synthesis. Added to this list are translation of research into practice, costeffectiveness analysis, and issues of false-negatives and multiple end points within a single study (data on many of which may not be published).
Knowledge Management
The current process for managing knowledge produces a dated, incomplete evidence base for decisions. First, it usually takes many years from conceptualization of a public health research idea to publication of findings in the scientific literature. Systems for managing knowledge may also lack the staffing or other resources needed to maintain currency.
Second, search of the "gray" literature (e.g., unpublished and internal reports, technical documents), although improved by the Internet, remains notoriously difficult to conduct and replicate.28 Even when a search uncovers relevant work, obtaining a copy can be difficult. In addition, some studies may never be written up, because time or interest is lacking. In the case of government agency– or industry-funded research, clearance and official checks may limit what can be released as public information. These problems are not resolved by even the most comprehensive literature search and review strategies, and they are accentuated among researchers who are not native English speakers or who reside in countries with less-developed access to the literature.
Knowledge Transfer
The current process for transferring public health knowledge is hindered by the insufficient investment in directed research. In the end, the clear identification of priorities by decisionmakers and funders of research may mean relatively little. Research may be funded and conducted in a targeted area, but the findings produced do not necessarily answer the strategic questions that guide policy and practice. The result is that those who make decisions in public health are often frustrated by their inability to base those decisions on relevant research.29
Much of public health research focuses on the discovery and characterization of health problems, rather than on the effective interventions or possible solutions. Yet, policymakers are often looking for intervention studies. Community-based intervention studies are less easily located, not only because they are methodologically challenging and, therefore, more difficult to conduct and publish, but many policy and program evaluations may be commissioned by non-academic sponsors and, therefore, may not be published or easily accessible because they have less incentive to publish.30 Even if community-based intervention studies are commissioned and published, there is still the challenge of how to apply the findings to the right population groups or community circumstances. There is a need for guidance for practitioners and decisionmakers on how best to assess external validity of studies and to apply the evidence in situ.
THE VISION
Our vision for a future knowledge-based information system is GRAPHS. Public health researchers and practitioners will benefit from such a globally collaborative registry, of which the key aim is to provide a platform for the following: (1) ensuring all of the relevant research data becomes publicly available, (2) the identification of research and researchers on specific topics, (3) the cross-validation of studies, (4) the prioritization of research funding for issues of national and international interest, (5) the advancement of research to the next level, and (6) the identification of knowledge gaps.
Research studies on human health, ecological and social studies (including economic evaluation and policy analysis), and other public health investigations would be entered in GRAPHS when they are commissioned or when they are funded by a granting agency. Research findings, whether positive or negative, will be tracked. The method of tracking would be similar to that used by the Cochrane Collaboration31 and Campbell Collaboration,32 global registries for evidence on controlled-trial interventions in health care and the social, behavioral, and educational arenas, respectively. To keep up to date, the registry will need an automated reminder system for researchers to submit ongoing project updates.33 A unique registration number and standardized format will also be needed. Results will be classified by study design34 and the type of data analysis. An effective search engine will be needed to retrieve information.35
GRAPHS must be freely accessible to all interested parties, which raises the issue of protecting intellectual rights. Operational guidelines should strike a careful balance between open access and the privacy (personal information, ownership of data, and copyright) of the individual researchers. More detailed information could be accessed by researchers with an access code.
Ideally, GRAPHS should include studies as soon as the research process begins and, thus, well before any results are obtained or published. GRAPHS registration could be required by all funding bodies and as a condition of granting ethical approval. It will be fundamentally different from a meta-analysis or any current system for synthesizing research, such as Cochrane or Campbell,31,32 which are inherently retrospective.
GRAPHS needs not be built from scratch. For example, it could be based on a process similar to that of US institutional review boards or other granting agencies that require researchers to obtain approval for their studies. To satisfy the institutional review board process, researchers must present comprehensive study protocols. Second, it could link to existing clinical trials databases and current efforts to catalog new knowledge. Examples include the US federal databases36,37 for clinical trials and a clearinghouse for evidence-based clinical practice guidelines, the metaregister and the International Standard Randomized Controlled Trial Number register in the United Kingdom run by Current Controlled Trials,38 and the Australian Clinical Trials Register.39 GRAPHS would be strengthened by giving more attention to links between experimental and observational studies, perhaps by specifying the type of links that could be established between evidence provided by experimental studies (e.g., clinical trials) and by observational studies (e.g., studies listed in GRAPHS), as well as how to relate the results from these 2 types of study designs and how to resolve their incompatible findings.
Third, it could link up and expand on existing databases, such as the Human Genome Epidemiology Network, which is a registry of genetic epidemiological studies40,41; Computer Retrieval of Information on Scientific Projects, which covers biomedical research funded by the National Institutes of Health42; Canadian Research Index on Canada’s government and research literature43; Science and Engineering Knowledge Network, which covers projects publicly funded in the United Kingdom44; the Internet-based International Agency for Research on Cancer Directory of On-going Research in Cancer Prevention45; Trials Register of Promoting Health Interventions on intervention evaluations46; and the Sharing Point Server database, which has a heavy emphasis on developing countries.47
Fourth, "hierarchies of evidence" tables, indicating appropriate research designs to answer specific types of questions and their application frameworks, are available to guide the rating and use of evidence from research studies.34 Fifth, the science of "bibliometrics" (statistical bibliography)48 can help describe and analyze "information epidemics," for example, by studying the numbers of papers published on a new idea and charting their spread.49 Sixth, efforts in the selective dissemination of information by academic libraries over the last 30 years have led to the development of "push" technologies touted for the Internet a few years ago.50,51 These technologies can provide a basis for systems to filter and distribute information, such as an automatic e-mail posting triggered by changes in a Web page and an Internet-based electronic table of contents service.52
Seventh, there are existing methods of review, evaluation, and dissemination that can be integrated into GRAPHS.53 These include meta-analysis54; systematic reviews of qualitative studies55,56; translation of research into practice, such as the Reach, Effectiveness, Adoption, Implementation, and Maintenance framework57,58; and reporting standards and rating scales using Consolidated Standards of Reporting Trials criteria.59 GRAPHS needs to resolve issues such as how criteria for methodologic quality in reporting (Consolidated Standards of Reporting Trials) could be applied to observational research. Progress has been made. For example, Reach, Effectiveness, Adoption, Implementation, and Maintenance has been used to structure a set of criteria for judging the external validity, generalizability, and relevance of a research study beyond the setting and circumstances in which it was conducted and in line with Consolidated Standards of Reporting Trials.53
Because both internal and external validity are important for population health studies, a final component of GRAPHS could be the incorporation of Reach, Effectiveness, Adoption, Implementation, and Maintenance principles.53 When studies are complete, authors could be asked to provide specification of the key components of an evidence-based program and the range of permissible adaptation that would still retain the essential elements of the original efficacy study. In this way, GRAPHS would help provide a platform for research transfer, with explicit consideration being given to reach representativeness, program implementation, outcomes of decisionmaking, maintenance, and institutionalization.
DISCUSSION
There is no quick fix to the current problems in generating, managing, and transferring knowledge in public health. A broad, well-conceived approach is needed, such as GRAPHS.
Strengths
GRAPHS would minimize the effect of the false-positive research cycle by including results of all studies or at least providing a clear paper trail of all studies initiated. It could complement research synthesis efforts by making it easier to identify all of the applicable studies. It would provide a more comprehensive evidence base to facilitate decision-making; for example, one would have a better idea of the proportion of studies with positive results. In the future, simple counts of positive and negative studies will be replaced by more sophisticated techniques to adjust for study quality and to properly assess causation.
In addition, GRAPHS would help researchers and granting agencies avoid excessive investment in topics that are being overly studied (circular epidemiology) and allow due consideration of the legitimate need to replicate some studies. This would allow better use of limited public health research funding. Furthermore, it would help identify gaps in our knowledge, which could lead to a focused call for more studies.60 It would also provide added value to the knowledge base by systematically evaluating all studies, indicating, for example, whether a false-positive study is the result of random chance or poor design. Finally, it would strengthen opportunities for national and international collaboration among investigators.
Weaknesses and Unanswered Questions
GRAPHS would require much discussion among stakeholders to hammer out its details, such as who will fund it, provide it with data, and use its information. Some will question whether the investment (both initial and ongoing) would be warranted. In addition, the primary targets and audiences must be defined more sharply. One should also ask how GRAPHS information would be used by nonacademic audiences, such as policymakers, how it would facilitate knowledge transfer, how it would be integrated with other research registry systems, and whether it might inhibit new research.
It is important to address whether notification should be required by funders of research, whether there should be incentives and penalties to promote registration, or both. Although restrictions by editors on publishing nonregistered studies may be effective, failing to publish such studies may exacerbate the problem of publication bias, which the GRAPHS is designed to combat.
More generally, the question must be settled of how much detail of ongoing research should be put in the public domain and why it should be put there—to better society or simply to encourage collaboration? Additionally, what studies would be excluded from the database? For example, would student masters and doctoral theses be excluded? The private sector and industy is the locus of approximately 40% to 45% of global research and development,61 but should industry-sponsored studies be excluded? Are there public policy, legitimacy, and ethical concerns with the use of such data for centralized planning? Who decides globally what areas need additional funding or other emphases? Do we lose opportunities to allow paradigm shifts?
Threats and challenges
The most critical challenge to GRAPHS involves a relative lack of incentives for individual researchers to submit information on current or planned studies, let alone keep it updated. The experience of universities and other research institutions is instructive here because they often fail to get their own researchers, with their busy schedules, to even complete their institutional profiles of research and expertise on the Internet.
There is also the issue of intellectual property rights and the fear of having one’s ideas stolen. Only on publication do investigators believe their intellectual property rights can be defended. This challenge is even more formidable in basic science, where commercial applications or patents may be at stake.
OPPORTUNITIES
We believe that, in the short term, the incentives for GRAPHS most probably lie with the funders of research in each country who could keep a better log of what they fund and then link these logs internationally. In addition, major users of research, such as governments, may wish to pay for registries of current or forthcoming evidence on key public policy issues.
Some current barriers to GRAPHS, such as lack of incentives (for investigators to participate in GRAPHS), probable disincentives, research traditions, funding, and copyrights, may change in the future. For example, there are ongoing initiatives on "portals" to link existing databases or new databases, such as the entire "open access" (Public Library of Science, BioMed Central) and "open source" movements (e.g., Tropical Disease Initiative).62 The Centers for Disease Control and Prevention have made available, free of charge, an online e-journal (Preventing Chronic Disease), of which the articles are not copyright protected.63
At least in government, many scientists and other researchers conduct studies according to perceived policy needs. Shared information, collaborative efforts, and wide consultation are encouraged during the whole research process. This culture may one day be extended to academic research. Systems for public health research may then be re-engineered to link more closely with policy and practice. GRAPHS should have, at minimum, the capacity to actively monitor research outcomes and perform basic knowledge transfer functions. It should document dissemination of research to date, including citation in policy documents. Because policymakers are not likely to read primary research papers or project proposals or even full structured reviews, 1- or 2-page policy briefs or lay summaries would be necessary.29
In the future, granting agencies may become more assertive with journals, and in turn journals with researchers, in pushing the case that negative studies (of sufficient methodologic quality, including statistical power) are important to publish. In addition, GRAPHS may establish global standards to assist journals and granting agencies in prioritizing research for publication and funding, perhaps even addressing the issue of when the evidence for a finding is sufficient to obviate the need for future studies in that area. In return, scientific journals may establish guidelines that studies are to be published only if they had been initially registered with GRAPHS at inception.
With widespread computer and communication technology, GRAPHS may one day become a virtual database of all studies in public health, readily accessible to all researchers and decisionmakers in every country, subject to suitable anonymization and data safeguards. With good decision aids and access to either raw data or aggregated data, research users could evaluate the research objectively. This would be perceived by some as risky, but scrutiny can only improve an investigator’s work.
CONCLUSIONS
GRAPHS, for all its potential value, would be a major challenge to implement. The need to allocate formidable resources and overcome sociopolitical considerations may keep such a global registry from being born. Conversely, progress in technology and informatics, as well as a rethinking of the research enterprise, may soon make implementation much easier to achieve. This article is a call for public discussion to improve the way new public health scientific knowledge is produced, managed, and transferred.
Acknowledgments
The authors would like to thank Peter L. Taylor for the editorial contribution made to an early version.
Human Participant Protection
No approval was required.
Footnotes
Note. Opinions expressed in this article are solely those of the authors and do not necessarily represent the views of any agencies, organizations, or countries.
Contributors
B.C.K. Choi conceived and designed the study and acquired the data. B.C.K. Choi, J. Frank, J. S. Mindell, A. Orlova, V. Lin, A.D.M.G. Vaillancourt, P. Puska, T. Pang, H. A. Skinner, M. Marsh, A. H. Mokdad, S-Z. Yu, M. C. Lindner, G. Sherman, S.M. Barreto, L.W. Green, L.W. Svenson, P. Sainsbury, Y. Yan, Z-F. Zhang, J. C. Zevallos, S. C. Ho, and L. M. de Salazar were responsible for analysis and interpretation of data, drafting of the article, critical revision of the article for important intellectual content, and final approval of the version to be published.
Accepted for publication September 24, 2006.
References
1. De Angelis CD, Drazen JM, Frizelle FA, et al. Clinical trial registration: a statement from the International Committee of Medical Journal eds. N Engl J Med 2004;351:1250–1251.
2. Medical News Today. Clinical trial registration, a statement from the International Committee of Medical Journal eds. September 9, 2004. Available at: http://www.medicalnewstoday.com/medicalnews.php?newsid=13143. Accessed September 9, 2004.
3. De Angelis CD, Drazen JM, Frizelle FA, et al. Is this clinical trial fully registered?—a statement from the International Committee of Medical Journal eds. N Engl J Med. 2005;352: 2436–2438.
4. Gulmezoglu AM, Pang T, Horton R, Dickersin K. WHO facilitates international collaboration in setting standards for clinical trial registration. Lancet. 2005;365:1829–1831.[CrossRef][Web of Science][Medline]
5. Last JM. A Dictionary of Epidemiology. New York, NY: Oxford University Press; 1995.
6. Mindell J, Joffe M. Mathematical modelling of health impacts. J Epidemiol Community Health. 2005;59: 617–618.
7. Choi BCK. Twelve essentials of science-based policy. Preventing Chronic Dis [serial online]. 2005;2:1–11. Available at: http://www.cdc.gov/pcd/issues/2005/oct/05_0005.htm. Accessed April 3, 2006.
8. Davenport T, Prusak L. Working Knowledge: How Organisations Manage What They Know. Cambridge, Mass: Harvard University Press; 1998.
9. Gerberding JL. Protecting health—the new research imperative. JAMA 2005;294:1403–1406.
10. Wanless D. Securing Good Health for the Whole Population. London, England: HM Treasury; 2003.
11. National Cancer Research Institute. Strategic Analysis 2002. London, England: National Cancer Research Institute; 2003.
12. Millward LM, Kelly MP, Nutbeam D. Public Health Intervention Research—The Evidence. London, England: Health Development Agency; 2003. Available at: http://www.publichealth.nice.org.uk/download.aspx?o=502583. Accessed August 5, 2006.
13. The Guide to Community Preventive Services (Community Guide). Available at: http://www.thecommunityguide.org. Accessed August 10, 2006.
14. Zaza S, Briss PA, Harris KW, eds. Task Force on Community Preventive Services. The Guide to Community Preventive Services: What Works to Promote Health. New York, NY: Oxford University Press; 2005.
15. Briss PA. Evidence-based: US road and public-health side of the street. Lancet. 2005;365:828–830.[CrossRef][Web of Science][Medline]
16. Choi BCK. Perspectives on epidemiologic surveillance in the 21st century. Chronic Dis Can. 1998;19:145–151.[Medline]
17. Kuller LH. Circular epidemiology. Am J Epidemiol 1999;150:897–903.
18. Young C, Horton R. Putting clinical trials into context. Lancet. 2005;366: 107–108.[CrossRef][Web of Science][Medline]
19. Odent M. Between circular and cul-de-sac epidemiology. Lancet. 2000; 355:1371.[Web of Science][Medline]
20. Beaglehole R, Magnus P. The search for new risk factors for coronary heart disease: occupational therapy for epidemiologists? Int J Epidemiol. 2002; 31:1117–1122.
21. Yusuf S, Hawken S, Ôunpuu S, et al., for the INTERHEART Study Investigators. Effect of potentially modifiable risk factors associated with myocardial infarction in 52 countries (the INTER-HEART study): case-control study. Lancet. 2004;364:937–952.[CrossRef][Web of Science][Medline]
22. McGinnis JM, Foege WH. The immediate vs. the important. JAMA. 2004; 291:1263–1264.
23. Mokdad AH, Marks JS, Stroup DF, Gerberding JL. Actual causes of death in the United States, 2000. JAMA. 2004;291:1238–1245.
24. Green LW. From research to best practices in other settings and populations. Am J Health Behav. 2001;25: 165–178.[Web of Science][Medline]
25. Choi BCK, Pak AWP. Bias, overview. In: Armitage P, Colton T, eds. Encyclopedia of Biostatistics. vol. 1. Chichester, England: John Wiley & Sons; 2005:416–423.
26. Choi BCK, Pak AWP. Understanding and minimizing epidemiologic bias in public health research. Can J Public Health 2005;96:284–286.[Web of Science][Medline]
27. Choi BCK, Pak AWP. A catalogue of biases in questionnaires. Preventing Chronic Dis [serial online]. 2005;2: 1–13. Available at: http://www.cdc.gov/pcd/issues/2005/jan/04_0050.htm. Accessed April 3, 2006.
28. Mindell J, Boaz A, Joffe M, Curtis S, Birley M. Enhancing the evidence base for health impact assessment. J Epidemiol Community Health. 2004;58: 546–551.
29. Choi BCK, Pang T, Lin V, et al. Can scientists and policy-makers work together? J Epidemiol Community Health 2005;59:632–637.
30. Lin V, Gibson B, eds. Evidence-Based Healthy Policy: Problems and Possibilities. Melbourne, Australia: Oxford University Press; 2003.
31. The Cochrane Collaboration. Available at: http://www.cochrane.org. Accessed April 1, 2006.
32. The Campbell Collaboration. Available at: http://www.campbellcollaboration.org. Accessed April 1, 2006.
33. Lord Hunt of Kings Health. Research Governance Framework for Health and Social Care. London, England: Department of Health; 2001.
34. Petticrew M, Roberts H. Evidence, hierarchies, and typologies: horses for courses. J Epidemiol Community Health. 2003;57:527–529.
35. Humphreys BL, Hole WT, McCray AT, Fitzmaurice JM. Planned NLM/AHCPR large-scale vocabulary test: using UMLS technology to determine the extent to which controlled vocabularies cover terminology needed for health care and public health. J Am Med Inform Assoc. 1996;3:281–287.
36. US National Institutes of Health. ClinicalTrials.gov. Available at: http://www.clinicaltrials.gov. Accessed August 10, 2006.
37. Agency for Healthcare Research and Quality. National Guideline Clearinghouse. Available at: http://www.guidelines.gov. Accessed August 10, 2006.
38. Science Navigation Group. Current controlled trials. Available at: http://www.controlled-trials.com. Accessed August 10, 2006.
39. Australian Clinical Trials Registry. Available at: http://www.actr.org.au. Accessed August 10, 2006.
40. Centers for Disease Control and Prevention. Human Genome Epidemiology Network. Available at: http://www.cdc.gov/genomics/hugenet/default.htm. Accessed October 17, 2005.
41. Ioannidis JPA, Bernstein J, Boffetta P, et al. A network of investigator networks in human genome epidemiology. Am J Epidemiol 2005;162:302–304.
42. Computer Retrieval of Information on Scientific Projects. Available at: http://crisp.cit.nih.gov. Accessed April 1, 2006.
43. Canadian Research Index. Available at: http://www.micromedia.ca/Products_Services/CRI.htm. Accessed April 1, 2006.
44. Science and Engineering Knowledge Network. Available at: http://www.seknet.co.uk. Accessed May 26, 2004.
45. World Health Organization International Agency for Research on Cancer. Directory of on-going research in cancer prevention. Available at: http://www-dep.iarc.fr/prevent.htm. Accessed October 1, 2004.
46. Trials Register of Promoting Health Interventions. Available at: http://eppi.ioe.ac.uk/EPPIWeb/home.aspx?Control=Search&SearchDB=trials&page=/hp. Accessed April 3, 2006.
47. SHARing Point Server - The SHARED Network. Available at: http://www.sharingpoint.net. Accessed April 4, 2006.
48. Hertzel DH. Bibliometrics history. In: Drake M, ed. Encyclopedia of Library and Information Science. Second Edition. New York, NY: Marcel Dekker, Inc; 2003:288–328.
49. Tabah A. Information Epidemics and the Growth of Physics [dissertation]. Montreal, Quebec: Graduate School of Library and Information Studies, McGill University; 1996.
50. Gant S. Bibliography of Selective Dissemination of Information. Available at: http://www.ils.unc.edu/gants/sdibib.html. Accessed May 26, 2004.
51. Selective Dissemination of Information. Available at: http://143.169.20.1/MAN/SDIE/t1.html. Accessed May 26, 2004.
52. Oxford Journals Content Alerting. Available at: http://www3.oup.co.uk/jnls/tocmail. Accessed on May 26, 2004.
53. Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29: 126–153.
54. Dale KM, Coleman CI, Henyan NN, Kluger J, White CM. Statins and cancer risk: a meta-analysis. JAMA 2006;295: 74–80.
55. Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qual Health Res 1998;8:341–351.
56. Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Newbury Park, CA: Sage; 1988.
57. Glasgow RE, Vogt EM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999;89:1322–1327.
58. Glasgow RE. Evaluation of theory-based interventions: the RE-AIM model. In: Glanz K, Lewis FM, Rimer BK, eds. Health Behavior and Health Education. San Francisco, CA: Jossey-Bass; 1997: 531–544.
59. Mohrer D, Schulz KF, Altman DG, Lepage L. The CONSORT statement: revised recommendations for improving the quality of reports. JAMA 2001;285: 1987–1991.
60. Choi BCK. Re: "invited commentary: circular epidemiology". Am J Epidemiol 2000;151:1036–1037.
61. Global Forum for Health Research. 10/90 Report on Health Research 2003–2004. Geneva, Switzerland: Global Forum for Health Research; 2004.
62. Economist. An open-source shot in the arm? June 10, 2004. Available at: http://www.economist.co.uk/science/tq/PrinterFriendly.cfm?Story_ID=2724420. Accessed September 30, 2004.
63. Centers for Disease Control and Prevention. Preventing Chronic Disease [serial online]. Available at: http://www.cdc.gov/pcd. Accessed April 3, 2006.
This article has been cited by other articles:
|
|
 |
|
  B. C K Choi, A. Gupta, and B. Ward Good thinking: six ways to bridge the gap between scientists and policy makers J Epidemiol Community Health, March 1, 2009; 63(3): 179 - 180. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |