| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
RESEARCH |
At the time of the study, Michael D. Kluger and Rajesh K. Sodhi were with the Connecticut Emerging Infections Program, Department of Epidemiology and Public Health, Yale University School of Medicine, New Haven, Conn. Andre N. Sofair is with the Departments of Internal Medicine and Epidemiology and Public Health and the Connecticut Emerging Infections Program, Yale University School of Medicine. Constance J. Heye is with Urban Policy Strategies, New Haven, CT. James I. Meek is with the Connecticut Emerging Infections Program, Department of Epidemiology and Public Health, Yale University School of Medicine. James L. Hadler is with the Bureau of Community Health, Infectious Diseases Division, State of Connecticut Department of Public Health, Hartford.
Correspondence: Requests for reprints should be sent to Andre N. Sofair, MD, MPH, Connecticut Emerging Infections Program, Yale University School of Medicine, Department of Epidemiology and Public Health, 40 Temple St, Suite 1B, New Haven, CT 06510 (e-mail: andre.sofair{at}yale.edu).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONEMERGING INFECTIONS
PROGRAMSMETHODSRESULTSDISCUSSIONReferences |
|---|
Objectives. This study investigated retrospective validation of a prospective surveillance system for unexplained illness and death due to possibly infectious causes.
Methods. A computerized search of hospital discharge data identified patients with potential unexplained illness and death due to possibly infectious causes. Medical records for such patients were reviewed for satisfaction of study criteria. Cases identified retrospectively were combined with prospectively identified cases to form a reference population against which sensitivity could be measured.
Results. Retrospective validation was 41% sensitive, whereas prospective surveillance was 73% sensitive. The annual incidence of unexplained illness and death due to possibly infectious causes during 1995 and 1996 in the study county was conservatively estimated to range from 2.7 to 6.2 per 100 000 residents aged 1 to 49 years.
Conclusions. Active prospective surveillance for unexplained illness and death due to possibly infectious causes is more sensitive than retrospective surveillance conducted through a published list of indicator codes. However, retrospective surveillance can be a feasible and much less labor-intensive alternative to active prospective surveillance when the latter is not possible or desired.
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONEMERGING INFECTIONS
PROGRAMSMETHODSRESULTSDISCUSSIONReferences |
|---|
Reliance on traditional responsive methods to identify infectious agents may delay prevention and control efforts. While advancements in biomedical technology have allowed for more rapid identification of microbial agents, population-based surveillance networks capable of identifying trends in infectious disease symptomatology have deteriorated.7 Systematic prospective study of the epidemiology of infectious disease syndromes is needed for earlier detection of and response to emerging infections.6,8
|
EMERGING INFECTIONS PROGRAMS |
|---|
|
TOPABSTRACTINTRODUCTIONEMERGING INFECTIONS
PROGRAMSMETHODSRESULTSDISCUSSIONReferences |
|---|
A preliminary study estimated that the annual incidence of unexplained death due to possibly infectious causes among previously healthy New Haven County, Connecticut, residents aged 1 to 49 years was 14.2 per 100 000.6 This figure was based on a retrospective review of multiple cause-of-death data included in the 1992 National Center for Health Statistics death record that selected for 77 International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes believed by the study authors to be indicative of unexplained death due to possibly infectious causes.9 Persons not previously healthy, as indicated by another series of ICD-9-CM codes, were excluded.
Beginning in August 1995, the Connecticut Emerging Infections Program conducted surveillance for unexplained illness and death due to possibly infectious causes (hereafter "unexplained illness and death") in the 7 acute care hospitals of New Haven County. Between August 1, 1995, and December 31, 1996, 16 cases of unexplained illness and death were prospectively identified in New Haven County, yielding an annualized incidence of 1.9 episodes per 100 000 residents aged 1 to 49 years (based on an estimated surveillance population of 584 507).10 This annualized incidence rate was well below the 14.2 deaths per 100 000 population estimated for 1992. This disparity was unexpected, because the prospective surveillance aimed to identify both critical illnesses and deaths, whereas the preliminary 1992 retrospective study examined only deaths.6
Given the personnel, time, and financial resources required for prospective surveillance; the significance of the study's objective to the public's health; and the discrepancy between preliminary estimates, it was crucial to evaluate the efficacy of this system. We report on a retrospective validation study performed to assess the sensitivity of prospective surveillance for unexplained illness and death at the 7 participating acute care facilities for the period August 1, 1995, through December 31, 1996.
|
METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONEMERGING INFECTIONS
PROGRAMSMETHODSRESULTSDISCUSSIONReferences |
|---|
Persons not falling in the 1- to 49-year age range were excluded because of increased susceptibility to infection and increased occurrence of underlying morbidity. Intensive care units were selected as the point of access for the prospective surveillance on the basis of the assumption that individuals with life-threatening illnesses would probably be admitted to an intensive care unit during their course of treatment. The clinical definition of unexplained illness and death was based on the methods and recommendations of Perkins et al.6 with 1 major modification: Perkins et al. included only unexplained death in their investigation, whereas we included both unexplained illness and unexplained death.
Both active and passive surveillance techniques were used to identify cases. Active surveillance refers to that in which surveillance staff make regular contact with physicians or other qualified individuals or use electronic medical record systems to elicit reports of disease occurrence. In contrast, passive surveillance refers to that in which surveillance staff receive disease reports from physicians, other qualified individuals, or electronic medical record systems.11
Active prospective surveillance was conducted at hospital A, the largest hospital in the county. This surveillance consisted of routine contact with physicians, nurses, and infection control personnel to identify incidents of unexplained illness and death. In addition, an Emerging Infections Program epidemiologist (Constance J. Heye) reviewed weekday computerized intensive care unit census information to identify potential cases based on preliminary diagnoses.
Passive prospective surveillance was conducted at the 6 additional hospitals (hospitals B–G). Physicians, nurses, and infection control personnel at these hospitals were encouraged to report potential cases of unexplained illness and death to our study staff. In the case of passive surveillance, study staff did not review computerized intensive care unit census information and did not work as closely with hospital personnel to identify cases of unexplained illness and death. In both active and passive surveillance, suspected cases were referred to the study physician (Andre N. Sofair), who made the final determination as to whether a patient satisfied the case definition.
Retrospective Validation
The
unexplained illness and death case definition for the retrospective validation
was identical to the definition used for prospective surveillance. To conform
with the inclusion criteria of the prospective surveillance, we requested data
from each of the 7 participating hospitals on all patients aged 1 to 49 years
admitted to an intensive care unit during the period August 1, 1995, through
December 31, 1996. The following information was requested for each patient:
medical record number or name (or both), date of birth, admission and discharge
dates, residence zip code, and all discharge ICD-9-CM
codes.
In assessing cases of unexplained illness and death, data were
sorted via an Epi Info program to identify specific ICD-9-CM
codes.12 A
patient's chart was abstracted if his or her computerized discharge
diagnoses contained at least 1 of the 77 inclusion ICD-9-CM codes
previously reported by Perkins et al.6 or 7 inclusion ICD-9-CM codes added
by the Emerging Infections Program (Table 1
). These 7 codes were determined to be possible indicators
of unexplained illness and death during a limited pilot study of the
retrospective validation carried out in 1 New Haven County hospital. The
conditions indicated by the 7 codes would have triggered further investigation in
the prospective surveillance; thus, addition of these codes did not bias the
retrospective validation.
|
). Medical records identified by computerized search were
reviewed by trained abstractors (Michael D. Kluger and Rajesh K. Sodhi) unaware
of the case findings of the prospective surveillance. Records were carefully
reviewed to determine whether patients fit all inclusion criteria and did not
exhibit any of the exclusionary conditions listed in Table 1 (i.e., whether they should have been identified by the
prospective surveillance). Abstracted data were recorded on a standardized
screening form. Those medical records in which the patient appeared to satisfy
inclusion criteria were referred to the study physician for final classification.
Incomplete, inconsistent, or questionable medical records were also referred for
final classification. Cases of unexplained illness and death identified through the prospective surveillance (n = 16) were combined with cases identified through the retrospective chart review to form a reference population against which the sensitivity of the 2 surveillance techniques could be measured. Because prospective surveillance involved both active (hospital A) and passive (hospitals B–G) surveillance techniques, sensitivity was assessed separately for each technique.
We calculated rates of unexplained illness and death by using the reference population in the numerator and the estimated total surveillance population of 584 507 in the denominator.10 In addition, we calculated incidence rates based on a capture–mark–recapture estimate of the total number of cases of unexplained illness and death in the numerator (n = 48).13 This allowed for a conservative estimate of the incidence of unexplained illness and death in the study county.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONEMERGING INFECTIONS
PROGRAMSMETHODSRESULTSDISCUSSIONReferences |
|---|
Upon chart review, 78 patients whose computerized discharge data satisfied all inclusion criteria were found to violate demographic and study-period inclusion criteria. Of these 78 patients, 34 had not been admitted to intensive care units during the surveillance period, 26 did not meet the age criteria, and 18 resided outside of New Haven County. These patients were excluded from the analysis. Another 186 patients who satisfied all computerized inclusion criteria were excluded on the basis of information abstracted from their medical records. Of these 186 patients, 93 had underlying conditions, 69 had illnesses with noninfectious causes, and 24 had illnesses in which a likely infectious agent was identified.
The 44 remaining patients were referred to the study physician
for final classification. Of these patients, 9 were classified by the study
physician as representing "definite" cases of unexplained illness and
death, and the remaining 35 were excluded because of underlying conditions (n =
19), identification of probable infectious agents (n = 11), or noninfectious
etiologies (n = 5). Overall, 97% (310 of 319) of the subjects who met the
inclusion criteria based solely on computerized administrative data were excluded
when their medical records were reviewed. Figure 1 illustrates the flow of case identification.
|
Sensitivities of the retrospective validation and prospective
surveillance were measured against a reference population composed of the total
cases identified through either method (n = 22). Overall, the retrospective
validation was 41% (9/22) sensitive, whereas the prospective surveillance was 73%
(16/22) sensitive (Table 2). The
retrospective validation conducted at hospital A, a major tertiary care
institution, was only 21% (3/14) sensitive. The active prospective surveillance
was 86% (12/14) sensitive at this hospital. The retrospective validation
performed at hospitals B through G was 75% (6/8) sensitive. The passive
prospective surveillance at these hospitals was 50% (4/8) sensitive.
|
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONEMERGING INFECTIONS
PROGRAMSMETHODSRESULTSDISCUSSIONReferences |
|---|
In the current investigation, cases of unexplained illness and death identified by prospective surveillance were combined with retrospectively identified cases to assess the sensitivity of the 2 surveillance techniques. Our findings demonstrate that, overall, prospective surveillance was more sensitive than retrospective surveillance (73% vs 41%). In other words, the majority of cases identified by the prospective surveillance would not have been identified had only retrospective surveillance relying on ICD-9-CM codes been used. The limitations of retrospective surveillance, despite the benefits, help to explain this difference in sensitivity.
Since its inception, the ICD-9-CM nosologic coding system has played a central role in clinical research and disease surveillance throughout the world. Assigned by hospitals to designate symptoms, diagnoses, and procedures and entered into administrative databases, ICD-9 coding has a number of advantages for retrospective surveillance. Most important, because administrative databases include virtually the entire patient universe, they potentially offer the best estimates of rare events.15,16 This is critical in investigations, similar to the present study, in which incidence is expected to be extremely low.6 However, administrative data sets are not fundamentally designed for research use; therefore, their sensitivity, specificity, and timeliness in terms of any given use may not be optimal. Review of patients' medical records is necessary if greater accuracy is desired.15,17
In a study of ischemic stroke, it was concluded that a retrospective review involving ICD-9-CM codes could be accomplished without examination of discharge summaries only if an error rate of 15% to 20% was deemed acceptable.17 In the current study, which included intricate inclusion and exclusion criteria for the retrospective validation, a much larger error rate was observed. This was largely a consequence of ICD-9-CM codes not being specifically designed to identify newly emerging infectious diseases, thereby leading to inclusion of inappropriate cases.6 Imprecise and poorly defined codes, multiple codes describing similar pathologic processes, and misleading conventions compound this problem.18 Even when ICD-9-CM codes are well defined, they may not be applied correctly by nosologists unfamiliar with the cases they are coding.6,15 These are probably the reasons that 61% of cases identified through the prospective surveillance were absent from the administrative databases obtained from the 7 participating hospitals for the retrospective validation.
The difference in sensitivity between the prospective surveillance and retrospective validation was also due in part to the greater sensitivity of the active vs passive prospective surveillance techniques: prospective surveillance was 86% sensitive when conducted actively at hospital A and only 50% sensitive when conducted passively at hospitals B through G. In addition, the retrospective validation was found to be 25% more sensitive than passive prospective surveillance at the 6 hospitals where only passive surveillance was used. In light of the many limitations of retrospective surveillance previously discussed, these findings further illustrate the shortcomings of passive prospective surveillance.
Despite the benefits of passive surveillance—chiefly, integration of the medical community in the recognition of unusual and potentially new infections, and the smaller resource requirements to operate the system—it is understandable that this technique was not as sensitive as the active prospective surveillance and retrospective validation. Unlike passive surveillance, in which reporting relies on individuals not closely involved with or dedicated to the surveillance project, active surveillance hinges on the efforts of individuals fully committed to identifying possible cases. Underreporting by infectious disease practitioners, nurses, and physicians may be a consequence of inconvenience or a result of these individuals' simply forgetting to report a rare event given the multiple responsibilities of their daily work.
For these reasons, among others, it is well recognized that even common communicable diseases that require mandatory reporting are underreported in passive surveillance systems, thereby affecting sensitivity.19 It can be expected that diseases not requiring mandatory reporting, as in the current investigation, will be reported even less often.
Active surveillance may have limitations as well, to the extent that case identification is based on information systems designed for clinical care rather than case detection. At hospital A, where active surveillance was conducted, cases were identified by reviewing weekday intensive care unit patient census reports that included patients' preliminary diagnoses. Opportunities for missed cases included preliminary or working diagnoses that did not fit the patient profile we were seeking and instances of very short or weekend intensive care unit stays that may have resulted in the exclusion of a case patient from the real-time census report.
In our study, the annualized incidence of unexplained illness and death in New Haven County ranged from 2.7 to 6.2 per 100 000 residents aged 1 to 49 years during the study period. This conservative range is well below the annual rate of 14.2 per 100 000 found by Perkins et al. in their review of 1992 National Center for Health Statistics multiple-cause-of-death data.6 The discrepancy can be largely explained by the fact that the Perkins et al. study was limited to a computerized search of death certificates based on ICD-9-CM inclusion and exclusion codes and did not include review of medical records of each potential case meeting the inclusion criteria. In our experience, more than 97% of cases from hospital discharge databases identified by ICD-9-CM code data are excluded upon abstraction and review of medical charts.
Active prospective surveillance proved to be the most sensitive technique used, followed by retrospective surveillance and passive prospective surveillance. The ICD-9-CM coding system and administrative databases contain many inherent problems that compromise the efficacy of surveillance systems based on their use, including inaccurate and incomplete coding and recording. Therefore, suspect medical charts must be abstracted and reviewed if retrospective surveillance is to be an accurate technique. Furthermore, retrospective surveillance is limited by timeliness, which may be critical to an investigation; in the current prospective unexplained illness and death surveillance, collection of clinical specimens and exposure information was integral to the project. Despite the fiscal costs of active prospective surveillance, this system most effectively meets the objective of identifying unexplained illness and death due to possibly infectious causes.
|
Acknowledgments |
|---|
We are indebted to the following individuals for their participation in both the prospective surveillance study and the retrospective validation efforts: Dr Louise Dembry, Dorothy Mazon, Lavern Jenkins, and members of the infection control staff at Yale–New Haven Hospital; Dr Howard Quentzel at Griffin Hospital; Ann Tudino at Milford Hospital; Diane Dumigan, Cathy Ligi, and Cindy Kohan at the Hospital of St. Raphael; Linda Brown and Dr Michael Simms at St. Mary's Hospital; Alice Stankus at Waterbury Hospital; and Kathryn Ross at the US Veterans Affairs Medical Center, West Haven.
We would also like to acknowledge Dr Michael Virata, Susan Smith, and the other staff of the Connecticut Emerging Infections Program for developing and maintaining the Unexplained Illness and Death Surveillance Program in Connecticut. Finally, we acknowledge Dr Bradley A. Perkins and Dr Rana Hajjeh for their contributions in coordinating the Unexplained Illness and Death Surveillance Program at the Centers for Disease Control and Prevention in Atlanta, Ga.
|
Footnotes |
|---|
Accepted for publication August 30, 2000.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONEMERGING INFECTIONS
PROGRAMSMETHODSRESULTSDISCUSSIONReferences |
|---|
2. Centers for Disease Control and Prevention. Human plague—India, 1994. MMWR Morb Mortal Wkly Rep.1994;43:689–691.[Medline]
3.
Campbell GL, Hughes JM. Plague in India: a new
warning from an old nemesis. Ann Intern Med.1995; 122:151–153.
4. Centers for Disease Control and Prevention. Outbreak of West Nile-like viral encephalitis—New York, 1999. MMWR Morb Mortal Wkly Rep.1999; 48:845–849.[Medline]
5. Centers for Disease Control and Prevention. Hantavirus pulmonary syndrome—United States, 1995 and 1996. MMWR Morb Mortal Wkly Rep.1996;45:291–295.[Medline]
6. Perkins BA, Flood JM, Danila R, et al. Unexplained death due to possibly infectious causes in the United States: defining the problem and designing surveillance and laboratory approaches. Emerg Infect Dis.1996; 2:47–53.[Medline]
7. Berkelman RL, Pinner RW, Hughes JM. Addressing emerging microbial threats in the United States. JAMA.1996; 275:315–317.[Medline]
8. Pinner RW. Addressing the challenges of emerging infectious diseases. Am J Med Sci.1996; 311:3–8.[Medline]
9. Manual of the International Statistical Classification of Diseases, Injuries, and Causes of Death. Vol. 1. Geneva, Switzerland: World Health Organization; 1977.
10. Intercensal Estimates of the Population of Counties by Age, Sex, and Race: 1970–1992 [machine-readable data file]. Washington, DC: US Bureau of the Census; 1995.
11. Sullivan KM, Gibbs NP, Knowles CM. Management of the surveillance system and quality control of data. In: Teutsch SM, Churchill RE, eds. Principles and Practice of Public Health Surveillance. New York, NY: Oxford University Press Inc; 1994:86–95.
12. Dean AG, Dean JA, Coulombier D, et al. Epi Info, Version 6: A Word-Processing, Database, and Statistics Program for Public Health on IBM-Compatible Microcomputers. Atlanta, Ga: Centers for Disease Control and Prevention; 1995.
13. Stroup DF. Special analytic issues. In: Teutsch SM, Churchill RE, eds. Principles and Practice of Public Health Surveillance. New York, NY: Oxford University Press Inc; 1994:136–149.
14. Goldacre MJ, Miller DL. Completeness of statutory notification for acute bacterial meningitis. BMJ.1976; 2:501–503.
15. Campos-Outcalt DE. Accuracy of ICD-9 codes in identifying reportable communicable diseases. Qual Assur.1990; 5:86–89.
16. Romano PS, Roos LL, Luft HS, Jollis JG, Doliszny K, Ischemic Heart Disease Patient Outcomes Research Team. A comparison of administrative versus clinical data: coronary artery bypass surgery as an example. J Clin Epidemiol.1994; 47:249–260.[Medline]
17. Cherkin DC, Deyo RA, Volinn E, Loeser JD. Use of the International Classification of Diseases (ICD-9-CM) to identify hospitalizations for mechanical low back problems in administrative databases. Spine.1992; 17:817–825.[Medline]
18.
Goldstein LB. Accuracy of ICD-9-CM coding
for the identification of patients with acute ischemic stroke: effect of modifier
codes. Stroke.1998; 29:1602–1604.
19. Iezzoni LI. Using administrative diagnostic data to assess the quality of hospital care. Int Technol Assess Health Care.1990; 6:272–281.
This article has been cited by other articles:
|
|
 |
|
  J.-E. C. Holty, D. M. Bravata, H. Liu, R. A. Olshen, K. M. McDonald, and D. K. Owens Systematic review: a century of inhalational anthrax cases from 1900 to 2005. Ann Intern Med, February 21, 2006; 144(4): 270 - 280. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
