| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
RESEARCH AND PRACTICE |
David J. Dausey is with the RAND Corporation and the Heinz School of Public Policy and Management, Carnegie Mellon University, Pittsburgh, PA. Anita Chandra, Ben Bahney, and Sarah Zakowski are with the RAND Corporation, Arlington, VA. Agnes G. Schaefer, Amelia Haviland, and Nicole Lurie are with the RAND Corporation, Pittsburgh.
Correspondence: Requests for reprints should be sent to David J. Dausey, RAND Corp, 4570 Fifth Ave, Pittsburgh, PA 15213 (e-mail: dausey{at}rand.org).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
Objectives. We tested telephone-based disease surveillance systems in local health departments to identify system characteristics associated with consistent and timely responses to urgent case reports.
Methods. We identified a stratified random sample of 74 health departments and conducted a series of unannounced tests of their telephone-based surveillance systems. We used regression analyses to identify system characteristics that predicted fast connection with an action officer (an appropriate public health professional).
Results. Optimal performance in consistently connecting callers with an action officer in 30 minutes or less was achieved by 31% of participating health departments. Reaching a live person upon dialing, regardless of who that person was, was the strongest predictor of optimal performance both in being connected with an action officer and in consistency of connection times.
Conclusions. Health departments can achieve optimal performance in consistently connecting a caller with an action officer in 30 minutes or less and may improve performance by using a telephone-based disease surveillance system in which the phone is answered by a live person at all times.
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
Assessing information about threats to public health, including those caused by infectious disease, and ensuring that adequate services are provided to meet these threats are core functions of health departments.6 The telephone-based disease surveillance (TBDS) systems that local health departments have in place to receive case reports from the field are among the first lines of defense in identifying these threats. For more than 20 years, the Centers for Disease Control and Prevention (CDC) has provided health departments with guidelines to evaluate the performance of their TBDS systems.7,8
In 2003, the CDC expanded its guidelines and developed performance standards to evaluate the ability of health departments to receive urgent case reports 24 hours a day, 7 days a week9. These standards, although not binding performance obligations, emphasized the need for TBDS systems to consistently receive urgent case reports in a timely manner. The CDC encouraged health departments to regularly test their TBDS systems to assess their compliance with these standards because of concerns regarding the reliability of self-assessments not based on test results. The standards, however, were not, accompanied by guidance on how health departments should measure their performance, and it was unclear at the time whether the goals were achievable.
To address this gap, Dausey et al. developed a method to assess whether local health departments could meet these standards and pilot-tested it in a convenience sample of 19 health departments.10 The pilot tests found dramatic variations both in the response capabilities of TBDS systems and in their structure.11 These findings suggested that there may be certain types of TBDS systems that perform better than others. In addition, these findings raised questions about whether TBDS systems could consistently achieve optimal performance as outlined by the CDC and whether quality improvement in these systems was possible.
No research has described how health departments might improve their performance in receiving and responding to urgent case reports or which components of TBDS systems contribute to better performance. Literature exists on telephone response systems in other sectors that operate 24 hours a day, 7 days a week, ranging from emergency medicine to environmental hazard control. For example, literature exists on the effectiveness of the emergency response infrastructure in these areas,12–15 as well as on the evaluation of emergency response in the field of emergency management.16,17 Factors found to be associated with successful response in other sectors include structuring the system so that callers reach a live person, using a single telephone number instead of multiple numbers, building redundancies into the system in case of failure, requiring telephone operators to go through extensive training, and using formal protocols for call triage.
We sought to identify the characteristics of TBDS systems associated with the ability of health departments to meet the CDC’s standard requiring that all urgent case reports be connected to a trained public health professional in 30 minutes or less. We tested the TBDS systems of a random sample of 74 local health departments from across the United States.
|
METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
1145 000)—such that 25% of the US population was served by health departments in each size category. We selected 100 health departments for our sample by simple random sampling within each stratum. An equal number of health departments were selected across the 4 population-size categories (n = 25), with the number selected in each region varying proportionally to the population of the region. In the resulting sample, each selected health department represented health departments covering an equal fraction (1/100th) of the population of interest; in the largest population-size category, a selected health department could only represent itself; in the smallest category, a selected health department might represent as many as 60 other health departments. We replaced those health departments that we could not contact after 4 attempts or that declined to participate with another randomly selected health department from the same region-size stratum.
Data Collection
We adapted our test-call method from Dausey et al.10 Prior to test calling, we obtained consent from health department directors and conducted a short interview to obtain information on their TBDS systems. We asked the consenting health department officers not to share the details of the project with their staff to ensure that test calls were not anticipated and that we could assess actual or realistic call connection time.
To assess whether health departments could connect medical personnel to an action officer—defined as a public health professional such as a public health physician, nurse, or epidemiologist—a trained test caller contacted participating health departments, asserting that he or she was a doctor or nurse at a local health care facility calling with an urgent case report regarding an infectious disease. Sample caller scripts can be found elsewhere.10 Callers were instructed to respond to inquiries about cases (prior to reaching an action officer) by saying that the case was confidential and that specific case information could only be provided to the action officer. If a department responded in more than 30 minutes to either all or none of its first 5 test calls, it received no more calls, because statistical calculations from previous research revealed a low probability that additional calls would yield different results (the probability that the sixth test call result would be different was approximately P = .003).11 All other departments received 10 test calls. Calls were placed both during business hours (Monday to Friday, 8 AM to 5 PM, local time) and after hours (all other times) from May to October 2006.
Measures
For each test call, we recorded measures on the characteristics of the TBDS system that we identified through our literature review and previous work, as well as the TBDS system components currently required by CDC standards. These measures included recording whether the caller initially reached a live person or an automated system, whether the caller had to hang up and dial a second number before reaching an action officer, and whether the caller was transferred to the action officer with a warm transfer (i.e., immediately transferred to an action officer) or required a callback (i.e., left a number to be called back). We also identified whether the phone number called was the department’s general number, a general communicable disease line, or a dedicated all hours health department line for urgent case reports. In instances in which departments had multiple numbers, we placed calls to all of them.
Our measures of system performance were developed to capture the event of a connection to an action officer and speed of connection to an action officer for each call and to provide a benchmark system for evaluating consistency of health department connection times. These measures included whether the caller was connected to an action officer in 30, 60, or 240 minutes or less or not connected at all. We aggregated these call-level measures, such as the average time to call connection, to the health department level. From the call-level findings, we categorized health departments as excellent if all calls were connected in 30 minutes or less (the CDC standard), fair if 1 or more calls took more than 30 minutes but none took more than 240 minutes, and poor if 1 or more calls took more than 240 minutes or was not connected at all.
At the conclusion of the test calls, we interviewed the health directors of 5 health departments that answered all calls in 30 minutes or less to obtain their perspectives on what may have contributed to the optimal performance of their TBDS systems.
Data Analysis
We placed 596 calls to 74 health departments. We began our analysis by comparing the percentage of calls connected with an action officer in 30 minutes or less at each health department to the percentage predicted by the department director before testing. We analyzed the determinants of call connection at both the health department and individual call level. We used sampling weights to calculate descriptive statistics and to report the data in a nationally representative manner. Our analyses at the health department level allowed us to determine which factors contributed to a fast connection with an action officer (call connection) and which factors were associated with the ability of a health department to consistently connect the test caller with an action officer from call to call (call consistency). The individual call–level analysis allowed us to further examine which call-level factors predicted the amount of time to reach an action officer and the probability of ever reaching an action officer in addition to investigating whether these factors differed between the business day and after hours.
To assess call connection, we modeled the mean time to connect to an action officer at the health department level and the associations between connecting with an action officer in 30 minutes or less or not connecting with an action officer at all, and TBDS system variables at the call level. We used linear ordinary least squares regression for the health department model and logistic regression with random effects specified at the health department level for the call-level models. For the call-level outcome models, we also analyzed our data separately by time of call initiation (business hours vs after hours). The call-level models used the full set of test calls in the sample; health departments whose performance was inconsistent in the first 5 calls had more calls, providing more variability in the outcome within health departments.
To assess call consistency at the health department level, we used a multinomial logit regression. We tested determinants of excellent consistency (connection in 30 minutes or less for all test calls), fair consistency (1 or more calls taking 31–240 minutes), and poor consistency (1 or more calls in which the action officer was reached in more than 240 minutes or was never reached).
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
Before conducting our test calls, we asked health department directors to predict the percentage of calls in which they thought our test caller would connect with an action officer in 30 minutes or less. Figure 1
shows the percentage of directors who accurately predicted, overestimated, or underestimated their actual call connection according to our test call data and the percentage of directors who stated that they did not have enough information to predict their performance. Forty-seven percent of department directors (n = 35) predicted that their performance would be substantially better than it actually was, 24% (n = 13) predicted their performance fairly accurately, and 13% (n = 14) underestimated their performance.
|
).
|
). We found that for each 10% increase in the number of calls fielded by a live person (instead of a machine), there was an average reduction of approximately 37 minutes (P < .01) in response time. For example, health departments that fielded 20% more of their test calls with a live person had a shorter average connection time of approximately 74 minutes (P < .01). The timeliness of call connection with an action officer did not differ between systems that required callers to call more than 1 telephone number before reaching an action officer and those that were automated. After preliminary bivariate analyses, we included the 2 largest population-size categories in the model to compare them with the 2 smallest population-size categories, but this attribute was also not significantly associated with average connection time.
|
) and significantly decreased the probability that a health department would be categorized as having poor consistency. Health departments that had 1 more test call out of 10 that connected first with a live person had a 43% reduction in the likelihood of being rated as having poor consistency (P < .01).
Call Connection at the Call Level
Table 3 presents information on factors contributing to connection time at the call level overall and separately by calls made during business hours and after hours. For all calls, having a person answer the phone increased the odds of having a call connected in 30 minutes or less and having a call connected at all (both results, P < .01). For calls placed during business hours (n = 391), we again found that systems using a live person to answer the phone had a faster connection with an action officer (P < .01).
|
In these models, we included both call-level independent variables and department-level random effects, which allowed us to estimate the degree to which the outcomes varied within health departments compared with between health departments (as indicated by the rho values in Table 3, measuring intercluster correlation). We found little of the variability in timely call response during business hours to be between health departments; rather, much of the variability in this outcome was within departments. By contrast, for calls made after hours, nearly half the variation for both outcomes was between health departments, suggesting much more similar outcomes within departments. Thus, at all hours, a large component of call connection success was determined at the department level; the department determined a large component of call timeliness only after hours.
Brief Interviews With Health Department Directors
In brief follow-up interviews conducted at departments that had optimal performance (n = 5), the department directors described several factors that they felt may have contributed to their departments’ success. All directors indicated having high performance expectations for their staffs. Four of the 5 interviewees reported that they regularly tested their system, provided feedback to their staff members, and updated call lists (which included 1 or more backup person for occasions when the primary action officer could not be reached). One health department employed an automated call system activated by the person answering the phone. This system sequentially called down a list until a live person was reached.
Four health department directors indicated that they were stimulated to improve their telephone response systems because of performance expectations and measures set forth by their states; 1 of these indicated that his budget was, in part, contingent on performance, along with a series of other measures. One director stated that the stimulus came from a survey of community physicians, which indicated dissatisfaction with the responsiveness of the health department.
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
In our quantitative analyses, 1 key factor—whether callers reached a live person when they called, regardless of who that person was—was a strong predictor of optimal performance, both for time to reach an action officer and consistency in doing so. Participating health departments used a variety of mechanisms to ensure that a live person would answer the phone, including hiring an answering service or forwarding calls to another local entity, such as the sheriff’s dispatch or a local poison control center. The importance of reaching a live person was found to be particularly strong in protecting against poor performance. This indicates that it may be particularly helpful for departments with a high prevalence of slow connections or calls not returned to devote their resources to updating their TBDS systems to direct callers to live respondents.
Although the CDC has focused attention on the presence of a single all hours line for urgent case reports, this feature of a health department’s TBDS system was less critical to optimal performance than the ability to directly connect with a live person. However, having a live person answer the phone did not guarantee perfect performance, suggesting that other unmeasured attributes of system performance may also have played an important role. Qualitative interviews suggested that several other factors, including practice, routine performance measurement compared with standards, department leadership, and clear expectations for performance may have played a role in the ability of some health departments to achieve a high level of performance.
Our results contribute to existing knowledge in several ways. First, they documented performance related to the ability to connect a caller with an action officer in a representative sample of health departments. They also identified a consistent and readily modifiable factor—whether a live person answered the phone—that was associated with optimal performance. Finally, our data indicated that perfect performance was achievable. This finding was complemented by qualitative interviews that indicated that improvement was possible. Indeed, all directors from optimally performing health departments with whom we spoke were able to point to a time when performance was not optimal and to identify a set of changes they made that led to improved performance.
Limitations
To our knowledge, this study is the first to systematically investigate the types of phone systems employed by health departments and their effect on call response. However, these findings should be interpreted in light of the study’s limitations. First, although the sample was drawn to represent health departments throughout the United States (excluding those that served less than 7200 people), the small sample size limited our statistical power to identify other potential success factors. Second, although we based our selection of predictor variables on previous work, the literature, and case studies, there may be other relevant predictors that we did not measure.
Third, we did not assess the nature and quality of the response by the action officer; we are not aware of any existing guidelines or standards for what an action officer should say in response to an urgent case report. Next, it is possible that callers in some health departments were anticipating the test calls and acted accordingly. We doubt this was the case, however, because our early developmental work on testing procedures suggested that most respondents were surprised by the test. Finally, this study did not examine the role of state health departments, which have a contractual obligation to meet CDC standards. Although local health departments receive federal funding that is passed through state health departments, they are not technically required to meet an all hours standard to receive funds.
Conclusions
Our findings suggest that many health departments have not yet begun to regularly test their TBDS systems despite CDC recommendations. The poor correlation between health department director expectations of performance and actual performance highlights the need for objective measurement. This study, however, did not clearly indicate by which process this testing should be done. Nonetheless, some options include having the CDC test state and local health departments, having state public health departments test local health departments in their jurisdiction, and having local departments conduct self-evaluations.
Our findings also suggest that it may be prudent to revisit some aspects of the current CDC recommendations regarding TBDS systems. For example, we did not find any significant difference between health departments that had a separate dedicated all hours telephone line to receive urgent case reports and those that did not. It is also not clear what is an acceptable length of time for a caller to reach an action officer. The CDC set a standard of 30 minutes and has considered changing this standard to 15 minutes. Many health departments maintain that either standard is unrealistic; even if the bar were set at 60 minutes, a significant number of health departments did not meet this standard.
The field of public health is moving toward performance measurement, accountability, and quality improvement. This study provides an example of objective performance measurement and suggests methods of quality improvement. Improving the performance of TBDS systems is clearly amenable to classical quality improvement approaches, which stress the use of multiple small-cycle tests of change and improvement, followed by regular assessments of performance to ensure that the improvements are maintained and goals are reached. Developing and improving measurement of other core local health department processes and functions will likely be necessary to achieve improvements.
|
Acknowledgments |
|---|
We would like to thank all of the health departments that participated in the project.
Note. The views expressed are solely those of the authors and do not necessarily reflect those of HHS.
Human Participant Protection
This study was approved by the RAND Corporation’s human subjects protection committee.
|
Footnotes |
|---|
Contributors
D. J. Dausey originated the study, led the writing of the article, and supervised all aspects of the study implementation. A. Chandra synthesized analyses and led the writing of the methods and results. A. G. Schaefer assisted with the study and writing of the article. B. Bahney, A. Haviland, and S. Zakowski assisted with the study and completed the analyses. N. Lurie was the senior adviser for the project, conducted the brief interviews, and assisted in managing the study and writing the article.
Accepted for publication June 11, 2007.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONReferences |
|---|
2. Beitsch LM, Thielen L, Mays G, et al. The multistate learning collaborative, states as laboratories: informing the national public health accreditation dialogue. J Public Health Manag Pract.2006;12:217–231.[Web of Science][Medline]
3. Nelson C, Lurie N, Wasserman J. Assessing public health emergency preparedness: concepts, tools, and challenges. Annu Rev Public Health.2007;28:1–18.[CrossRef][Web of Science][Medline]
4. Tilson H, Berkowitz B. The public health enterprise: examining our twenty-first-century policy challenges. Health Aff.2006;25:900–910.
5. Jajosky RA, Groseclose SL. Evaluation of reporting timeliness of public health surveillance systems for infectious diseases. BMC Public Health.2004;26:4–29.
6. Institute of Medicine. The Future of Public Health. Washington, DC: National Academy Press; 1988.
7. Centers for Disease Control and Prevention. Guidelines for Evaluating Surveillance Systems. MMWR Morb Mortal Wkly Rep.1988;37(suppl 5):1–18.[Medline]
8. Centers for Disease Control and Prevention. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. July 27, 2001. Available at: http://www.cdc.gov/mmwr/preview/mmwrhtml/rr5013a1.htm. Accessed October 21, 2007.
9. Centers for Disease Control and Prevention. Improving surveillance infrastructure for terrorism detection: the 8-cities project resource materials. Available at: http://www.cdc.gov/epo/dphsi/8city.htm. Accessed March 11, 2007.
10. Dausey DJ, Lurie N, Diamond A, et al. Tests to Evaluate Public Health Disease Reporting Systems in Local Public Health Agencies. Santa Monica, Calif: RAND Corp; 2005.
11. Dausey DJ, Lurie N, Diamond A. Public health response to urgent case reports. Health Aff (Millwood).2005;W5:412–419.
12. Blackwell T, Kaufman J. Response time effectiveness: comparison of response time and survival in an urban emergency medical services system. Acad Emerg Med.2002;9:288–295.[CrossRef][Web of Science][Medline]
13. Campbell J, Gridley T, Muelleman R. Measuring response intervals in a system with a 911 primary and an emergency medical services secondary public safety answering point. Ann Emerg Med.1997;29:492–496.[CrossRef][Web of Science][Medline]
14. Davis S. Virtual emergency operations. Risk Manage.2002;49:46–52.[Medline]
15. Hale J. A layered communications architecture for the support of crisis response. J Manage Inf Syst.1997; 14:235–255.
16. Jain S, McLean C. Simulation for emergency response: a framework for modeling and simulation for emergency response. In: Chick SE, Sánchez PJ, eds. Proceedings of the 35th Conference on Winter Simulation: Driving Innovation, New Orleans, La, December 7–10, 2003 [CD-ROM]. Omnipress; 2003.
17. Mrvos R, Krenelok KP. Realtime testing of a regional poison information center’s disaster plan. J Homeland Secur Emerg Manage.2005;2:1–3.
18. Who’s Who in Local Public Health. Washington, DC: National Association of County and City Health Officials; 2005.
19. Lurie N, Wasserman J, Nelson CD. Public health preparedness: evolution or revolution? Health Aff (Millwood).2006;25:935–945.
20. Lurie N, Wasserman J, Stoto M, et al. Local variation in public health preparedness: lessons from California. Health Aff (Millwood).2004;W4:341–353.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
