| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
RESEARCH AND PRACTICE |
Barbara C. Lee and Richard L. Berg are with Marshfield Clinic Research Foundation, Marshfield, Wisc. James D. Westaby is with Teachers College, Columbia University, New York.
Correspondence: Requests for reprints should be sent to Barbara C. Lee, PhD, RN, Marshfield Clinic Research Foundation, 1000 North Oak Avenue, Marshfield, WI 54449 (e-mail: lee. barbara{at}mcrf.mfldclin.edu).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMETHODSRESULTSDISCUSSIONEPILOGUEReferences |
|---|
Objectives. We conducted a comprehensive evaluation of a rural youth health and safety initiative implemented in 4000 National FFA (formerly Future Farmers of America) chapters across the United States.
Methods. Data were collected from high school students and their FFA advisers at 3 time intervals (preintervention, immediate postintervention, and 1 year postintervention) with a 3-group (standard, enhanced, and control), cluster-randomized, controlled trial design.
Results. Matched data from 3081 students and 81 advisers revealed no significant effect of this initiative on agricultural health and safety knowledge, safety attitudes, leadership, self-concept, and self-reported injuries of project participants. Data from 30 public health nurses following the intervention confirmed the program’s failure to develop sustainable community partnerships.
Conclusions. This nationally coordinated initiative was funded with more than $1 million donated by agribusinesses. Program implementation was inconsistent, and desired outcomes were not achieved. Future efforts should better guide effective use of private sector resources aimed at reducing agricultural disease and injury among rural youths.
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONEPILOGUEReferences |
|---|
FFA’s desire to undertake this initiative was driven by its leaders, who noted that the farm press and rural communities were increasingly more aware of traumatic childhood agricultural injuries and hazardous work conditions, starting with a lengthy, 1989 feature story, "We Kill Too Many Farm Kids," published in a national farm magazine.3 By 1996, the National Action Plan for Childhood Agricultural Injury Prevention was developed and adopted by the US Congress as a blueprint for a public health response to this problem.4 FFA adopted this cause, intending to promote agricultural health and safety with financial support from major tractor manufacturers, chemical producers, and seed dealers. The marketing value of this initiative was obvious to corporate donors that would be acknowledged in promotional materials and educational resources. Yet there were questions regarding its effect on the knowledge, attitudes, and practices of FFA members. The question was raised: is the Partners Program an effective use of a $1 million private sector investment aimed toward adolescent agricultural disease and injury prevention?
The stated goals of the FFA’s Partners Program were to promote students’ agricultural health and safety knowledge and attitudes, increase students’ leadership skills, and develop sustainable community partnerships that promote health and safety. The Partners Program involved national and state-level training of FFA advisers who used a modified version of the Teaching Agricultural Safety to Kids curriculum, which trains youths to become junior teachers, mentors, and role models for young children.5 All participating FFA chapters were given a multivolume educational kit of videotapes, CD-ROMS, printed resources, and instructions (Table 1
). Ongoing communications, quarterly newsletters, and encouragement to already trained FFA advisers were provided by the Partners Program director. Across the United States, about 4000 FFA chapters were enrolled in this highly marketed initiative. The National FFA hired a program evaluator to monitor activities and assess state trainers’ perspectives during the start-up phase. No mechanisms were employed to ensure that the program was implemented according to the detailed protocol.
|
|
METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONEPILOGUEReferences |
|---|
depicts interventions for each group.
Study Population
Our intent was to sample FFA chapters that were representative of sites where agricultural health and safety issues were relevant. States with the highest number of farms were identified from the US Census of Agriculture.12 Inclusion criteria included FFA chapters located in cities of < 50 000 people that had more than 10 female and male chapter members in the ninth and tenth grades. Within each FFA chapter, the adviser enrolled agricultural students in the ninth, tenth, and eleventh grades, allowing for follow-up at the conclusion of their subsequent academic year. Given turnover among FFA advisers, along with changing patterns of courses as class schedules changed by semester and year, modest dropout rates were anticipated (students were not necessarily enrolled in subsequent classes when data were collected).
Instruments
Four data collection instruments were developed for this study: a student instrument for measuring program effect on attitudinal, knowledge, and behavioral domains; an FFA adviser instrument for measuring program implementation (process) and effect; an FFA chapter activity report form for quantifying program activities; and a community nurse instrument for assessing program integration. The student instrument required the most extensive testing and modification; the adviser form was developed with many of the same subscales. All scales were pilot tested with factor analyses and measures of internal consistency, allowing for elimination of items that poorly reflected constructs of interest.13
The student and adviser assessment instrument was administered in its entirety 3 times: at the beginning of academic year 1 (preintervention), at the end of academic year 1 (postintervention), and at the end of academic year 2 (postintervention 2). Unless stated otherwise, respondents rated scale items on a 5-point scale ranging from strongly disagree (1) to strongly agree (5). Self-esteem was assessed with items from the Rosenberg scale (Cronbach 
= .83).14 Leadership self-concept was assessed by 9 items representing a person’s perceived leadership level (Cronbach = .88). Safety consciousness was assessed by 7 items tapping general attitudes toward acting safely, such as, "I always take extra time to do things safely" (= .83). A risk-taking scale was included with 5 items (e.g., "I’d rather take risks than be overly cautious") with a Cronbach of .73. A safety knowledge scale was based on self-reported learning across several areas relevant to agricultural safety (= .82). Safety campaign participation had 7 items assessing participation in various activities that promote safety (= .82). Self-reported injuries were assessed, with 2 items measuring the extent to which the participant experienced injuries within the last 3 months ( = .71). Results from confirmatory factor analyses demonstrated that the above concepts represented construct valid latent variables.13 Demographic information was also gathered.
The adviser instrument comprised the same subscales as used for students, minus the safety campaign scale. This instrument assessed advisers’ guidance in helping youths get involved in agricultural safety and health activities. For the final collection of data, advisers in the standard and enhanced groups were asked about their perspectives on the implementation and effect of the Partners Program.
To assess fidelity to program implementation guidelines, a 1-page "Relevant Chapter Activities" query was developed, addressing implementation recommendations. The form was updated monthly and submitted to FFA chapters in the standard and enhanced groups, with a request to complete and return it within 2 weeks. Responses diminished over time, despite multiple telephone reminders. After 5 months, evaluation data regarding implementation of the Partners Program were no longer collected because of inadequate response rates.
A 12-item telephone interview tool was used with community nurses working with the enhanced intervention FFA chapters. The purpose was to assess the extent to which new partnerships between FFA chapters and community health leaders were addressing local agricultural health and safety concerns (a stated goal of the Partners Program).
Matched Data
To optimize the likelihood of honest responses from students, the anonymity of respondents was protected. A self-generated, anonymous identification code was used to allow investigators to match responses from individuals at 3 points in time.15 The self-generated identification code, located in the final section of the instrument, asked respondents for the following information: first 2 letters of US state of birth, first letter of month of birth, first 2 letters of mother’s first name, and number of older brothers. A computer algorithm was developed to match the identification codes of study participants at times 1, 2, and 3. Data from the subset of matched cases formed the basis for evaluating the effect of the Partners Program.
Data Collection
Data were collected through a detailed system with written protocols for FFA chapter advisers. Investigators sent the advisers an appropriate number of student survey forms and 1 adviser form, with preaddressed return labels for boxed materials. Telephone and fax contacts were made if completed forms were not returned within 1 month.
Data entry was conducted by trained staff, who used a glossary of problem resolutions to deal consistently with aberrant responses. Forms that had < 50% responses complete or that were not handled completely seriously were discarded ( < 1% of the surveys processed). Quality assurance for all data entry was conducted for each of the 3 time periods with > 99.7% accuracy.
For measuring community partnership activity, trained telephone interviewers contacted the designated public health nurse 6 months after intervention interactions between the nurses and FFA chapters were likely to have occurred. Following the structured interview, nurses provided general comments regarding the project.
In addition to methods described here, the National FFA hired an evaluator to monitor the pilot testing and nationwide implementation of the Partners Program. Reports from this evaluator provided additional documentation to guide FFA’s decisions regarding program maintenance.
Statistical Methods
To assess the effect of the Partners Program, we measured specific constructs, using the same measurement tool at 3 points in time. To demonstrate support for the initiatives, we expected a statistically significant difference between the standard and enhanced treatment groups relative to respondents in the control group. Further, we expected the differences between the enhanced and the control group to be greater than between the standard and the control group, given the increased resources devoted to the enhanced manipulations.
To statistically examine the efficacy of experimental conditions to the control condition, each dependent variable was examined with analysis of variance (ANOVA) and analysis of covariance (ANCOVA); the latter also was used to control for demographic variables (i.e., gender, age, working status, rural residences, and farm residence) and pretest scale scores. Both analyses yielded the same conclusions in this study concerning comparisons across standard, enhanced, and control conditions. In these analyses, each dependent variable was represented by its scale score (i.e., average composite across items, none of which deviated sufficiently from normal distributions to preclude the use of parametric analyses). The between-subjects factor was represented by 1 of 3 levels, corresponding to the standard, enhanced, and control experimental conditions, respectively. Analyses were conducted at both the individual and aggregate cluster level, which accounts for important group-level effects.15 Because both analyses yielded exactly the same conclusions, and because no significant cluster effects were manifest (P > .10), individual-level data are reported below, giving greater precision in mean point estimates across the youth participants.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONEPILOGUEReferences |
|---|
Preintervention data were collected from 8068 students, and of these, 68% were male, 42% were farm residents, 83% were active FFA members, and 84% were aged 17 years or younger. Postintervention data were collected from 5925 students in 98 FFA chapters. Because of class enrollment and student attrition, only 3081 were matched with the same individual who provided preintervention data. This data set formed the basis for the assessment of the immediate effect of the Partners Program. One year later (postintervention 2), data were collected from 6604 students, with 1167 being matched to the same individual responding 2 previous times. Ultimately, the program’s effect could be assessed by these 1167 students, along with 60 advisers whose data were matched at 3 time intervals (Figure 1). There were no significant differences among the students or advisers in the standard, enhanced, or control groups relative to gender, residency on or off farm, or hours of work. We report the findings from the preintervention and first postintervention data (n = 3081 students) to reflect the immediate program effect on the students whose characteristics are depicted in Table 2. Results from the second postintervention data collection were the same as for the first postintervention findings reported in this article. ANCOVA results are presented in Table 2, given their convergence with ANOVA results for the between-subjects factor.
|
|
). This was true for both ANOVA and ANCOVA analyses.
|
Safety consciousness. Comparative findings indicated that safety consciousness did not differ significantly across groups, according to both analysis of covariance and analysis of variance tests (overall mean = 3.0; SD = 0.73). However, safety consciousness was a relatively strong negative correlate of self-reported injury (r= –0.15; P< .001).
Self-reported leadership. There was no significant difference between the groups on self-reported leadership (overall mean = 3.3; SD = 0.81), and across conditions, youths with strong leadership self concepts were more likely to be safety conscious (r = 0.28; P < .001).
Dangerous risk taking. Results following the intervention revealed no significant difference between the 3 groups on participants’ dangerous risk-taking tendencies (overall mean = 3.2; SD = 0.74). Across all conditions, dangerous risk taking was correlated with gender (i.e., males were more dangerous risk takers; r= 0.12; P< .001) and self-reported injury (r= 0.18, P< .001).
Self-esteem. Self-esteem did not differ significantly over time as a function of group assignment (overall mean = 3.8; SD = 0.76). Across conditions, self-esteem was significantly associated with safety consciousness (r= 0.22; P< .001).
Safety campaign participation. Results indicated no significant difference in safety campaign participation between the groups. The overall participation level in safety activities was relatively low (overall mean = 2.1; SD = .91). Those students participating in safety campaign activities primarily did so through 3 mechanisms: helping teach young people, planning and working at a community event, or participating in a health promotion activity. Safety campaign participation was also strongly correlated with safety knowledge (r= 0.34; P< .001) and safety consciousness (r= 0.34; P< .001) across groups.
Injury experiences. About one fourth of respondents reported they had been "injured in an accident" in the last 3 months. However, there were no significant differences between the 3 groups on self-reported injury (overall mean = 1.7; SD = 0.38). Injury was most strongly related to safety consciousness and dangerous risk taking across conditions both before and after the intervention.
In summary, all the analyses testing the standard, enhanced, or control groups (at both the individual and aggregate level both with and without additional control variables) revealed no substantive effect of the intervention conditions on participants’ self-reported knowledge, attitudes, activities, leadership, community participation, or injury.
FFA Adviser Outcomes
With each time period, the FFA adviser completed an assessment instrument similar to that of the students. Because of chapter attrition and employment changes, matched adviser responses were not secured from all participating chapters. Preintervention responses were secured from 108 FFA advisers, followed by 90 responses immediately after the intervention. Of these, 81 matches were confirmed. One year later, we received 85 FFA adviser responses, providing 60 FFA advisers with matched identification codes over 3 points in time.
Data revealed minor differences between the 3 treatment groups. The FFA advisers in the enhanced intervention group reported a statistically significant (P= .045) increase in the number of safety campaign activities over both the standard and control groups. However, this finding is not consistent with the students’ survey responses on the same factors (at both the individual and aggregate level). Another unexplainable finding was that following the intervention, FFA advisers in the control group reported significantly higher (P= .037) self-esteem over both the standard and enhanced intervention group advisers. No other significant differences were manifest on criterion variables.
When asked about perspectives on the National FFA’s Partners Program, most FFA advisers qualitatively indicated their support. However, time constraints limited their ability to implement the program as it was designed.
Community Nurse Outcomes
All but 2 of the original 35 public health agencies participated in the follow-up interviews, even if it was to report "nothing occurred." Of the 30 nurses who provided data (1 nurse covered 3 different FFA chapters), 37% indicated they had no contact with the FFA chapter. Of those nurses who had contact, 57% spent 1–5 hours with student activities, and 5% spent from 6 to 10 hours with student activities, typically giving a class lecture on a health topic. The majority of nurses (67%) deemed the Partners Program to be valuable. Regarding the $300 incentive for the public health agency, 33% of nurse respondents indicated it was "not at all important." For the most part, funds were put into the general operating account or used to purchase resources for the FFA chapter.
Implementation
Efforts of the National FFA’s program evaluator to assess the nationwide start-up phase of the Partners Program were severely hampered by low response rates to mail surveys and telephone interviews. Of survey respondents, less than half indicated they intended to implement the program. Among those starting the program, many reported adopting only selected aspects of the designed program (Sharon Dorfman, ScM, CHES; President of SPECTRA, Damond Beach, Fla; oral communication; February 2004). Our own efforts to collect monthly implementation data were terminated midway through the study because responses on a single-page questionnaire dropped to < 20%, despite repeated telephone reminders.
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONEPILOGUEReferences |
|---|
Our study also assessed the effect of the Partners Program on building sustainable community partnerships. Although the FFA advisers in the enhanced intervention group reported an increased involvement in community-based safety campaign activities, results from students and community nurses did not support the advisers’ data. Furthermore, no other differences were manifest.
Ideally, an education initiative would undergo testing for efficacy before being rolled out on a national scale. Our study findings reflect outcomes of a program that was untested for efficacy and implemented inconsistently across sites. We cannot state unequivocally that the program was a failure, but our results show that its effect as actually implemented was negligible.
Educational initiatives often promise greater results than those that can actually be achieved. Study results led to the conclusion that the National FFA’s Partners Program initiative, with more than $1 million in corporate support, failed to achieve the expected outcomes. Although local FFA chapter advisers appreciated receiving the Partners Program curriculum with its extensive resources and training, only a few FFA advisers had time and interest sufficient to implement the full program as prescribed. In addition, although community nurses expressed support for the Partners Program, the money and time invested in building a community partnership between FFA chapters and nurses yielded minimal results. Program sponsors, such as the agribusinesses supporting the Partners Program, would benefit from feedback on evaluation studies so that they can more effectively direct future contributions.
These results are consistent with other educational approaches for agricultural health and safety, and by indicating that educational programs alone are insufficient to prompt desired changes in agricultural health and safety, they contribute to the body of knowl-edge.17–20 Systematic reviews of health promotion and injury control strategies have revealed mixed results.21–24 A recent review of 35 successful prevention programs highlighted key characteristics of program content matched to target populations.25 The addition of some of these characteristics might have improved the outcomes of the Partners Program.
Low-cost evaluation methodologies that provide dependable results are limited, and the ideal randomized-controlled trial design can be challenging because of its inherent difficulties, including logistics, budgets, and politics.26 Although our evaluation study used a cluster-randomized–controlled trial design, we were evaluating an intervention that was designed by national program leaders and implemented at the local level beyond the control of its originators. In the absence of controlled study results, decisions regarding future initiatives should consider factors such as plausibility, adequacy, and maintenance.27–29
FFA continues to be a leading youth-serving organization in the United States, and efforts to determine the most effective methods for promoting agricultural health and safety to its young rural members should be made.
|
EPILOGUE |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONEPILOGUEReferences |
|---|
|
Acknowledgments |
|---|
We thank Bernie Staller, CEO of the National FFA, for allowing us to impose evaluation research activities on participants in their Partners for a Safer Community initiative. Gratitude is extended to the FFA chapter advisers and to their students who completed written surveys, as well as to community nurses who completed telephone interviews. We thank Julie Sessions of the National FFA and several Marshfield staff associated with data collection and intervention efforts, including Mary Jo Knobloch, Virginia Fischer, Susan Greenwood, Carla Finck, Deb Kempf, and Nita Herr. We also thank Alice Stargardt for her assistance in the preparation of this article.
|
Footnotes |
|---|
Human Participation Protection
This study received approval from the Marshfield Clinic institutional review board. Participating schools secured necessary approvals for data collection. Study participants received verbal and written information regarding the voluntary nature of their completion of written surveys or telephone interviews.
Accepted for publication May 8, 2004.
|
References |
|---|
|
TOPABSTRACTINTRODUCTIONMETHODSRESULTSDISCUSSIONEPILOGUEReferences |
|---|
2. National FFA Web site. About. Available at: http://www.ffa.org/about_ffa/index.html. Accessed August 18, 2004.
3. Tevis C, Finck C. We kill too many farm kids. Success Farm. 1989;87(3):18A–18P.
4. National Committee for Childhood Agricultural Injury Prevention. Children and Agriculture: Opportunities for Safety and Health. Marshfield, Wisc: Marshfield Clinic, 1996. Also available at: http://research.marshfieldclinic.org/children/action/title.htm. Accessed April 20, 2004.
5. Teaching Agricultural Safety to Kids. Springfield, Ill: Illinois Easter Seal Society; 1996.
6. Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93:1261–1267.
7. Haddix AC, Teutsch SM, Shaffer PA, Dunet DO. Prevention Effectiveness. A Guide to Decision Analysis and Economic Evaluation. New York: Oxford University Press; 1996.
8. Rossi PH, Freeman HE, Lipsey MW. Evaluation. A Systematic Approach. 6th ed. Thousand Oaks, Calif: Sage Publications; 1999.
9. Teutsch SM. A framework for assessing the effectiveness of disease and injury prevention. MMWR Recomm Rep. 1992;41:1–12.[Medline]
10. Flay BR. Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev Med. 1986;15:451–474.[ISI][Medline]
11. Centers for Disease Control and Prevention. Community Partners for Healthy Farming Intervention Research Program. Cincinnati, Ohio: CDC; 1999.
12. US Dept of Agriculture. Census of Agriculture. Washington, DC: Government Printing Office; 1992.
13. Westaby JD, Lee BC. Antecedents of injury among youth in agricultural settings: a longitudinal examination of safety consciousness, dangerous risk taking, and safety knowledge. J Safety Res. 2003;34(3): 227–240.[ISI][Medline]
14. Rosenberg M. Conceiving the Self. New York: Basic Books; 1979.
15. Damrosch SP. Ensuring anonymity by use of subject-generated identification codes. Res Nurs Health. 1986;9:61–63.[ISI][Medline]
16. Varnell SP, Murray DM, Janega JB, Blitstein JL. Design and analysis of group-randomized trials: a review of recent practices. Am J Public Health. 2004;94: 393–399.
17. DeRoo LA, Rautiainen RH. A systematic review of farm safety interventions. Am J Prev Med. 2000; 18(4S):51–62.[ISI][Medline]
18. Aherin R, Murphy D, Westaby J. Reducing Farm Injuries: Issues and Methods. Report of American Society of Agricultural Engineers. St. Joseph, Mich: ASAE; 1992.
19. Lee B, Gallagher S, Marlenga B, Hard D. Childhood Agricultural Injury Prevention: Progress Report and Updated National Action Plan from the 2001 Summit. Marshfield, Wisc: Marshfield Clinic; 2002.
20. Schieber RA, Vegega ME. Education versus environmental countermeasures: is it really an either-or proposition? Am J Health Educ. 2003;34(suppl): S54–S56.
21. Rivara FP, Thompson DC. Systematic reviews of injury-prevention strategies for occupational injuries. Am J Prev Med. 2000;18(4S):1–3.
22. Doll L, Bartenfeld T, Binder S. Evaluation of interventions designed to prevent and control injuries. Epidemiol Rev. 2003;25:51–59.
23. Pickett W, Hartling L, Crumley E, Klassen T, Brison RJ. A Systematic Review of Prevention Strategies for Childhood Farm Injuries. Toronto, Ontario: Safe Kids Canada; 2003.
24. Klassen TP, MacKay JM, Moher D, Walker A, Jones AL. Community-based injury prevention interventions. Future of Children. 2000;10:83–110.[ISI][Medline]
25. Nation M, Crusto C, Wandersman A, et al. What works in prevention. Principles of effective prevention programs. Am Psychol. 2003;58:449–456.[Medline]
26. Thompson NJ, McClintock HO. Demonstrating Your Program’s Worth. A Primer on Evaluation for Programs to Prevent Unintentional Injury. Atlanta, Ga: National Center for Injury Prevention and Control; 2000.
27. Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94:400–405.
28. Rivara FP. Introduction: the scientific basis for injury control. Epidemiol Rev. 2003;25:20–23.
29. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med. 2004;27:3–12.[ISI][Medline]
This article has been cited by other articles:
|
|
 |
|
  L. A. Yurek, J. Vasey, and D. Sullivan Havens The Use of Self-Generated Identification Codes in Longitudinal Research Eval Rev, October 1, 2008; 32(5): 435 - 452. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
