Assignment #1 – Evaluation of a
program evaluation
The completed
evaluation was of an anti-bullying program entitled, “Project Ploughshares
Puppets for Peace “(P4). The objectives of the program were two-fold; for
students to increase understanding of what constitutes bullying behavior, and
recognizing strategies that can be used when confronted with or observing
bullying. The evaluators ultimately
deemed that the program did not meet the outlined objectives, but that it did
uncover one unexpected outcome that was positive; increased student confidence
(perceived, not actualized) to manage bullying.
In looking at
methods used to conduct their research, I saw several components of the CIPP
evaluation model used. The report clearly discussed the context or relevance of the need for such programs as the
P4program. It referred to the increasing awareness of the negative impacts of
bullying that schools are realizing, and that the media is frequently
reporting. In terms of inputs, a
limited focus was placed on the prior knowledge (determined through
administration of a pre-puppet show questionnaire) that students already had
about bullying (types and coping strategies). Finally, the product included the re-administered questionnaire (plus one
additional values based, open ended question) that took place after the puppet
show (immediately and 3 months later) that determined whether the program objectives
were met. Based on the Scrivens model of
program evaluation, the evaluation was summative, in that it quite simply
determined that the end result of the P4 program was not successful in meeting
it’s primary objectives. With regards to implementation failure vs. theory
failure, the program evaluation indicates that the program was implemented as
planned, but that the program did not produce the anticipated results,
therefore the program experienced theory failure.
The evaluation
showed several areas of strength in its approach. It conducted the questionnaires with a
participant group and a non participant group so as to compare the effectiveness
in meeting program objectives. It acknowledged an unexpected outcome of the
program; student perception of increased confidence if having to deal with
bullies (hypothetically). Also, 100% of participants participated in the three
month re-administration of survey. Most importantly, it recognized and
described several limitations to the
evaluation findings including that student awareness of bullying was quite high
to begin with (thereby limiting potential growth that was possible) and that time between administration of the first
two questionnaires was potentially too short to ensure validity.
I felt that the
evaluation missed digging into several key factors that may have contributed to
effectively measuring the project’s success in meeting it’s objectives. While
the evaluation surveyed students from two different schools, I believe that
including a greater number of schools (with a detailed comparison of types of
bullying education that had taken place prior to program implementation) would
have increased the validity of the results. Also, the number of students involved in the evaluation
(129 students) seems small. Carefully considering the exposure that students
have had to bullying education, and the frequency of bullying they face daily
would, in my opinion, greatly impact the effectiveness of the program. Greater
attention to ensuring a varied sample group would be something I would
recommend. The evaluation recognized that objectives of program were not met,
but did not make substantial suggestions for change. Rather, a general
statement was made that a program such as P4 is only useful as a part of a
larger program that includes ongoing education, skills training, and
family/school based effort to support children as they deal with the issues
surrounding bullying.
Article:
Beran, T., & Shapiro, B.
(2005). Evaluation of an anti-bullying program: Student reports of knowledge
and confidence to manage bullying. Canadian Journal of Education, 28(4),
700-717. Retrieved from http://search.proquest.com/docview/215373879?accountid=14739
Candace
ReplyDeleteGreat selection of PE to review. The choice of a focused relatively well defined program allows you to really pick it apart. It is too bad that the program is not working and I agree with your assessment why this is so. Theory failure does appear to be an issue. Also the design of the study not including a wide enough sample or conducting a pre assessment of participants are flaws in my opinion. Very much a formative look a la Scrivens.
Jay