EvaluationClock

=Evaluation Clock=

The Evaluation Clock (adapted from Pietro, D. S. 1983. __Evaluation Sourcebook__. New York, American Council of Voluntary Agencies for Foreign Service) unpacks the "evaluation or systematic study" component of the Action Research cycle. It indirectly addresses the “planning component” by making you look ahead to consider which people might be influenced by the results and what they could do based on the possible outcomes.

The ultimate goal of using the Clock framework is that you can use it to design your own evaluation or systematic study mindfully, working both: > sequentially—addressing the whole range of considerations (moving from steps 0 to 11); and > recursively—adjusting your plans for the earlier steps in light of thinking ahead about possibilities for the later steps. In particular, evaluation and planning/design should be inextricably linked. For example, when you think about what could be done differently (step 11) on the basis of the specific measurements/ observations you include in the evaluation (step 3), you may refine your measurements/observations, or even decide that you need to separate out two or more different sub-issues within the overall issue (steps 0-2), each requiring a different evaluation. As Pietro (1983, 23) says: “The clock marks time in an unusual fashion, since it does not necessarily move in a clockwise direction, but rather jumps from one number to another until all the questions have been struck.” It has been suggested that the clock is more like a combination lock on a safe. Working sequentially and recursively is analogous to Action Research, except that with the Evaluation Clock each step might require tight, self-conscious method (e.g., statistical analysis).

Comparisons
When an evaluation is a systematic study of effects of some intervention/engagement, there is always a comparison involved. The comparison might be before vs. after some intervention is made, or it might be a comparison of one situation (with a particular curriculum, treatment, etc.) vs. another situation (without that curriculum, etc.) (steps 2 &amp; 3 of the clock). Did it have the intended effects? Was it better than other approaches? The idea of comparison can also be applied to continuous data, e.g., on the incidence of violent crimes in relation to unemployment rate. This is, more or less, like asking is there more (or less) violent crime in times of high unemployment than in times of low unemployment? In valid comparisons all other factors are supposed to be equal or unchanged. If they are not, then the comparison is suspect. Perhaps it needs to be broken into a number of comparisons, e.g., before vs. after for wealthy schools, and before vs. after for poor schools. When an evaluation is a systematic study of what has already been happening, it may only involve collecting information about one situation, e.g., finding what % of adults are able to read competently. The formulation of the evaluation criteria and interpretation of the results depends, however, on an implicit comparison with a desired situation, e.g, one of full adult literacy.  In order to get acquainted with the Clock, the comparison at its heart, and the sequential and recursive aspects of using the Clock, it is helpful to reconstruct an evaluation that has been conducted. When you do this you have to put yourself in the shoes of the group or person(s) who conducted the evaluation and fill in the steps they appear to have taken. In order to get the hang of comparisons, focus on steps 2 &amp; 3 for a simple case (e.g., the smoking in bars clipping). Steps 0, 4 &amp; 5 may help you as well. (See “stripped down clock” appended after the full clock.)

When you have the hang of the comparison idea, work on the sequential and recursive aspects of the Clock: The **sequential** part of this reconstruction means that the answers at each step are logically related to the previous ones, especially the immediately preceding one. For example, the lessons in step 10 are lessons learned from the reasons (step 9) for what is happening (step 8a). Similarly, the outlets (step 8b) should take into account the sponsors goals/audience (step 1). Sequentiality also means that the key issues of the evaluation (step 2) are not the issues that emerge after the results (steps 8-12). The key issues are the ones that the evaluator saw needed studying before they knew the actual results.

The **recursive** part of this reconstruction means that when you think about what the evaluator or their sponsors did with the results (steps 10 &amp; 11), or could conceivably do with them, you might go back and revise your interpretation of what decisions/ policies/ actions were at stake (steps 0 &amp; 1). For example, an evaluation that points out that a low % of NY City high school students are passing the Regents exam says little about causes of the low% or about ways to improve education in the school system. We might then suspect that what concerns the sponsors of the evaluation (step 0) was to discredit public education. This would have to be checked out, but someone wanting to improve public education would want to design a quite different evaluation.

When you try to make sense of evaluations that others have done or are proposing, you may see that parents, teachers, administrators, and policy makers want different things evaluated, even if they've been mixed into one "soup." For example, in high-stakes standardized tests evaluations of the following different things are mixed together: • students' knowledge • new curricular frameworks as a means to improve students' knowledge • performance of teachers • performance of schools; and • performance of school districts. You have to separate the different kinds of evaluation for any issue you choose, and address each appropriately. More generally, you should add notes from your own critical thinking about what others have done: Why evaluate in this situation? Why this evaluation and not another? What theories are hidden behind the intervention that was implemented? What supports are given to people to make the intervention?

A note on working from clippings: In using the clock to reconstruct an evaluation that has already been conducted, you have to put yourself in the shoes of the group or person(s) who conducted the evaluation and fill in the steps they appear to have taken. You should not answer the earlier steps with information that the people did not have until after they had conducted the evaluation. Often a newspaper clipping will not give you information for every step in the clock. In that case, fill in the step with what you would do as someone in the corresponding position, i.e., designing an evaluation (for the early steps), interpreting it (for the middle steps), or deciding on proposals to make (for the later steps). Deciding what you would do is a matter, as is the case in Action Research, of making proposals that follow from research results and presenting the proposals to potential constituencies who might take them up if the research supports them.

//April 08: In describing what is to be evaluated, the word "intervention" was substituted for the word "change" in order to eliminate the ambiguity that arises because the effect of an intervention is also a change.//

FULL CLOCK
[Copy and paste the clock into a word file to allow room for your responses.]

0a. The "intervention" (action/program/policy/curriculum/practice/treatment/difference/etc.) whose effect/effectiveness needs to be evaluated is... 0b. Interest or concern in the effectiveness of the intervention arises because...

_____________________________________________________________________________ 1a. The group or person(s) that sponsors the evaluation of the intervention are... 1b. The people they seek to influence with the results are... 1c. The actions/decisions/policies those people might improve or affirm concern...

_____________________________________________________________________________ 2. General Question:  The comparison needed to evaluate the effectiveness of the intervention is between two (or more) situations, namely a. a comparison of...

b. with respect to differences in the general area of….. _____________________________________________________________________________ 3. Specific observables:  To undertake that comparison, the effects of the intervention will be assessed by looking at the following specific variable(s) in the two (or more) situations...

_____________________________________________________________________________ 4. The methods to be used to produce observations or measurements of those variables are...(survey, questionnaire, etc.)

_____________________________________________________________________________ 5a. The people who will be observed/measured are... 5b. The observing/measuring is done in the following places/situations... or derived indirectly from the following sources...

_____________________________________________________________________________ 6. The observations/measurements will be analyzed to determine whether the two situations are significantly different in the following manner...

_____________________________________________________________________________ 7a. Given that people who will interpret (give meaning to) the analysis are... 7b. the analysis will be summarized/conveyed in the following form...

_____________________________________________________________________________ //When the results are available, the following steps can be pinned down. In the design stage, you should lay out different possiblities.// 8a. The results show that what has been happening is...  8b. This will be reported through the following outlets...

_____________________________________________________________________________ 9. What has been happening is happening because...

_____________________________________________________________________________ 10. The lessons learned by sponsors of evaluation are that...

_____________________________________________________________________________ 11. What the sponsors should now do differently is...

_____________________________________________________________________________

STRIPPING DOWN THE "CLOCK" TO FOCUS ON THE COMPARISON INVOLVED IN EVALUATING THE EFFECTS OF ANY EDUCATIONAL INTERVENTION
[Copy and paste the clock into a word file to allow room for your responses.]

0. The "intervention," i.e., action/program/policy/curriculum/practice/treatment/difference/etc. whose effectiveness needs to be evaluated is...

_____________________________________________________________________________ 2. The comparison needed to evaluate the effectiveness of the intervention is between two (or more) situations, namely comparing...

_____________________________________________________________________________ 3. To undertake that comparison, the effects of the intervention will be assessed by looking at the following specific variable(s) in the two situations...

_____________________________________________________________________________ 4. The methods to be used to produce observations or measurements of those variables are...(survey, questionnaire, etc.)

_____________________________________________________________________________ 5. The people who will be observed/measured are...  This is done in the following places/situations... or derived indirectly from the following sources...

(Original page by 1200255319)