At a loss for what to write, I once again went to one of my favorite books, Michael Scriven’s Evaluation Thesaurus . This time when I opened the volume randomly, I came upon the entry for meta-evaluation. This is a worthy topic, one that isn’t addressed often. So this week, I’ll talk about meta-evaluation and quote Scriven as I do.
First, what is meta-evaluation? This is an evaluation approach which is the evaluation of evaluations (and “indirectly, the evaluation of evaluators”). Scriven suggests the application of an evaluation-specific checklist or a Key Evaluation Checklist (KEC) (p. 228). Although this approach can be used to evaluate one’s own work, the results are typically unreliable which implies (if one can afford it) to use an independent evaluator to conduct a meta-evaluation of your evaluations.
Then, Scriven goes on to say the following key points:
He lists the parts a KEC involved in a meta evaluation; this process includes 13 steps (pp. 230-231).
He gives the following reference:
Stufflebeam, D. (1981). Meta-evaluation: Concepts, standards, and uses. In R. Berk (Ed.), Educational evaluation methodology: The state of the art. Baltimore, MD: Johns Hopkins.
This will be short.
I showed a revised version of Alkin’s Evaluation Theory Tree in last week’s post. It had leaves. It looked like this:
It was taken from the second edition of Alkin’s book.
I have had two comments about this tree.
In Chapter Two, he and Tina Christie talk about an evaluation theory tree and presents this idea graphically (all be it in draft form).
Think of your typical tree with three strong branches (no leaves) and two roots. Using this metaphor, the authors explain the development of evaluation theory as it appears in western (read global north) societies.
As you can see, the roots are “accountability and control” (positivist paradigm?) and social inquiry (post-positivist paradigm?). Read the rest of this entry »
I read. A lot.
I also blog. Weekly, unless I’m not in the office.
This past week I read (again) Harold Jarche’s blog. He posts periodically on interesting social media finds. Some of these finds are relevant to evaluation (even if they are not framed that way). His post on October 17 included a post from Kate Pinner called Half-baked ideas (She is found on twitter @kmpinner ). She says, “Just because you know how to do something doesn’t mean you should: It’s rewarding to give other people a chance to shine.”
Pinner’s comment is related to a thought I’ve been mulling for some time now (a couple of years, actually). That is the whole idea of “doing as.”
David Fetterman talks about empowerment evaluation Read the rest of this entry »
What? So what? Now what?
Sounds like an evaluation problem.
King and Stevahn (in press) tells us the first query requires thoughtful observation of a situation; the second query a discussion of possible options and implications of those options, and the third query calls for the creation of a list of potential next steps.
Yet these are the key words for “adaptive action” (If you haven’t looked at the web site, I suggest you do.) One quote that is reflective of adaptive action is, “Adaptive Action reveals how we can be proactive in managing today and influencing tomorrow.”( David W. Jamieson, University of St. Thomas). Adaptive action can help you
Evaluation is a proactive (usually) activity (oh, I know that sometimes evaluation is flying by the seat of your pants and is totally reactive). People are now recognizing that evaluation will benefit them, their programs, and their organizations and that it isn’t personal (although that fear is still out there).
Although the site is directed towards leadership in organizations, the key questions are evaluative. You can’t determine “what” without evidence (data); you can’t determine “so what” unless you have a plan (logic model), and you can’t think about “now what” unless you have an outcome that you can move toward. These questions are evaluative in contemporary times because there are no simple problems any more. (Panarchy approaches similar situations using a similar model .) Complex situations are facing program people and evaluators all the time. Using adaptive action may help. Panarchy may help (the book is called Panarchy by Gunderson and Hollings .)
Just think of adaptive action as another model of evaluation.
How do you approach evaluation?
Are you the expert?
Do you work in partnership?
Are you one of the group?
To which question did you answer yes?
If you are the expert and know the most (not everything, no one know everything [although teenagers think they do]), you are probably “doing to”. Extension has been “doing to” for most of its existence. Read the rest of this entry »
On May 9, 2014, Dr. Don Kirkpatrick died at the age of 90. His approach (called a model) to evaluation was developed in 1954 and has served the training and development arena well since then; it continues to do so.
For those of you who are not familiar with the Kirkpatrick model, here is a primer, albeit short. (There are extensive training programs for getting certified in this model, if you want to know more.)
Don Kirkpatrick, Ph. D. developed the Kirkpatrick model when he was a doctoral student; it was the subject of his dissertation which was defended in 1954. There are four levels (they are color coded on the Kirkpatrick website) and I quote:
Level 1: Reaction
To what degree participants react favorably to the training
Level 2: Learning
To what degree participants acquire the intended knowledge, skills, attitudes, confidence and commitment based on their participation in a training event
Level 3: Behavior
To what degree participants apply what they learned during training when they are back on the job
Level 4: Results
To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement
Sounds simple, right. (Reminiscent of a logic model’s short, medium, and long term outcomes). He was the first to admit that it is difficult to get to level four (no world peace for this guy, unfortunately). We all know that behavior can be observed and reported, although self-report is fraught with problems (self-selection, desired response, other cognitive bias, etc.). Read the rest of this entry »
I had a comment about last week’s post on Sustainability and Evaluation. I will share it here. I wonder what you readers think of this comment:
In preventive health/health promotion, ‘sustainability’ has generally been used to indicate that the intervention program, or elements of it, or benefits of it, are continued beyond the life of the funded program. It is about extending the value of the investment in a program, beyond the life of the funded program. So it’s about a legacy, about the continuation of things achieved, about leaving things, circumstances or communities better off than when you first arrived (more empowered, more resourceful, more able to continue improvements on their own).
I wonder how that fits with the definitions I provided? Is this a different sustainability? Does it speak to the future generations? Does that include equity and justice? Read the rest of this entry »
Intention to change
I’ve talked about intention to change and how stating that intention out loud and to others makes a difference. This piece of advice is showing up in some unexpected places and here. If you state your goal, there is a higher likelihood that you will be successful. That makes sense. If you confess publicly (or even to a priest), you are more likely to do the penance/make a change. What I find interesting is that this is so evaluation. What difference did the intervention make? How does that difference relate to the merit, worth, value of the program?
Lent started March 5. That is 40 days of discipline–giving up or taking on. That is a program. What difference will it make? Can you go 40 days without chocolate?
I got my last comment in November, 2013. I miss comments. Sure most of them were check out this other web site. Still there were some substantive comments and I’ve read those and archived them. My IT person doesn’t know what was the impetus for this sudden stop. Perhaps Google changed its search engine optimization code and my key words are no longer in the top. So I don’t know if what I write is meaningful; is worthwhile; or is resonating with you the reader in any way. I have been blogging now for over four years…this is no easy task. Comments and/or questions would be helpful, give me some direction.
Chris Lysy cartoons in his blog. This week he blogged about logic models. He only included logic models that are drawn with boxes. What if the logic model is circular? How would it be different? Can it still lead to outcomes? Non-linear thinkers/cultures would say so. How would you draw it? Given that mind mapping may also be a model, how do they relate?
I’ve been reading about models lately; models that have been developed, models that are being used today, models that may be used tomorrow.
Webster (Seventh New Collegiate) Dictionary has almost two inches about models–I think my favorite definition is the fifth one: an example for imitation or emulation. It seems to be most relevant to evaluation. What do evaluators do if not imitate or emulate others?
To that end, I went looking for evaluation models. Jim Popham’s book has a chapter (2, Alternative approaches to educational evaluation) on models. Fitzpatrick, Sanders, and Worthen has numerous chapters on “approaches” (what Popham calls models). (I wonder if this is just semantics?)
Models have appeared in other blogs (not called models, though). In the case of Life in Perpetual Beta (Harold Jarche) provides this view of how organizations have evolved and calls them forms.(The below image is credited to David Ronfeldt.)
(Looks like a model to me. I wonder what evaluators could make of this.)
The reading is interesting because it is flexible. It approaches the “if it works, use it” paradigm; the one I use regularly.
I’ll just list the models Popham uses and discuss them over the next several weeks. (FYI-both Popham and Fitzpatrick, et. al., talk about the overlap of models.) Why is a discussion of models important, you may ask? I’ll quote Stufflebeam: “The study of alternative evaluation approaches is important for professionalizing program evaluation and for its scientific advancement and operation” (2001, p. 9).
Popham lists the following models:
Popham does say that the model classification could have been done a different way. You will see that in the Fitzpatrick, Sanders, and Worthen volume where they talk about the following approaches:
They have a nice table that does a comparative analysis of alternative approaches (Table 10.1, pp. 249-251)
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Boston, MA: Pearson.
Popham, W. J. (1993). Educational Evaluation (3rd ed.). Boston, MA: Allyn and Bacon.
Stufflebeam, D. L. (2001). Evaluation models. New Directions for Evaluation (89). San Francisco, CA: Jossey-Bass.