I’ve been reading about models lately; models that have been developed, models that are being used today, models that may be used tomorrow.
Webster (Seventh New Collegiate) Dictionary has almost two inches about models–I think my favorite definition is the fifth one: an example for imitation or emulation. It seems to be most relevant to evaluation. What do evaluators do if not imitate or emulate others?
To that end, I went looking for evaluation models. Jim Popham’s book has a chapter (2, Alternative approaches to educational evaluation) on models. Fitzpatrick, Sanders, and Worthen has numerous chapters on “approaches” (what Popham calls models). (I wonder if this is just semantics?)
Models have appeared in other blogs (not called models, though). In the case of Life in Perpetual Beta (Harold Jarche) provides this view of how organizations have evolved and calls them forms.(The below image is credited to David Ronfeldt.)
(Looks like a model to me. I wonder what evaluators could make of this.)
The reading is interesting because it is flexible. It approaches the “if it works, use it” paradigm; the one I use regularly.
I’ll just list the models Popham uses and discuss them over the next several weeks. (FYI-both Popham and Fitzpatrick, et. al., talk about the overlap of models.) Why is a discussion of models important, you may ask? I’ll quote Stufflebeam: “The study of alternative evaluation approaches is important for professionalizing program evaluation and for its scientific advancement and operation” (2001, p. 9).
Popham lists the following models:
Popham does say that the model classification could have been done a different way. You will see that in the Fitzpatrick, Sanders, and Worthen volume where they talk about the following approaches:
They have a nice table that does a comparative analysis of alternative approaches (Table 10.1, pp. 249-251)
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Boston, MA: Pearson.
Popham, W. J. (1993). Educational Evaluation (3rd ed.). Boston, MA: Allyn and Bacon.
Stufflebeam, D. L. (2001). Evaluation models. New Directions for Evaluation (89). San Francisco, CA: Jossey-Bass.
People often say one thing and do another.
This came home clearly to me with a nutrition project conducted with fifth and sixth grade students over the course of two consecutive semesters. We taught them nutrition and fitness and assorted various nutrition and fitness concepts (nutrient density, empty calories, food groups, energy requirements, etc.). We asked them at the beginning to identify which snack they would choose if they were with their friends (apple, carrots, peanut butter crackers, chocolate chip cookie, potato chips). We asked them at the end of the project the same question. They said they would choose an apple both pre and post. On the pretest, in descending order, the students would choose carrots, potato chips, chocolate chip cookies, and peanut butter crackers. On the post test, in descending order, the students would choose chocolate chip cookies, carrots, potato chips, and peanut butter crackers. (Although the sample sizes were reasonable [i.e., greater than 30], I’m not sure that the difference between 13.0% [potato chips] and 12.7% [peanut butter crackers] was significant. I do not have those data.) Then, we also asked them to choose one real snack. What they said and what they did was not the same, even at the end of the project. Cookies won, hands down in both the treatment and control groups. Discouraging to say the least; disappointing to be sure. What they said they would do and what they actually did were different.
Although this program ran from September through April, and is much longer than the typical professional development conference of a half day (or even a day), what the students said was different from what the students did. We attempted to measure knowledge, attitude, and behavior. We did not measure intention to change.
That experience reminded me of a finding of Paul Mazmanian . (I know I’ve talked about him and his work before; his work bears repeating.) He did a randomized controlled trial involving continuing medical education and commitment to change. After all, any program worth its salt will result in behavior change, right? So Paul Mazmanian set up this experiment involving doctors, the world’s worst folks with whom to try to change behavior.
He found that “…physicians in both the study and the control groups were significantly more likely to change (47% vs 7%, p<0.001) IF they indicated an INTENT (emphasis added in both cases) to change immediately following the lecture ” (i.e., the continuing education program). He did a further study and found that a signature stating that they would change didn’t increase the likelihood that they would change.
Bottom line, measure intention to change in evaluating your programs.
Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowitz, M. P. (August 1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine, 73(8), 882-886.
Mazmanian, P. E., Johnson, R. E., Zhang, A., Boothby, J. & Yeatts, E. J. (June, 2001). Effects of a signature on rates of change: A randomized controlled trial involving continuing education and the commitment-to-change model. Academic Medicine, 76(6), 642-646.
I may have mentioned naturalistic models; if not I needed to label them as such.
Today, I’ll talk some more about those models.
These models are often described as qualitative. Egon Guba (who died in 2008) and Yvonna Lincoln (distinguished professor of higher education at Texas A&M University) talk about qualitative inquiry in their 1981 book, Effective Evaluation (it has a long subtitle–here is the cover). They indicate that there are two factors on which constraints can be imposed: 1) antecedent variables and 2) possible outcomes, with the first impinging on the evaluation at its outset and the second referring to the possible consequences of the program. They propose a 2×2 figure to contrast between naturalistic inquiry and scientific inquiry depending on the constraints.
Besides Eisner’s model, Robert Stake and David Fetterman have developed models that fit this model. Stake’s model is called responsive evaluation and Fetterman talks about ethnographic evaluation. Stake’s work is described in Standards-Based & Responsive Evaluation (2004) . Fetterman has a volume called Ethnography: Step-by-Step (2010) .
Stake contended that evaluators needed to be more responsive to the issues associated with the program and in being responsive, measurement precision would be decreased. He argued that an evaluation (and he is talking about educational program evaluation) would be responsive if it “oreints more directly to program activities than to program intents; responds to audience requirements for information and if the different value perspectives present are referred to in reporting the success and failure of the program” (as cited in Popham, 1993, pg. 42). He indicates that human instruments (observers and judges) will be the data gathering approaches. Stake views responsive evaluation to be “informal, flexible, subjective, and based on evolving audience concerns” (Popham, 1993, pg. 43). He indicates that this approach is based on anthropology as opposed to psychology.
More on Fetterman’s ethnography model later.
Fetterman, D. M. (2010). Ethnography step-by-step. Applied Social Research Methods Series, 17. Los Angeles, CA: Sage Publications.
Popham, W. J. (1993). Educational Evaluation (3rd ed.). Boston, MA: Allyn and Bacon.
Stake, R. E. (1975). Evaluating the arts in education: a responsive approach. Columbus, OH: Charles E. Merrill.
Stake, R. E. (2004). Standards-based & responsive evaluation. Thousand Oaks, CA: Sage Publications.
Warning: This post may contain information that is controversial .
Schools (local public schools) were closed (still are).
The University (which never closes) was closed for four days (now open).
The snow kept falling and falling and falling. (Thank you Sandra Thiesen for the photo.)
Eighteen inches. Then freezing rain. It is a mess (although as I write this, the sun is shining, and it is 39F and supposed to get to 45F by this afternoon).
This is a complex messy system (thank you Dave Bella). It isn’t getting better. This is the second snow Corvallis has experienced in the same number of months, with increasing amounts.
It rains in the valley in Oregon; IT DOES NOT SNOW.
Another example of a complex messy system is what is happening in the UK.
These are examples extreme events; examples of climate chaos.
Evaluating complex messy systems is not easy. There are many parts. If you hold constant one part, what happens to the others? If you don’t hold constant one part, what happens to the rest of the system?. Systems thinking and systems evaluation has come of age with the 21st century; there were always people who viewed the world as a system; one part linked to another, indivisible. Soft systems theory dates back to at least von Bertalanffy who developed general systems theory and published the book by the same name in 1968 (ISBN 0-8076-0453-4).
Evaluating systems is complicated and complex.
Bob Williams, along with Iraj Imam, edited the volume Systems Concepts in Evaluation (2007), and along with Richard Hummelbrunner, wrote the volume Systems Concepts in Action: A Practitioner’s Toolkit (2010). He is a leader in systems and evaluation.
These two books relate to my political statement at the beginning and complex messy systems. According to Amazon, the second book “explores the application of systems ideas to investigate, evaluate, and intervene in complex and messy situations”.
If you think your program works in isolation, think again. If you think your program doesn’t influence other programs, individuals, stakeholders, think again. You work in a complex messy system. Because you work in a complex messy system, you might want to simplify the situation (I know I do); only you can’t. You have to work within the system.
Might be worth while to get von Bertalanffy’s book; might be worth while to get Williams books; might be worth while to get a copy of Gunderson and Holling book Panarchy: Understanding Transformations in Systems of Humans and Nature.
After all, nature is a complex messy system.
On February 1 at 12:00 pm PT, I will be holding my annual virtual tea party. This is something I’ve been doing since February of 1993. I was in Minnesota and the winter was very cold, and although not as bleak as winter in Oregon, I was missing my friends who did not live near me. I had a tea party for the folks who were local and wanted to think that those who were not local were enjoying the tea party as well. So I created a virtual tea party. At that time, the internet was not available; all this was done in hard copy (to this day, I have one or two friends who do not have internet…sigh…). Today, the internet makes the tea party truly virtual–well the invitation is; you have to have a real cup of tea where ever you are.
How is this evaluative? Gandhi says that only you can be the change you want to see…this is one way you can make a difference. How will you know?
I know because my list of invitees has grown exponentially. And some of them share the invitation. They pass it on. I started with a dozen or so friends. Now my address list is over three pages long. Including my daughters and daughters of my friends (maybe sons, too for that matter…)
Other ways: Design an evaluation plan; develop a logic model; create a metric/rubric. Report the difference. This might be a good place for using an approach other than a survey or Likert scale. Think about it.
Evaluation models abound.
Models are a set of plans.
Educational evaluation models are plans that could “lead to more effective evaluations” (Popham, 1993, p. 23). Popham (1993) goes on to say that there was little or no thought given to a new evaluation model that would make it distinct from other models so that in sorting models into categories, the categories “fail to satisfy…without overlap” (p. 24). Popham employs five categories:
I want to acquaint you with one of the naturalistic models, the connoisseurship model. (I hope y’all recognize the work of Guba and Lincoln in the evolution of naturalistic models; if not I have listed several sources below.) Elliott Eisner drew upon his experience as an art educator and used art criticism as the basis for this model. His approach relies on educational connoisseurship and educational criticism. Connoisseurship focuses on complex entities (think art, wine, chocolate); criticism is a form which “discerns the qualities of an event or object” (Popham, 1993, p. 43) and puts into words that which has been experienced. This verbal presentation allows for those of us who do not posess the critic’s expertise can understand what was perceived. Eisner advocated that design is all about relationships and relationships are necessary for the creative process and thinking about the creative process. He proposed “that experienced experts, like critics of the arts, bring their expertise to bear on evaluating the quality of programs…” (Fitzpatrick, Sanders and Worthen, 2004). He proposed an artistic paradigm (rather than a scientific one) as a supplement other forms of inquiry. It is from this view that connoisseurship derives—connoisseurship is the art of appreciation; the relationships between/among the qualities of the evaluand.
Elliot Eisner died January 10, 2014; he was 81. He was the Lee Jacks Professor of Education at Stanford Graduate School of Education. He advanced the role of arts in education and used arts as models for improving educational practice in other fields. His contribution to evaluation was significant.
Eisner, E. W. (1975). The perceptive eye: Toward the reformation of educational evaluation. Occasional Papers of the Stanford Evaluation Consortium. Stanford, CA: Stanford University Press.
Eisner, E. W. (1991a). Taking a second look: Educational connoisseurship revisited. In Evaluation and education: At quarter century, ed. M. W. McLaughlin & D. C. Phillips. Chicago: University of Chicago Press.
Eisner, E. W. (1991b). The enlightened eye: Qualitative inquiry and the enhancement of educational practice. New York: Macmillian.
Eisner, E. W., & Peshkin, A. (eds.) (1990). Qualitative inquiry in education. NY:Teachers College Press.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program Evaluation: Alternative approaches and practical guidelines, 3rd ed. Boston, MA: Pearson
Guba, E. G., & Lincoln, Y. S. (1981). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches. San Francisco: Jossey-Bass.
Lincoln, Y. S. & Guba, E. G. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage Publications.
Patton, M. Q. (2002). Qualitative research & evaluation methods. 3rd ed. Thousand Oaks, CA: Sage Publications.
Popham, W. J. (1993). Educational evaluation. 3rd ed. Boston, MA: Allyn and Bacon.
It might be useful for those of you who are interested in evaluation to review a list of evaluation conferences offered around the world this year. Sarah Baughman cited the list offered by Better Evaluation. You could spend all year coming and going. What a way to see the world. That certainly has evaluative opportunities.
2014 beginning question.
One question I am asked by people new to evaluation is How often do I need to conduct an evaluation? How much budget/time/resources do I allot for evaluation? My evaluative answer is “It all depends.”
For new faculty who want to know if their programs are working, (not impact, just working), identify your most important program and evaluate it. Next year, so another program, and so on. If you want to know impact, you will need to wait at least three years, maybe five. Although some programs could show impact after one year. (We are not talking world peace here, only did the program make a difference, does it have merit, value, worth?)
For executive directors, my “it depends answer is still important. They have different needs than program planners and those who implement programs. My friend Stan says executive directors need to know: What is the problem? What caused the problem? How do I solve the problem (in two sentences or less)? Executive directors don’t have a lot of time to devote to evaluation; yet they need to know.
For people who are continuing a program of long standing, I would suggest you answer the question that is most pressing. (It all depends…)
I think these categories mostly cover everybody. If you can think of other situations, let me know. I’ll tell you what I think.
Did you know that there are at least 11 winter holidays besides Christmas–many of them related to light or the return of light.
One needs evaluation tools to determine the merit or worth, to evaluate the holiday’s value to you. For me, any that return light are important. So for me, there is Hanukkah (and eight candles), Solstice (and bonfires and yule logs), Christmas (and Advent wreaths with five candles), Kwanzaa ( and kinara seven candles). Sometimes Diwali falls late in November to be included (it is the ancient Hindu festival of lights that is a movable feast like Hanukkah).
I have celebrations for Hanukkah (I have several menorahs), for Solstice (I have two special candelabra that holds 12 candles–a mini-bonfire to be sure), for Advent/Christmas (I make a wreath each year), and for Kwanzaa (a handmade Kinara). And foods for each celebration as well. Because I live in a multicultural household, it is important that everyone understand that no holiday is more important than any other–all talk about returning light (literal or figurative). Sometimes the holidays over lap–Hanukkah, Solstice, Christmas all in the same week…phew, I’m exhausted just thinking about it. Sometimes it seems hard to keep them separate–then I realized that returning the light is not separate; it is light returning. It is an evaluative task.
So well come the new born sun/son…the light returns. Evaluation continues.
Happy Holidays…all of them!
I’m taking two weeks holiday–will see you in the new year.
Nelson Mandela died last week (Thursday, actually) at the age of 95. Invictus is the name of a movie which recounts the poem below. While in prison on Robbon Island, he recited this poem to other prisoners and was empowered by the self-mastery message in it. It is a powerful poem. Mandela was a powerful person. We and the world were blessed that he was with us for 95 years; that he was the master of his fate and captain of his soul.
Out of the night that covers me,
Black as the pit from pole to pole,
I thank whatever gods may be
For my unconquerable soul.
In the fell clutch of circumstance
I have not winced nor cried aloud.
Under the bludgeonings of chance
My head is bloody, but unbowed.
Beyond this place of wrath and tears
Looms but the horror of the shade,
And yet the menace of the years
Finds and shall find me unafraid.
It matters not how strait the gate,
How charged with punishments the scroll,
I am the master of my fate:
I am the captain of my soul.
~~William Ernest Henley
When I read this poem and think of Mandela (aka Madiba). I also think of the evaluator’s guiding principles, especially the last three: Honesty/Integrity, Respect for People, and Responsibilities for General and Public Welfare. Mandela could have been an evaluator as even the first two principles could apply (Systematic Inquiry and Competence). He was certainly competent and he did systematic inquiry. He used these principles in an arena other than evaluation. Yet by doing what he did, he was able to determine the merit and worth of what he did. The world was lucky to have him for so long. He was the change he wished to see; and he changed the world.
I was reminded about the age of this blog (see comment below). Then it occurred to me: I’ve been writing this blog since December 2009. That is 4 years of almost weekly posts. And even though evaluation is my primary focus, I occasionally get on my soap box and do something different (White Christmas Pie, anyone?). My other passion besides evaluation is food and cooking. I gave a Latke party on Saturday and the food was pretty–and it even tasted good. I was more impressed by the visual appeal of my table; my guests were more impressed by the array of tastes, flavors, and textures. I’d say the evening was a success. This blog is a metaphor for that table. Sometimes I’m impressed with the visual appeal; sometimes I’m impressed with the content. Today is an anniversary. Four years. I find that amazing (visual appeal). The quote below (a comment offered by a reader on the post “Is this blog making a difference?”, a post I made a long time ago) is about content.
“Judging just from the age of your blog I must speculate that you’ve done something right. If not then I doubt you’d still be writing regularly. Evaluation of your progress is important but pales in comparison to the importance of writing fresh new content on a regular basis. Content that can be found no place else is what makes a blog truly useful and indeed helps it make a difference.”
Audit or evaluation?
I’m an evaluator; I want to know what difference the “program” is making in the lives of the participants. The local school district where I live, work, and send my children to school has provided middle school children with iPads . They want to “audit” their use. I commend the school district for that initiative (both giving the iPads as well wanting to determine the effectiveness). I wonder if they really want to know what difference the electronics are making in the lives of the students. I guess I need to go re-read Tom Schwandt’s 1988 book, “Linking Auditing and Metaevaluation”, a book he wrote with Ed Halpern, as well as see what has happened in the last 25 years (and it is NOT that I do not have anything else to read…). I think it is important to note the sentence (taken from the forward), “Nontraditional studies are found not only in education, but also in…divers fields …(and the list they provide is a who’s who in social science). The problem of such studies is “establishing their merit”. That is always a problem with evaluation–establishing the merit, worth, value of a program (study).
We could spend a lot of time debating the merit, worth, value of using electronics in the pursuit of learning. (In fact, Jeffrey Selingo writes about the need to personalize instruction using electronics in his 2013 book “College (Un)bound”–very readable, recommended.) I do not think counting the number of apps or the number of page views is going to answer the question posed. I do not think counting the number of iPads returned in working condition will either. This is an interesting experiment. How , reader, would you evaluate the merit, worth, value of giving iPads to middle school children? All ideas are welcome–let me know because I do not have an answer, only an idea.