Once again, it is the whole ‘balance’ thing…(we) live in ordinary life and that ordinary life is really the only life we have…I’ll take it. It has some great moments…
These wise words come from the insights of Buddy Stallings, Episcopal priest in charge of a large parish in a large city in the US. True, I took them out of context; the important thing is that they resonated with me from an evaluation perspective.
Too often, faculty and colleagues come to me and wonder what the impact is of this or that program. I wonder, What do they mean? What do they want to know? Are they only using words they have heard–the buzz words? I ponder how this fits into their ordinary life. Or are they outside their ordinary life, pretending in a foreign country?
A faculty member at Oregon State University equated history to a foreign country. I was put in a mind that evaluation is a foreign country to many (most) people, even though everyone evaluates every day, whether they know it or not. Individuals visit that contry because they are required to visit; to gather information; to report what they discovered. They do this with out any special preparation. Visiting a foreign country entails preparation (at least it does for me). A study of customs, mores, foods, language, behavior, tools (I’m sure I’m missing something important in this list) is needed; not just necessary, mandatory. Because although the foreign country may be exotic and unique and novel to you, it is ordinary life for everyone who lives there. The same is true for evaluation. There are customs; students are socialized to think and act in a certain way. Mores are constantly being called into question; language, behaviors, tools, which not known to you in your ordinary life, present themselves. You are constantly presented with opportunities to be outside your ordinary life. Yet, I wonder what are you missing by not seeing the ordinary; by pretending that it is extraordinary? By not doing the preparation to make evaluation part of your ordinary life, something you do without thinking.
So I ask you, What preparation have you done to visit this foreign country called EVALUATION? What are you currently doing to increase your understanding of this country? How does this visit change your ordinary life or can you get those great moments by recognizing that this is truly the only life you have? So I ask you, What are you really asking when you ask, What are the impacts?
All of this has significant implications for capacity building.
I came across this quote from Viktor Frankl today (thanks to a colleague)
“…everything can be taken from a man (sic) but one thing: the last of the human freedoms – to choose one’s attitude in any given set of circumstances, to choose one’s own way.” Viktor Frankl (Man’s Search for Meaning – p.104)
I realized that, especially at this time of year, attitude is everything–good, bad, indifferent–the choice is always yours.
How we choose to approach anything depends upon our previous experiences–what I call personal and situational bias. Sadler* has three classifications for these biases. He calls them value inertias (unwanted distorting influences which reflect background experience), ethical compromises (actions for which one is personally culpable), and cognitive limitations (not knowing for what ever reason).
When we approach an evaluation, our attitude leads the way. If we are reluctant, if we are resistant, if we are excited, if we are uncertain, all these approaches reflect where we’ve been, what we’ve seen, what we have learned, what we have done (or not). We can make a choice how to proceed.
The America n Evaluation Association (AEA) has long had a history of supporting difference. That value is imbedded in the guiding principles. The two principles which address supporting differences are
AEA also has developed a Cultural Competence statement. In it, AEA affirms that “A culturally competent evaluator is prepared to engage with diverse segments of communities to include cultural and contextual dimensions important to the evaluation. Culturally competent evaluators respect the cultures represented in the evaluation.”
Both of these documents provide a foundation for the work we do as evaluators as well as relating to our personal and situational bias. Considering them as we enter into the choice we make about attitude will help minimize the biases we bring to our evaluation work. The evaluative question from all this–When has your personal and situational biases interfered with you work in evaluation?
Attitude is always there–and it can change. It is your choice.
Sadler, D. R. (1981). Intuitive data processing as a potential source of bias in naturalistic evaluations. Education Evaluation and Policy Analysis, 3, 25-31.
Hopefully, the technical difficulties with images is no longer a problem and I will be able to post the answers to the history quiz and the post I had hoped to post last week. So, as promised, here are the answers to the quiz I posted the week of July 5. The keyed responses are in BOLD
7. James W. Altschuldt is the go-to person for needs assessment. He is the editor of the Needs Assessment Kit (or everything you wanted to know about needs assessment and didn’t know where to find the answer). He is also the co-author with Bell Ruth Witkin of two needs assessment books, and .
11. Ellen Taylor-Powell, the former Evaluation Specialist at University of Wisconsin Extension Service and is credited with developing the logic model later adopted by the USDA for use by the Extension Service. To go to the UWEX site, click on the words “logic model”.
15. Thomas A. Schwandt, a philosopher at heart who started as an auditor, has written extensively on evaluation ethics. He is also the co-author (with Edward S. Halpern) of Linking Auditing and Metaevaluation.
19. William R. Shadish co-edited (with Laura C. Leviton and Thomas Cook) of Foundations of Program Evaluation: Theories of Practice . His work in theories of evaluation practice earned him the Paul F. Lazarsfeld Award for Evaluation Theory, from the American Evaluation Association in 1994.
Although I’ve only list 20 leaders, movers and shakers, in the evaluation field, there are others who also deserve mention: John Owen, Deb Rog, Mark Lipsey, Mel Mark, Jonathan Morell, Midge Smith, Lois-Ellin Datta, Patricia Rogers, Sue Funnell, Jean King, Laurie Stevahn, John, McLaughlin, Michale Morris, Nick Smith, Don Dillman, Karen Kirkhart, among others.
If you want to meet the movers and shakers, I suggest you attend the American Evaluation Association annual meeting. In 2011, it will be held in Anaheim CA, November 2 – 5; professional development sessions are being offered October 31, November 1 and 2, and also November 6. More conference information can be found here.
What are standard evaluation tools? What knowledge do you need to conduct an evaluation effectively and efficiently? For this post and the next two, I’m going to talk about just that.
Today–program planning. How does program planning relate to program evaluation?
A lot of hours goes into planning a program. Questions that need to be answered among others include:
Although you might think that these are planning questions, they are also evaluation questions. They point the program planner to the outcome of the program in the context in which the program is planned. Yet, what often happens is that evaluation is often left out of that planning. It is one detail that gets lost in all the rest–until the end. Unfortunately, retrofitting an evaluation after the program has already run often results in spurious data, leading to specious results, resulting in unusable findings and unfortunately–a program that can’t be replicated. What’s an educator to do?
The tools that help in program planning are ones you have seen and probably used before: logic models, theories of change, and evaluation proposals.
Logic models have already been the topic of this blog. Theories of change have been mentioned. Evaluation proposals are a new topic. More and more, funding agencies want an evaluation plan. Some provide a template–often a modified logic model; some ask specifically for a program specific logic mode. Detailing how your program will bring about change and what change is expected is all part of an evaluation proposal. A review of logic models and theories of change and the program theory related to your proposed program will help you write an evaluation proposal.
Keep in mind that you may be writing for a naive audience, an audience who isn’t as knowledgeable as you in your subject matter OR in the evaluation process. A simple evaluation proposal will go a long way to getting and keeping all stakeholders on the same page.
Hi! It’s Tuesday, again.
I was thinking–If evaluation is an everyday activity, why does it FEEL so monumental–you know–over whelming, daunting, aversive even
I can think of several reasons for that feeling:
All those are good reasons. Yet, in today’s world you have to show your programs are making a difference. You have to provide evidence of impact. To do that (show impact, making a difference) you must evaluate your program.
How do you make your evaluation manageable? How do you make it an everyday activity? Here are several ways.
We can talk more about the how, later. Now it is enough to know that evaluation isn’t as monumental as you thought.