Monthly Archives: December 2010

New Year’s Resolutions for 2011

Posted on by

The American Evaluation Association will celebrate the 25th anniversary of its founding in 2011.  Seems like a really good reason to declare 2011 the Year of Evaluation.  Consider it declared!

What can you do to celebrate the Year of Evaluation?  Here are a few suggestions that could be made into New Year’s Resolutions:

All kidding aside–below is a list of evaluation related suggestions that could be made into resolutions.

  1. Post a comment to this blog–a comment can be a question, an idea, an experience, or a thought…the idea is to get a conversation going;
  2. Consult with an evaluation specialist about an evaluation question;
  3. Join the American Evaluation Association–its a great resource and it is inexpensive–$80.00 for full members; $30.00 for students;
  4. Use your evaluation findings for something other than reporting your accountability;
  5. Read an evaluation reference–a thread on the AEA LinkedIn site recently asked for the top five evaluation references (I’ve added the link).
  6. Explore a facet of evaluation that you have not used before…always go to a survey to gather your data?  Try focus groups…always use quantitative approaches?  Try qualitative approaches?

Just some ideas–my resolution you ask??  I’ll  continue to blog weekly–please join me!

Happy New Year!

A bonus post :)

Posted on by

My creative effort this past year (other than my blog) has been to create new and (hopefully) wonderful pie.  This pie is vegetarian, not vegan, and obviously not dairy free…contains milk products and coconut.

Today, a bonus post–my gift to you:

WHITE CHRISTMAS PIE

You will need a 9 inch pie crust, fully baked.  (Although I make mine, getting one premade and following the directions for prebaking the crust will also work.)

Enough crushed peppermint candy to cover the pie crust that has cooled to room temperature.  Save about 1 tsp for garnish.

Melt over a double boiler, 12 ounces of white chocolate chips. A double boiler helps keep the chocolate warm and pourable.

Whip to soft peaks, 1 1/2 Cups of whipping cream. Stir in 1/8 tsp of mint or peppermint extract.

Continue whipping until firm peaks form.  SLOWLY fold into the cream, the cool white chocolate.  There will be layers of cooled chocolate throughout the cream.  That is the way it is supposed to be.

Spoon the chocolate mixture into the prepared pie crust.

Freeze for at least one hour or over night.

Prior to serving, remove pie from the freezer.

Whip 1 Cup of whipping cream until soft peaks form.

Add 1/4 tsp of mint or peppermint extract.

Sift into the cream, 2 Tbs of powdered sugar.

Continue whipping until stiff peaks form.

Spoon over the frozen pie, peaking the whipped cream to look like snow drifts.

Sprinkle with 1 Tbs unsweetened coconut and 1 tsp. crushed peppermint candy (that which you had left over above).

Cut into small slices.  Serves 12.  Happy Holidays!

Logic model revisited

Posted on by

My wishes to you:  Blessed Solstice.  Merry Christmas.  Happy Kwanzaa. and the Very Best Wishes for the New Year!

A short post today.

Ellen Taylor-Powell, my counterpart at University of Wisconsin Extension, has posted the following to the Extension Education Evaluation TIG list serve.  I think it is important enough to share here.

When you down load this PDF to save a copy, think of where your values come into the model; where others values can affect the program, and how you can modify the model to balance those values.

Ellen says:  “I just wanted to let everyone know that the online logic model course, “Enhancing Program Performance with Logic Models” has been produced as a PDF in response to requests from folks without easy or affordable internet access or with different learning needs.  The PDF version (216 pages, 3.35MB) is available at:

http://www.uwex.edu/ces/pdande/evaluation/pdf/lmcourseall.pdf

Please note that no revisions or updates have been made to the original 2003 online course.

Happy Holidays!

Ellen”

Twelve days TO Christmas

Posted on by

My older daughter (I have two–Morgan, the older, and Mersedes, the younger, ) suggested I talk about the evaluative activities around the holidays…hmmm.

Since I’m experiencing serious writers block this week, I thought I’d revisit evaluation as an everyday activity, with a holiday twist.

Keep in mind that the root of evaluation is from the French after the Latin is value (Oxford English Dictionary on line says:  [a. Fr. évaluation, f. évaluer, f. é- =es- (:{em} L. ex) out + value VALUE.]).


 Perhaps this is a good time to mention that the theme for Evaluation 2011 put forth by incoming AEA President, Jennifer Greene, is Values and Valuing in Evaluation.  I want to quote from her invitation letter, “…evaluation is inherently imbued with values.  Our work as evaluators intrinsically involves the process of valuing, as our charge is to make judgments (emphasis original) about the “goodness” or the quality, merit, or worth of a program.”

Let us consider the holidays “a program”. The Winter Holiday season starts (at least in the US and the northern hemisphere) with the  Thanksgiving holiday followed shortly thereafter by the first Sunday in Advent.  Typically this period of time includes at least the  following holidays:  St. Nicholas Day, Hanukkah, Winter Solstice, Christmas, Kwanzaa, Boxing Day, New Year’s, and Epiphany (I’m sure there are ones I didn’t list that are relevant).  This list typically takes us through January 6.  (I’m getting to the value part–stay with me…)

When I was a child, I remember the eager expectation of anticipating Christmas–none of the other holidays were even on my radar screen.  (For those of you who know me, you know how long ago that was…)  Then with great expectation (thank you, Charles),   I would go to bed and, as patiently as possible, await the moment when my father would turn on the tree lights, signaling that we children could descend to the living room.  Then poof!  That was Christmas. In 10 minutes it was done. The emotional bath I always took diminished greatly the value of this all important holiday.  Vowing that my children would grow up without the emotional bath of great expectations and dashed hopes, I choose to Celebrate the Season.  In doing so,  found value in the waiting of Advent, the majic of Hanukkah,  sharing of Kwanzaa, the mystery of Christmas and the traditions that come with all of these holidays.  There are other traditions that we revisit yearly, yet we find delight in remembering what the Winter Holiday traditions are and mean; remembering the foods we eat; the times we’ve shared.  From all this we find value in our program.  Do I still experience the emotional bath of childhood during this Holiday Season–not any more–and my children tell me that they like spreading the holidays out over the six week period.  I think this is the time of the year where we can take a second look at our programs (whether they are the holidays, youth development, watershed stewardship, nutrition education, or something else) and look for value in our programs–the part of the program that matters.  Evaluation is the work of capturing that value.  How we do that is what evaluation is all about.

Program theory

Posted on by

According to the counter on this blog, I’ve published 49 times.  Since last week was the one year anniversary of the inception of “Evaluation is an Everyday Activity”, which means 52 weeks, I missed a few weeks.  Not surprising with vacations, professional development, and writer’s block.  Today is a writer’s block day…I thought I’d do something about program theory.  I’m sure you are asking what has program theory to do with evaluating my program.   Let me explain…

An evaluation that is theory driven uses program theory as a tool to (according to Jody Fitzpatrick):

  1. understand the program to be evaluated
  2. guide the evaluation.

Pretty important contributions.  Faculty have often told me, I know my program’s good; everyone likes it.  But–

Can you describe the program theory that supports your program?

Huey Chen (1) defines program theory as, “a specification of what must be done to achieve the desired goals, what other important impacts may be anticipated, and how these goals and impacts would be generated.”  There are two parts of program theory:  normative theory and causative theory.  Normative theory (quoting Fitzpatrick) “…describes the program as it should be, its goals and outcomes, its interventions and the rationale for these, from the perspectives of various stakeholders.”  Causative theory, according to Fitzpatrick, “… makes use of existing research to describe the potential outcomes of the program based on characteristics of  the clients (read, target audience) and the program actions.  Using both normative and causative theories, one can develop a “plausible program model” or logic model.

Keep in mind that a “plausible program model” is only one of the possible models and the model  developed before implementation may need to change before the final evaluation.  Although anticipated outcomes are the ones you think will happen as a result of the program, Jonny Morell (2) provides a long list of programs where unanticipated outcomes happen before, during, and after the program implementation.  It might be a good idea to think of all potential outcomes–not just the ones you think might happen.  This is why program theory is important…to help you focus on the potential outcomes.

1.  Chen, H. (1990). Theory-driven evaluations.  Newbury Park, CA: Sage.

2.  Morell, J. A. (2010). Evaluation in the Face of Uncertainty. NY: Guilford Press.

Impacts and outcomes

Posted on by

There is an ongoing discussion about the difference between impact and outcome.  I think this is an important discussion because Extension professionals are asked regularly to demonstrate  the impact of their program.

There is no consensus about these terms.  They are often used interchangeably. Yet, the consensus is that they are not the same.  When Extension professionals plan an evaluation, it is important to keep these terms separate.  Their meaning is distinct and different.

So what exactly is IMPACT?

And what is an OUTCOME?

What points do we need to keep in mind when considering if the report we are making is a report of OUTCOMES or a report of IMPACTS.  Making explicit the meaning of these words before beginning the program is important.  If there is no difference in your mind, then that needs to be stated.  If there is a difference from your perspective, that needs to be stated as well.  It may all depend on who the audience is for the report.  Have you asked your supervisor (Staff Chair, Department Head, Administrator) what they mean by these terms?

One way to look at this issue is to go to simpler language:

  • What is the result (effect) of the intervention (read ‘program’)–that is, SO WHAT?  This is impact.
  • What is the intervention influence (affect) on the target audience–that is, WHAT HAPPENED?  This is outcome.

I would contend that impact is the effect (i.e., the result) and outcome is the affect (i.e., the influence).

Now to complicate this discussion a bit–where do OUTPUTS fit?

OUTPUTS are necessary and NOT sufficient to determine the influence (affect) or results (effect) of an intervention.  Outputs count things that were done–number of people trained; feet of stream bed reclaimed; number of curriculum written; number of…(fill in the blank).  Outputs do not tell you either the affect or the effect of the intervention.

The difference I draw may be moot if you do not draw the distinction.  If you don’t that is OK.  Just make sure that you are explicit with what you mean by these terms:  OUTCOMES and IMPACT.