A faculty member asked me to provide evaluation support for a grant application.  Without hesitation, I agreed.

I went to the web site for funding to review what was expected for an evaluation plan.  What was provided was their statement about why evaluation is important.

Although I agree with what is said in that discussion, I think we have a responsibility to go further.  Here is what I know.

Extension professionals evaluate programs because there needs to be some evidence that the imputs for the program–time, money, personnel, materials, facilities, etc.–are being used advantageously, effectively.  Yet, there is more to the question, “Why evaluate” than accountability. (Michael Patton talks about the various uses to which evaluation findings can be put–see his book on Utilization Focused Evaluation.) Programs are evaluated to determine if people are satisfied, if their expectations were met, whether the program was effective in changing something.

This is what I think.  None of what is stated above addresses the  “so what” part of “why evaluate”.  I think that answering this question (or attempting to) is a compelling reason to justify the effort of evaluating.  It is all very well and good to change people’s knowledge of a topic; it is all very well and good to change people’s behavior related to that topic; and it is all very well and good to have people intend to change (after all, stated intention to change is the best predictor of actual change).  Yet, it isn’t enough.  Being able to answer the “so what” question gives you more information.   And doing that–asking and answering the “so what” question–makes evaluation an everyday activity.   And, who knows.  It may even result in world peace.

My wishes to you:  Blessed Solstice.  Merry Christmas.  Happy Kwanzaa. and the Very Best Wishes for the New Year!

A short post today.

Ellen Taylor-Powell, my counterpart at University of Wisconsin Extension, has posted the following to the Extension Education Evaluation TIG list serve.  I think it is important enough to share here.

When you down load this PDF to save a copy, think of where your values come into the model; where others values can affect the program, and how you can modify the model to balance those values.

Ellen says:  “I just wanted to let everyone know that the online logic model course, “Enhancing Program Performance with Logic Models has been produced as a PDF in response to requests from folks without easy or affordable internet access or with different learning needs.  The PDF version (216 pages, 3.35MB) is available at:

http://www.uwex.edu/ces/pdande/evaluation/pdf/lmcourseall.pdf

Please note that no revisions or updates have been made to the original 2003 online course.

Happy Holidays!

Ellen”

There is an ongoing discussion about the difference between impact and outcome.  I think this is an important discussion because Extension professionals are asked regularly to demonstrate  the impact of their program.

There is no consensus about these terms.  They are often used interchangeably. Yet, the consensus is that they are not the same.  When Extension professionals plan an evaluation, it is important to keep these terms separate.  Their meaning is distinct and different.

So what exactly is IMPACT?

And what is an OUTCOME?

What points do we need to keep in mind when considering if the report we are making is a report of OUTCOMES or a report of IMPACTS.  Making explicit the meaning of these words before beginning the program is important.  If there is no difference in your mind, then that needs to be stated.  If there is a difference from your perspective, that needs to be stated as well.  It may all depend on who the audience is for the report.  Have you asked your supervisor (Staff Chair, Department Head, Administrator) what they mean by these terms?

One way to look at this issue is to go to simpler language:

  • What is the result (effect) of the intervention (read ‘program’)–that is, SO WHAT?  This is impact.
  • What is the intervention influence (affect) on the target audience–that is, WHAT HAPPENED?  This is outcome.

I would contend that impact is the effect (i.e., the result) and outcome is the affect (i.e., the influence).

Now to complicate this discussion a bit–where do OUTPUTS fit?

OUTPUTS are necessary and NOT sufficient to determine the influence (affect) or results (effect) of an intervention.  Outputs count things that were done–number of people trained; feet of stream bed reclaimed; number of curriculum written; number of…(fill in the blank).  Outputs do not tell you either the affect or the effect of the intervention.

The difference I draw may be moot if you do not draw the distinction.  If you don’t that is OK.  Just make sure that you are explicit with what you mean by these terms:  OUTCOMES and IMPACT.

September 25 – October 2 is Banned Book Week.

All of the books shown below have been or are banned.

and the American Library Association has once again published a list of banned or challenged books.  The September issue of the AARP Bulletin listed 50 banned books.  The Merriam Webster Dictionary was banned in a California elementary school in January 2010.

Yes, you say, so what?  How does that relate to program evaluation?

Remember the root of the work “evaluation” is value.  Someplace in the United States, some group used some criteria to “value” (or not) a book– to lodge a protest, successfully (or not), to remove a book from a library, school, or other source.  Establishing a criteria means that evaluation was taking place.  In this case, those criteria included being “too political,” having “too much sex,” being “irreligious,” being “socially offensive,” or some other criteria.   Some one, some place, some where has decided that the freedom to think for your self, the freedom to read, the importance of the First Amendment, the importance of free and open access to information are not important parts of our rights and they used evaluation to make that decision.

Although I don’t agree with censorship–I agree with the right that a person has to express her or his opinion as guaranteed by the First Amendment.  Yet in expressing an opinion, especially an evaluative opinion, an individual has a responsibility to express that opinion without hurting other people or property; to evaluate responsibly.

To aid evaluators to evaluate responsibly, the The American Evaluation Association has developed a set of five guiding principles for evaluators and even though you may not consider yourself a professional evaluator, considering these principals when conducting your evaluations is important and responsible.  The Guiding Principles are:

A. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries;

B. Competence: Evaluators provide competent performance to stakeholders;

C. Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process;

D.  Respect for People:  Evaluators respect the security, dignity, and self-worth of respondents, program participants, clients, and other evaluation stakeholders; and

E. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation.

I think free and open access to information is covered by principle D and E.  You may or may not agree with the people who used evaluation to challenge a book and in doing so used evaluation.  Yet, as someone who conducts evaluation, you have a responsibility to consider these principles, making sure that your evaluations respect people and are responsible for general and public welfare (in addition to employing systematic inquiry, competence, and integrity/honesty).  Now–go read a good (banned) book!

newyearresolution1 Ok–Christmas is over and now is the time to reflect on what needs to be different…self-deception, not with standing. I went looking for something salient to say today and found the following 10 reasons to NOT make new year’s resolutions posted on the Happy Lists (a blog about personal development and positive change for those who love lists found at http://happylists.wordpress.com/)

Do these make sense? Remember–you evaluate everyday.

1. They set you up to fail.

2. Everybody does it.

3. Losing weight should be a whole lifestyle change.

4. January is the wrong reason. T

5. Just make one.

6. There are better ways.

7. Bad economy.

8. No need for reminders.

9. You’re already stressed out enough.

10. They’re probably the same as last year.

I want to suggest one resolution that I’d like you to consider in 2010…one that will succeed…one that makes sense.

Learn one thing you didn’t know about evaluation.  Practice it.  If you need suggestions about the one thing, let me know (comment on the blog; email me if you want to remain anonymous on the blog; or call me–they all work.)

So if you’ve made some new year’s resolutions, throw them away. Make the changes in your life because you want to.

My wish for you is a wonderful 2010–make some part of your life better than 2009.

HAPPY NEW YEAR!!!

Merry Christmas–the greeting for the upcoming holiday–Hanukkah ended December 18 (I hope your was very happy–mine was); Solstice was last night (and the sun returned today–a feat in Oregon, in winter, so Solstice was truly blessed); kwanzaa 1

Kwanzaa won’t happen until Dec 26–and the greeting there is Habari Gani (Swahili for “What’s the news?”).

Now, how do I get an evaluation topic from that opening…hmmm…perhaps a gift…yes…a gift.

The gift I give y’all is this:

Think about your blessings.

Think about the richness of your life.

Think about those for whom you care.

And remember…even those thoughts are evaluative because you know how blessed you are; because you know how rich (we are not talking money here…) your life is; because you have people in your life for whom you care AND who care for you.

The light returns regardless of the tradition you follow, and that, too, is evaluative–because you can ask yourself is the light enough–and if it isn’t you CAN figure out how to solve that problem.

newyearresolution1 Next week, I’ll suggest some New Year’s  resolutions–evaluative, of course with no self-deception–you CAN do evaluation!