evaluation use « Evaluation is an Everyday Activity

We recently held Professional Development Days for the Division of Outreach and Engagement. This is an annual opportunity for faculty and staff in the Division to build capacity in a variety of topics. The question this training posed was evaluative:

How do we provide meaningful feedback?

Evaluating a conference or a multi-day, multi-session training is no easy task. Gathering meaningful data is a challenge. What can you do? Before you hold the conference (I’m using the word conference to mean any multi-day, multi-session training), decide on the following:

Are you going to evaluate the conference?
What is the focus of the evaluation?
How are you going to use the results?

The answer to the first question is easy: YES. If the conference is an annual event (or a regular event), you will want to have participants’ feedback of their experience, so, yes, you will evaluate the conference. Look at a Penn State Tip Sheet 16 for some suggestions. (If this is a one time event, you may not; though as an evaluator, I wouldn’t recommend ignoring evaluation.)

The second question is more critical. I’ve mentioned in previous blogs the need to prioritize your evaluation. Evaluating a conference can be all consuming and result in useless data UNLESS the evaluation is FOCUSED. Sit down with the planners and ask them what they expect to happen as a result of the conference. Ask them if there is one particular aspect of the conference that is new this year. Ask them if feedback in previous years has given them any ideas about what is important to evaluate this year.

This year, the planners wanted to provide specific feedback to the instructors. The instructors had asked for feedback in previous years. This is problematic if planning evaluative activities for individual sessions is not done before the conference. Nancy Ellen Kiernan, a colleague at Penn State, suggests a qualitative approach called a Listening Post. This approach will elicit feedback from participants at the time of the conference. This method involves volunteers who attended the sessions and may take more persons than a survey. To use the Listening Post, you must plan ahead of time to gather these data. Otherwise, you will need to do a survey after the conference is over and this raises other problems.

The third question is also very important. If the results are just given to the supervisor, the likelihood of them being used by individuals for session improvement or by organizers for overall change is slim. Making the data usable for instructors means summarizing the data in a meaningful way, often visually. There are several way to visually present survey data including graphs, tables, or charts. More on that another time. Words often get lost, especially if words dominate the report.

There is a lot of information in the training and development literature that might also be helpful. Kirkpatrick has done a lot of work in this area. I’ve mentioned their work in previous blogs.

There is no one best way to gather feedback from conference participants. My advice: KISS–keep it simple and straightforward.

I’ve talked about how each phase of a logic model has evaluative activities. I’ve probably even alluded to the fact that needs assessment is the evaluative activity for that phase called situation (see the turquoise area on the left end of the image below.)

What I haven’t done is talk about is the why, what, and how of needs assessment (NA). I also haven’t talked about the utilization of the findings of a needs assessment–what makes meaning of the needs assessment.

OK. So why is a NA conducted? And what is a NA?

Jim Altschuld is my go-to person when it comes to questions about needs assessment. He recently edited a series of books on the topic.

Although Jim is my go-to person, Belle Ruth Witkin (a colleague, friend, and collaborator of Jim Altschuld) says in the preface to the co-authored volume (Witkin and Altschuld, 1995–see below), that the most effective way to decide the best way to divide the (often scarce) resources among the demands (read programs) is to conduct a needs assessment when the planning for the use of those resources begins.

Book 1 of the kit discusses an overview. In that volume, Jim defines what a needs assessment is: “Needs assessment is the process of identifying needs, prioritizing them, making needs-based decisions, allocating resources, and implementing actions in organizations to resolve problems underlying important needs (pg.20).” Altschuld states that there are many models for assessing needs and provides citations for those models. I think the most important aspect of this first volume is the presentation of the phased model developed by Belle Ruth Witkin in 1984 and revised by Altschuld and Witkin in their 1995 and 2000 volumes.Those phases are preassessment, assessment, and postassessment. They divide those three phases into three levels, primary, secondary, and tertiary, each level targeting a different group of stakeholders. This volume also discusses the why and the how. Subsequent volumes go into more detail–volume 2 discusses phase 1 (getting started); volume 3 discusses phase II (collecting data); volume 4 discusses analysis and priorities; and volume 5 discusses phase III (taking action).

Laurie Stevahn and Jean A. King are the authors of this volume. In chapter 3, they discuss strategies for the action plan using facilitation procedures that promote positive relationships, develop shared understanding, prioritize decisions, and assess progress. They warn of interpersonal conflict and caution against roadblocks that impede change efforts. They also promote the development of evaluation activities at the onset of the NA because that helps ensure the use of the findings.

Needs assessment is a political experience. Some one (or ones) will feel disenfranchised, loose resources, have programs ended. These activities create hard feelings and resentments. These considerations need to be identified and discussed at the beginning of the process. It is like the elephant and the blind people–everyone has an image of what the creature is, there may or may not be consensus, yet for the NA to be successful, consensus is important. Without it, the data will sit on someone’s shelf or in someone’s computer. Not useful.

There has been a lot of buzz recently about the usefulness of the Kirkpatrick model.

I’ve been talking about it (in two previous posts) and so have others. This model has been around a long time and has continued to be useful in the training field. Extension does a lot of training. Does that mean this model should be used exclusively when training is the focus? I don’t think so. Does this model have merits. I think so. Could it be improved upon? That depends on the objective of your program and your evaluation, so probably.

If you want to know about whether your participants react favorably to the training, then this model is probably useful.

If you want to know about the change in knowledge, skills, attitudes, then this model may be useful. You would need to be careful because knowledge is a slippery concept to measure.

If you want to know about the change in behavior, probably not. Kirkpatrick on the website says that application of learning is what is measured in the behavioral stage. How do you observe behavior change at a training? Observation is the obvious answer here and one does not necessarily observe behavior change at a training. Intention to change is not mentioned in this level.

If you want to know what difference you made in the social, economic, and/or environmental conditions in which your participants live, work, and practice, then the Kirkpatrick model won’t take you there. The 4th level (which is where evaluation starts for this model, according to Kirkpatrick) says: To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement. I do not see this as condition change or what I call impact.

A faculty member asked me for specific help in assessing impact. First, one needs to define what is meant by impact. I use the word to mean change in social, environmental, and/or economic conditions over the long run. This means changes in social institutions like family, school, employment (social conditions). It means changes in the environment which may be clean water or clean air OR it may mean removing the snack food vending machine from the school (environmental conditions). It means changes in some economic indicator, up or down, like return on investment, change in employment status, or increase revenue (economic conditions). This doesn’t necessarily mean targeted outcomes of the training event.

I hope that any training event will move participants to a different place in their thinking and acting that will manifest in the LONG RUN in changes in one of the three conditions mentioned above. To get there, one needs to be specific in what one is asking the participants. Intention to change doesn’t necessarily get to impact. You could anticipate impact if participants follow through with their intention. The only way to know that for sure is to observe it. We approximate that by asking good questions.

What questions are you asking about condition change to get at impacts of your training and educational programs?

Next week: TIMELY TOPIC. Any suggestions?

You’ve developed your program. You think you’ve met a need. You conduct an evaluation. Low and behold! Some of your respondents give you such negative feedback you wonder what program they attended. Could it really have been your program?

This is the phenomena I call “all of the people all of the time”, which occurs regularly in evaluating training programs. And it has to do with use–what you do with the results of this evaluation. And you can’t do it–please all of the people all of the time, that is. There will always be some sour grapes. In fact, you will probably have more negative comments than positive comments. People who are upset want you to know; people are happy are just happy.

Now, I’m sure you are really confused. Good. At least I’ve got your attention and maybe you’ll read to the end of today’s post.

You have seen this scenario: You ask the participants for formative data so that you can begin planning the next event or program. You ask about the venue, the time of year, the length of the conference, the concurrent offerings, the plenary speakers. Although some of these data are satisfaction data (the first level, called Reaction, in Don Kirkpatrick’s training model and the Reaction category in Claude Bennett’s TOPs Hierarchy [see diagram]

they are important part of formative evaluation; an important part of program planning. You are using the evaluation report. That is important. You are not asking if the participants learned something. You are not asking if they intend to change their behavior. You are not asking about what conditions have changed. You only want to know about their experience in the program.

What do you do with the sour grapes? You could make vinegar, only that won’t be very useful and use is what you are after. Instead, sort the data into those topics over which you have some control and those topics over which you have no control. For example–you have control over who is invited to be a plenary speaker, if there will be a plenary speaker, how many concurrent sessions, who will teach those concurrent sessions; you have no control over the air handling at the venue, the chairs at the venue, and probably, the temperature of the venue.

You can CHANGE those topics over which you have control. Comments say the plenary speaker was terrible. Do not invite that person to speak again. Feedback says that the concurrent sessions didn’t provide options for classified staff, only faculty. Decide the focus of your program and be explicit in the program promotional materials–advertise it explicitly to your target audience. You get complaints about the venue–perhaps there is another venue; perhaps not.

You can also let your audience know what you decided based on your feedback. One organization for which I volunteered sent out a white paper with all the concerns and how the organization was addressing them–or not. It helped the grumblers see that the organization takes their feedback seriously.

And if none of this works…ask yourself: Is it a case of all of the people all of the time?

Although I have been learning about and doing evaluation for a long time, this week I’ve been searching for a topic to talk about. A student recently asked me about the politics of evaluation–there is a lot that can be said on that topic, which I will save for another day. Another student asked me about when to do an impact study and how to bound that study. Certainly a good topic, too, though one that can wait for another post. Something I read in another blog got me thinking about today’s post. So, today I want to talk about gathering demographics.

Last week, I mentioned in my TIMELY TOPIC post about the AEA Guiding Principles. Those Principles along with the Program Evaluation Standards make significant contributions in assisting evaluators in making ethical decisions. Evaluators make ethical decisions with every evaluation. They are guided by these professional standards of conduct. There are five Guiding Principles and five Evaluation Standards. And although these are not proscriptive, they go along way to ensuring ethical evaluations. That is a long introduction into gathering demographics.

The guiding principle, Integrity/Honesty states that “Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.” When we look at the entire evaluation process, as evaluators, we must strive constantly to maintain both personal and professional integrity in our decision making. One decision we must make involves deciding what we need/want to know about our respondents. As I’ve mentioned before, knowing what your sample looks like is important to reviewers, readers, and other stakeholders. Yet, if we gather these data in a manner that is intrusive, are we being ethical?

Joe Heimlich, in a recent AEA365 post, says that asking demographic questions “…all carry with them ethical questions about use, need, confidentiality…” He goes on to say that there are “…two major conditions shaping the decision to include – or to omit intentionally – questions on sexual or gender identity…”:

When such data would further our understanding of the effect or the impact of a program, treatment, or event.
When asking for such data would benefit the individual and/or their engagement in the evaluation process.

The first point relates to gender role issues–for example are gay men more like or more different from other gender categories? And what gender categories did you include in your survey? The second point relates to allowing an individual’s voice to be heard clearly and completely and have categories on our forms reflect their full participation in the evaluation. For example, does marital status ask for domestic partnerships as well as traditional categories and are all those traditional categories necessary to hear your participants?

The next time you develop a questionnaire that includes demographic questions, take a second look at the wording–in an ethical manner.

Hello, readers. This week I’m doing something different with this blog. This week, and the third week in each month from now on, I’ll be posting a column called Timely Topic. This will be a post on a topic that someone (that means you reader) has suggested. A topic that has been buzzing around in conversations. A topic that has relevance to evaluation. This all came about because a colleague from another land grant institution is concerned about the dearth of evaluation skills among Extension colleagues. (Although this comment makes me wonder to whom this colleague is talking, that question is content for another post, another day.) So thinking about how to get core evaluation information out to more folks, I decided to devote one post a month to TIMELY TOPICS. To day’s post is about “THINKING CAREFULLY”.

Recently, I’ve been asked to review a statistics text book for my department. This particular book uses a program that is available on everyone’s computer. The text has some important points to make and today’s post reflects one of those points. The point is thinking carefully about using statistics.

As an evaluator–if only the evaluator of your own programs–you must think critically about the “…context of the data, the source of the data, the method used in data collection, the conclusions reached, and the practical implications” (Triola, 2010, p. 18). The author posits that to understand general methods of using sample data; make inferences about populations; understand sampling and surveys; and important measures of key characteristics of data, as well as the use of valid statistical methods, one must recognize the misuse of statistics.

I’m sure all of you have heard the quote, “Figures don’t lie; liars figure,” which is attributed to Mark Twain. I’ve always heard the quote as “Statistics lie and liars use statistics.” Statistics CAN lie. Liars CAN use statistics. That is where thinking carefully comes in–to determine if the statistical conclusions being presented are seriously flawed.

As evaluators, we have a responsibility (according to the AEA guiding principles) to conduct systematic, data-based inquiry; provide competent performance; display honesty and integrity…of the entire evaluation process; respect the security, dignity, and self-worth of all respondents; and consider the diversity of the general and public interests and values. This demands that we think carefully about the reporting of data. Triola cautions, “Do not use voluntary response sample data for making conclusions about a population.” How often have you used data from individuals who decide themselves (self-selected) whether to participate in your survey or not? THINK CAREFULLY about your sample. These data cannot be generalized to all people like your respondents because of the bias that is introduced by self-selection.

Other examples of misuse of statistics include

using correlation for concluding causation;
reporting data that involves a sponsors product;
identifying respondents inappropriately;
reporting data that is affected with a desired response bias;
using small samples to draw conclusions for large groups;
implying that being precise is being accurate; and
reporting misleading or unclear percentages. (This cartoon was drawn by Ben Shabad.)

When reporting statistics gathered from your evaluation, THINK CAREFULLY.

Three weeks ago, I promised you a series of posts on related topics–Program planning, Evaluation implementation, monitoring and delivering, and Evaluation utilization. This is the third one–using the findings of evaluation.

Michael Patton’s book is my reference.

I’ll try to condense the 400+ page book down to 500+ words for today’s post. Fortunately, I have the Reader’s Digest version as well (look for Chapter 23 [Utilization-Focused Evaluation] in the following citation: Stufflebeam, D. L., Madaus, G. F. Kellaghan, T. (2000). Evaluation Models: Viewpoints on educational and human services evaluation, 2ed. Boston, MA: Kluwer Academic Publishers). Patton’s chapter is a good summary–still it is 14 pages.

To start, it is important to understand exactly how the word “evaluation” is used in the context of utilization. In the Stufflebeam, Madaus, & Kellaghan publication cited above, Patton (2000, p. 426) describes evaluation as, “the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness and/or inform decisions about future programming. Utilization-focused evaluation (as opposed to program evaluation in general) is evaluation done for and with specific intended primary users for specific, intended uses (emphasis added). ”

There are four different types of use–instrumental, conceptual, persuasive, and process. The interest of potential stakeholders cannot be served well unless the stakeholder(s) whose interests are being served is made explicit.

To understand the types of use, I will quote from a document titled, “Non-formal Educator Use of Evaluation Findings: Factors of Influence” by Sarah Baughman.

“Instrumental use occurs when decision makers use the findings to change or modify the program in some way (Fleisher & Christie, 2009; McCormick, 1997; Shulha & Cousins, 1997). The information gathered is used in a direct, concrete way or applied to a specific decision (McCormick, 1997).

Conceptual use occurs when the evaluation findings help the program staff or key stakeholders understand the program in a new way (Fleisher & Christie, 2009).

Persuasive use has also been called political use and is not always viewed as a positive type of use (McCormick, 1997). Examples of negative persuasive use include using evaluation results to justify or legitimize a decision that is already made or to prove to stakeholders or other administrative decision makers that the organization values accountability (Fleisher & Christie, 2009). It is sometimes considered a political use of findings with no intention to take the actual findings or the evaluation process seriously (Patton, 2008). Recently persuasive use has not been viewed as negatively as it once was.

Process use is the cognitive, behavioral, program, and organizational changes resulting, either directly or indirectly, from engagement in the evaluation process and learning to think evaluatively” (Patton, 2008, p. 109). Process use results not from the evaluation findings but from the evaluation activities or process.”

Before beginning the evaluation, the question, “Who is the primary intended user of the evaluation?” must not only be asked; it also must be answered. What stakeholders need to be at the table? Those are the people who have a stake in the evaluation findings and those stakeholders may be different for each evaluation. They are probably the primary intended users who will determine the evaluations use.

Citations mentioned in the Baughman quotation include:

Fleischer, D. N. & Christie, C. A. (2009). Evaluation use: Results from a survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30(2), 158-175
McCormick, E. R. (1997). Factors influencing the use of evaluation results. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 58, 4187 (UMI 9815051).
Shula, L. M. & Cousins, J. B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208.
Patton, M. Q. (2008). Utilization Focused Evaluation (4th ed.). Thousand Oaks: Sage Publications.

A faculty member asked me to provide evaluation support for a grant application. Without hesitation, I agreed.

I went to the web site for funding to review what was expected for an evaluation plan. What was provided was their statement about why evaluation is important.

Although I agree with what is said in that discussion, I think we have a responsibility to go further. Here is what I know.

Extension professionals evaluate programs because there needs to be some evidence that the imputs for the program–time, money, personnel, materials, facilities, etc.–are being used advantageously, effectively. Yet, there is more to the question, “Why evaluate” than accountability. (Michael Patton talks about the various uses to which evaluation findings can be put–see his book on Utilization Focused Evaluation.) Programs are evaluated to determine if people are satisfied, if their expectations were met, whether the program was effective in changing something.

This is what I think. None of what is stated above addresses the “so what” part of “why evaluate”. I think that answering this question (or attempting to) is a compelling reason to justify the effort of evaluating. It is all very well and good to change people’s knowledge of a topic; it is all very well and good to change people’s behavior related to that topic; and it is all very well and good to have people intend to change (after all, stated intention to change is the best predictor of actual change). Yet, it isn’t enough. Being able to answer the “so what” question gives you more information. And doing that–asking and answering the “so what” question–makes evaluation an everyday activity. And, who knows. It may even result in world peace.

My older daughter (I have two–Morgan, the older, and Mersedes, the younger, ) suggested I talk about the evaluative activities around the holidays…hmmm.

Since I’m experiencing serious writers block this week, I thought I’d revisit evaluation as an everyday activity, with a holiday twist.

Keep in mind that the root of evaluation is from the French after the Latin is value (Oxford English Dictionary on line says: [a. Fr. évaluation, f. évaluer, f. é- =es- (: L. ex) out + value VALUE.]).

Perhaps this is a good time to mention that the theme for Evaluation 2011 put forth by incoming AEA President, Jennifer Greene, is Values and Valuing in Evaluation. I want to quote from her invitation letter, “…evaluation is inherently imbued with values. Our work as evaluators intrinsically involves the process of valuing, as our charge is to make judgments (emphasis original) about the “goodness” or the quality, merit, or worth of a program.”

Let us consider the holidays “a program”. The Winter Holiday season starts (at least in the US and the northern hemisphere) with the Thanksgiving holiday followed shortly thereafter by the first Sunday in Advent. Typically this period of time includes at least the following holidays: St. Nicholas Day, Hanukkah, Winter Solstice, Christmas, Kwanzaa, Boxing Day, New Year’s, and Epiphany (I’m sure there are ones I didn’t list that are relevant). This list typically takes us through January 6. (I’m getting to the value part–stay with me…)

When I was a child, I remember the eager expectation of anticipating Christmas–none of the other holidays were even on my radar screen. (For those of you who know me, you know how long ago that was…) Then with great expectation (thank you, Charles), I would go to bed and, as patiently as possible, await the moment when my father would turn on the tree lights, signaling that we children could descend to the living room. Then poof! That was Christmas. In 10 minutes it was done. The emotional bath I always took diminished greatly the value of this all important holiday. Vowing that my children would grow up without the emotional bath of great expectations and dashed hopes, I choose to Celebrate the Season. In doing so, found value in the waiting of Advent, the majic of Hanukkah, sharing of Kwanzaa, the mystery of Christmas and the traditions that come with all of these holidays. There are other traditions that we revisit yearly, yet we find delight in remembering what the Winter Holiday traditions are and mean; remembering the foods we eat; the times we’ve shared. From all this we find value in our program. Do I still experience the emotional bath of childhood during this Holiday Season–not any more–and my children tell me that they like spreading the holidays out over the six week period. I think this is the time of the year where we can take a second look at our programs (whether they are the holidays, youth development, watershed stewardship, nutrition education, or something else) and look for value in our programs–the part of the program that matters. Evaluation is the work of capturing that value. How we do that is what evaluation is all about.

I’ve been writing for almost a year, 50 some columns. This week, before the Thanksgiving holiday, I want to share evaluation resources I’ve found useful and for which I am thankful. Although there are probably others with which I am not familiar, these are ones for which I am thankful.

My colleagues at UWEX, University of Wisconsin Extension Service, Ellen Taylor-Powell, and at Penn State Extension Service,

Nancy Ellen Kiernan,

both have resources that are very useful, easily accessed, clearly written. Ellen’s can be found at the Quick Tips site and Nancy Ellen’s can be found at her Tipsheets index. Both Nancy Ellen and Ellen have other links that may be useful as well. Access their sites through the links above.

Last week, I mentioned the American Evaluation Association. One of the important structures in AEA is the Topical Interest Groups (or TIGs). Extension has a TIG called the Extension Education Evaluation which helps organize Extension professionals who are interested or involved in evaluation. There is a wealth of information on the AEA web site. about the evaluation profession, access to the AEA elibrary, links to AEA on Facebook, Twitter, and LinkedIn. You do NOT have to be a member, to subscribe to blog, AEA365, which as the name suggests, is posted daily by different evaluators. Susan Kistler, AEA’s executive director, posts every Saturday. The November 20 post talks about the elibrary–check it out.

Many states and regions have local AEA affiliates. For example, OPEN, Oregon Program Evaluators Network, serves southern Washington and Oregon. It has an all volunteer staff who live mostly in Portland and Vancouver WA. The AEA site lists over 20 affiliates across the country, many with their own website. Those websites have information about connecting with local evaluators.

In addition to these valuable resources, National eXtension (say e-eXtension) has developed a community of practice devoted to evaluation and Mike Lambur, eXtension Evaluation and Research Leader, who can be reached at mike.lambur@extension.org. According to the web site, National eXtension “…is an interactive learning environment delivering the best, most researched knowledge from the smartest land-grant university minds across America. eXtension connects knowledge consumers with knowledge providers—experts like you who know their subject matter inside out.”

Happy Thanksgiving. Be safe.

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: evaluation use

Conference evaluation

Needs Assessment

Impact or not?

All of the people all of the time?

Demographics

TIMELY TOPIC: Thinking carefully

Evaluation Use

Why is evaluation important?

Twelve days TO Christmas

Evaluation Resources

Contact Info