Evaluation is an Everyday Activity « Program Evaluation Discussions

Last week, I spoke about how to questions and applying them to program planning, evaluation design, evaluation implementation, data gathering, data analysis, report writing, and dissemination. I only covered the first four of those topics. This week, I’ll give you my favorite resources for data analysis.

This list is more difficult to assemble. This is typically where the knowledge links break down and interest is lost. The thinking goes something like this. I’ve conducted my program, I’ve implemented the evaluation, now what do I do? I know my program is a good program so why do I need to do anything else?

YOU need to understand your findings. YOU need to be able to look at the data and be able to rigorously defend your program to stakeholders. Stakeholders need to get the story of your success in short clear messages. And YOU need to be able to use the findings in ways that will benefit your program in the long run.

Remember the list from last week? The RESOURCES for EVALUATION list? The one that says:

1. Contact your evaluation specialist.

2. Listen to stakeholders–that means including them in the planning.

3. Read

Good. This list still applies, especially the read part. Here are the readings for data analysis.

First, it is important to know that there are two kinds of data–qualitative (words) and quantitative (numbers). (As an aside, many folks think words that describe are quantitative data–they are still words even if you give them numbers for coding purposes, so treat them like words, not numbers).

Qualitative data analysis. When I needed to learn about what to do with qualitative data, I was given Miles and Huberman’s book. (Sadly, both authors are deceased so there will not be a forthcoming revision of their 2nd edition, although the book is still available.)

Citation: Miles, M. B., & Huberman, A. Michael. (1994). Qualitative data analysis: An expanded source book. Thousand Oaks, CA: Sage Publications.

Fortunately, there are newer options, which may be as good. I will confess, I haven’t read them cover to cover at this point (although they are on my to-be-read pile).

Citation: Saldana, J. (2009). The coding manual for qualitative researchers. Los Angeles, CA: Sage.

Bernard, H. R. & Ryan, G. W. (2010). Analyzing qualitative data. Los Angeles, CA: Sage.

If you don’t feel like tackling one of these resources, Ellen Taylor-Powell has written a short piece (12 pages in PDF format) on qualitative data analysis.

There are software programs for qualitative data analysis that may be helpful (Ethnograph, Nud*ist, others). Most people I know prefer to code manually; even if you use a soft ware program, you will need to do a lot of coding manually first.

Quantitative data analysis. Quantitative data analysis is just as complicated as qualitative data analysis. There are numerous statistical books which explain what analyses need to be conducted. My current favorite is a book by Neil Salkind.

Citation: Salkind, N. J. (2004). Statistics for people who (think they) hate statistics. (2nd ed. ). Thousand Oaks, CA: Sage Publications.

NOTE: there is a 4th ed. with a 2011 copyright available. He also has a version of this text that features Excel 2007. I like Chapter 20 (The Ten Commandments of Data Collection) a lot. He doesn’t talk about the methodology, he talks about logistics. Considering the logistics of data collection is really important.

Also, you need to become familiar with a quantitative data analysis software program–like SPSS, SAS, or even Excel. One copy goes a long way–you can share the cost and share the program–as long as only one person is using it at a time. Excel is a program that comes with Microsoft Office. Each of these has tutorials to help you.

A part of my position is to build evaluation capacity. This has many facets–individual, team, institutional.

One way I’ve always seen as building capacity is knowing where to find the answer to the how to questions. Those how to questions apply to program planning, evaluation design, evaluation implementation, data gathering, data analysis, report writing, and dissemination. Today I want to give you resources to build your tool box. These resources build capacity only if you use them.

RESOURCES for EVALUATION

1. Contact your evaluation specialist.

2. Listen to stakeholders–that means including them in the planning.

3. Read.

If you don’t know what to read to give you information about a particular part of your evaluation, see resource Number 1 above. For those of you who do not have the luxury of an evaluation specialist, I’m providing some reading resources below (some of which I’ve mentioned in previous blogs).

1. For program planning (aka program development): Ellen Taylor-Powell’s web site at the University of Wisconsin Extension. Her web site is rich with information about program planning, program development, and logic models.

2. For evaluation design and implementation: Jody Fitzpatrick”s book.

Citation: Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation: Alternative approaches and practical guidelines. (3rd ed.). Boston: Pearson Education, Inc.

3. For evaluation methods, that depends on the method you want to use for data gathering; it doesn’t cover the discussion of evaluation design, though.

For needs assessment, the books by Altschuld and Witkin (there are two).

(Yes, needs assessment is an evaluation activity).

Citation: Witkin, B. R. & Altschuld, J. W. (1995). Planning and conducting needs assessments: A practical guide. Thousand Oaks, CA: Sage Publications.

Citation: Altschuld, J. W. & Witkin B. R. (2000). From needs assessment to action: Transforming needs into solution strategies. Thousand Oaks, CA: Sage Publications, Inc.

For survey design: Don Dillman’s book.

Citation: Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys: The tailored design method. (3rd. ed.). Hoboken, New Jersey: John Wiley & Son, Inc.

For focus groups: Dick Krueger’s book.

Citation: Krueger, R. A. & Casey, M. A. (2000). Focus groups: A practical guide for applied research. (3rd. ed.). Thousand Oaks, CA: Sage Publications, Inc.

For case study: Robert Yin’s classic OR

Bob Brinkerhoff’s book.

Citation: Yin, R. K. (2009). Case study research: Design and methods. (4th ed.). Thousand Oaks, CA: Sage, Inc.

Citation: Brinkerhoff, R. O. (2003). the success case method: Find out quickly what’s working and what’s not. San Francisco: Berrett-Koehler Publishers, Inc.

For multiple case studies: Bob Stake’s book.

Citation: Stake, R. E. (2006). Multiple case study analysis. New York: The Guilford Press.

Since this post is about capacity building, a resource for evaluation capacity building:

Hallie Preskill and Darlene Russ-Eft’s book .

Citation: Preskill, H. & Russ-Eft, D. (2005). Building Evaluation Capacity: 72 Activities for teaching and training. Thousand Oaks, CA: Sage Publications.

I’ll cover reading resources for data analysis, report writing, and dissemination another time.

I’ve mentioned language use before.

I’ll talk about it today and probably again.

What the word–any word– means is the key to a successful evaluation.

Do you know what it means? Or do you think you know what it means?

How do you find out if what you think it means is what your key funder (a stakeholder) thinks it means? Or what the participants (target audience) thinks it means? Or any other stakeholder (partners, for example) thinks it means…

You ask them.

You ask them BEFORE the evaluation begins. You ask them BEFORE you have implemented the program. You ask them when you plan the program.

During program planning, I bring to the table relevant stakeholders–folks similar to and different from those who will be the recipients of the program. I ask them this evaluative question: “If you participated in this program, how will you know that the program is successful? What has to happen/change to know that a difference has been made?”

Try it–the answers are often revealing, informative, and enlightening. They are not often the answers you thought. Listen to those stakeholders. They have valuable insights. They actually know something.

Once you have those answers, clarify any and all terminology so that everyone is on the same page. What something means to you may means something completely different to someone else.

Impact is one of those words–it is both a noun and a verb. Be careful how you use it and how it is used. Go to a less loaded word–like results or effects. Talk about measurable results that occur within a certain time frame–immediately after the program; several months after the program; several years after the program–depending on your program. (If you are a forester, you may not see results for 40 years…)

Historically, April 15 is tax day (although in 2011, it is April 18 )–the day taxes are due to the revenue departments.

State legislatures are dealing with budgets and Congress is trying to balance a Federal budget.

Everywhere one looks, money is the issue–this is especially true in these recession ridden time. How does all this relate to evaluation, you ask? This is the topic for today’s blog. How does money figure into evaluation.

Let’s start with the simple and move to the complex. Everything costs–and although I’m talking about money, time, personnel, and resources (like paper, staples, electricity, etc.) must also be taken into consideration.

When we talk about evaluation, four terms typically come to mind: efficacy, effectiveness, efficiency, and fidelity.

Efficiency is the term that addresses money or costs. Was the program efficient in its use of resources? That is the question asked addressing efficiency.

To answer that question, there are three (at least) approaches that are used to address this question:

Cost or cost analysis;
Cost effectiveness analysis; and
Cost-benefit analysis.

Simply then:

Cost analysis is the number of dollars it takes to deliver the program, including salary of the individual(s) planning the program.
Cost effectiveness analysis is a computation of the target outcomes in an appropriate unit in ratio to the costs.
Cost-benefit analysis is also a ratio of the costs of outcomes to the benefits of the program measured in the same units, usually money.

How are these computed?

Cost can be measured by how much the consumer is willing to pay. Costs can be the value of each resource that is consumed in the implementation of the program. Or cost analysis can be “measuring costs so they can be related to procedures and outcomes” (Yates, 1996, p. 25). So you list the money spent to implement the program, including salaries, and that is a cost analysis. Simple.
Cost effectiveness analysis says that there is some metric in which the outcomes are measured (number of times hands are washed during the day, for example) and that is put in ratio of the total costs of the program. So movement from washing hands only once a day (a bare minimum) to washing hands at least six times a day would have the costs of the program (including salaries) divided by the changed number of times hands are washed a day (i.e., 5). The resulting value is the cost-effectiveness analysis. Complex.
Cost-benefit analysis puts the outcomes in the same metric as the costs–in this case dollars. The costs (in dollars) of the program (including salaries) are put in ratio to the outcomes (usually benefits) measured in dollars. The challenge here is assigning a dollar amount to the outcomes. How much is frequent hand washing worth? It is often measured in days saved from communicable/chronic/ acute illnesses. Computations of health days (reduction in days affected by chronic illness) is often difficult to value in dollars. There is a whole body of literature in health economics for this topic, if you’re interested. Complicated and complex.

Yates, B. T. (1996). Analyzing costs, procedures, processes, and outcomes in human services. Thousand Oaks, CA: Sage.

There has been a lot of buzz recently about the usefulness of the Kirkpatrick model.

I’ve been talking about it (in two previous posts) and so have others. This model has been around a long time and has continued to be useful in the training field. Extension does a lot of training. Does that mean this model should be used exclusively when training is the focus? I don’t think so. Does this model have merits. I think so. Could it be improved upon? That depends on the objective of your program and your evaluation, so probably.

If you want to know about whether your participants react favorably to the training, then this model is probably useful.

If you want to know about the change in knowledge, skills, attitudes, then this model may be useful. You would need to be careful because knowledge is a slippery concept to measure.

If you want to know about the change in behavior, probably not. Kirkpatrick on the website says that application of learning is what is measured in the behavioral stage. How do you observe behavior change at a training? Observation is the obvious answer here and one does not necessarily observe behavior change at a training. Intention to change is not mentioned in this level.

If you want to know what difference you made in the social, economic, and/or environmental conditions in which your participants live, work, and practice, then the Kirkpatrick model won’t take you there. The 4th level (which is where evaluation starts for this model, according to Kirkpatrick) says: To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement. I do not see this as condition change or what I call impact.

A faculty member asked me for specific help in assessing impact. First, one needs to define what is meant by impact. I use the word to mean change in social, environmental, and/or economic conditions over the long run. This means changes in social institutions like family, school, employment (social conditions). It means changes in the environment which may be clean water or clean air OR it may mean removing the snack food vending machine from the school (environmental conditions). It means changes in some economic indicator, up or down, like return on investment, change in employment status, or increase revenue (economic conditions). This doesn’t necessarily mean targeted outcomes of the training event.

I hope that any training event will move participants to a different place in their thinking and acting that will manifest in the LONG RUN in changes in one of the three conditions mentioned above. To get there, one needs to be specific in what one is asking the participants. Intention to change doesn’t necessarily get to impact. You could anticipate impact if participants follow through with their intention. The only way to know that for sure is to observe it. We approximate that by asking good questions.

What questions are you asking about condition change to get at impacts of your training and educational programs?

Next week: TIMELY TOPIC. Any suggestions?

You’ve developed your program. You think you’ve met a need. You conduct an evaluation. Low and behold! Some of your respondents give you such negative feedback you wonder what program they attended. Could it really have been your program?

This is the phenomena I call “all of the people all of the time”, which occurs regularly in evaluating training programs. And it has to do with use–what you do with the results of this evaluation. And you can’t do it–please all of the people all of the time, that is. There will always be some sour grapes. In fact, you will probably have more negative comments than positive comments. People who are upset want you to know; people are happy are just happy.

Now, I’m sure you are really confused. Good. At least I’ve got your attention and maybe you’ll read to the end of today’s post.

You have seen this scenario: You ask the participants for formative data so that you can begin planning the next event or program. You ask about the venue, the time of year, the length of the conference, the concurrent offerings, the plenary speakers. Although some of these data are satisfaction data (the first level, called Reaction, in Don Kirkpatrick’s training model and the Reaction category in Claude Bennett’s TOPs Hierarchy [see diagram]

they are important part of formative evaluation; an important part of program planning. You are using the evaluation report. That is important. You are not asking if the participants learned something. You are not asking if they intend to change their behavior. You are not asking about what conditions have changed. You only want to know about their experience in the program.

What do you do with the sour grapes? You could make vinegar, only that won’t be very useful and use is what you are after. Instead, sort the data into those topics over which you have some control and those topics over which you have no control. For example–you have control over who is invited to be a plenary speaker, if there will be a plenary speaker, how many concurrent sessions, who will teach those concurrent sessions; you have no control over the air handling at the venue, the chairs at the venue, and probably, the temperature of the venue.

You can CHANGE those topics over which you have control. Comments say the plenary speaker was terrible. Do not invite that person to speak again. Feedback says that the concurrent sessions didn’t provide options for classified staff, only faculty. Decide the focus of your program and be explicit in the program promotional materials–advertise it explicitly to your target audience. You get complaints about the venue–perhaps there is another venue; perhaps not.

You can also let your audience know what you decided based on your feedback. One organization for which I volunteered sent out a white paper with all the concerns and how the organization was addressing them–or not. It helped the grumblers see that the organization takes their feedback seriously.

And if none of this works…ask yourself: Is it a case of all of the people all of the time?

Although I have been learning about and doing evaluation for a long time, this week I’ve been searching for a topic to talk about. A student recently asked me about the politics of evaluation–there is a lot that can be said on that topic, which I will save for another day. Another student asked me about when to do an impact study and how to bound that study. Certainly a good topic, too, though one that can wait for another post. Something I read in another blog got me thinking about today’s post. So, today I want to talk about gathering demographics.

Last week, I mentioned in my TIMELY TOPIC post about the AEA Guiding Principles. Those Principles along with the Program Evaluation Standards make significant contributions in assisting evaluators in making ethical decisions. Evaluators make ethical decisions with every evaluation. They are guided by these professional standards of conduct. There are five Guiding Principles and five Evaluation Standards. And although these are not proscriptive, they go along way to ensuring ethical evaluations. That is a long introduction into gathering demographics.

The guiding principle, Integrity/Honesty states that “Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.” When we look at the entire evaluation process, as evaluators, we must strive constantly to maintain both personal and professional integrity in our decision making. One decision we must make involves deciding what we need/want to know about our respondents. As I’ve mentioned before, knowing what your sample looks like is important to reviewers, readers, and other stakeholders. Yet, if we gather these data in a manner that is intrusive, are we being ethical?

Joe Heimlich, in a recent AEA365 post, says that asking demographic questions “…all carry with them ethical questions about use, need, confidentiality…” He goes on to say that there are “…two major conditions shaping the decision to include – or to omit intentionally – questions on sexual or gender identity…”:

When such data would further our understanding of the effect or the impact of a program, treatment, or event.
When asking for such data would benefit the individual and/or their engagement in the evaluation process.

The first point relates to gender role issues–for example are gay men more like or more different from other gender categories? And what gender categories did you include in your survey? The second point relates to allowing an individual’s voice to be heard clearly and completely and have categories on our forms reflect their full participation in the evaluation. For example, does marital status ask for domestic partnerships as well as traditional categories and are all those traditional categories necessary to hear your participants?

The next time you develop a questionnaire that includes demographic questions, take a second look at the wording–in an ethical manner.

Hello, readers. This week I’m doing something different with this blog. This week, and the third week in each month from now on, I’ll be posting a column called Timely Topic. This will be a post on a topic that someone (that means you reader) has suggested. A topic that has been buzzing around in conversations. A topic that has relevance to evaluation. This all came about because a colleague from another land grant institution is concerned about the dearth of evaluation skills among Extension colleagues. (Although this comment makes me wonder to whom this colleague is talking, that question is content for another post, another day.) So thinking about how to get core evaluation information out to more folks, I decided to devote one post a month to TIMELY TOPICS. To day’s post is about “THINKING CAREFULLY”.

Recently, I’ve been asked to review a statistics text book for my department. This particular book uses a program that is available on everyone’s computer. The text has some important points to make and today’s post reflects one of those points. The point is thinking carefully about using statistics.

As an evaluator–if only the evaluator of your own programs–you must think critically about the “…context of the data, the source of the data, the method used in data collection, the conclusions reached, and the practical implications” (Triola, 2010, p. 18). The author posits that to understand general methods of using sample data; make inferences about populations; understand sampling and surveys; and important measures of key characteristics of data, as well as the use of valid statistical methods, one must recognize the misuse of statistics.

I’m sure all of you have heard the quote, “Figures don’t lie; liars figure,” which is attributed to Mark Twain. I’ve always heard the quote as “Statistics lie and liars use statistics.” Statistics CAN lie. Liars CAN use statistics. That is where thinking carefully comes in–to determine if the statistical conclusions being presented are seriously flawed.

As evaluators, we have a responsibility (according to the AEA guiding principles) to conduct systematic, data-based inquiry; provide competent performance; display honesty and integrity…of the entire evaluation process; respect the security, dignity, and self-worth of all respondents; and consider the diversity of the general and public interests and values. This demands that we think carefully about the reporting of data. Triola cautions, “Do not use voluntary response sample data for making conclusions about a population.” How often have you used data from individuals who decide themselves (self-selected) whether to participate in your survey or not? THINK CAREFULLY about your sample. These data cannot be generalized to all people like your respondents because of the bias that is introduced by self-selection.

Other examples of misuse of statistics include

using correlation for concluding causation;
reporting data that involves a sponsors product;
identifying respondents inappropriately;
reporting data that is affected with a desired response bias;
using small samples to draw conclusions for large groups;
implying that being precise is being accurate; and
reporting misleading or unclear percentages. (This cartoon was drawn by Ben Shabad.)

When reporting statistics gathered from your evaluation, THINK CAREFULLY.

Three weeks ago, I promised you a series of posts on related topics–Program planning, Evaluation implementation, monitoring and delivering, and Evaluation utilization. This is the third one–using the findings of evaluation.

Michael Patton’s book is my reference.

I’ll try to condense the 400+ page book down to 500+ words for today’s post. Fortunately, I have the Reader’s Digest version as well (look for Chapter 23 [Utilization-Focused Evaluation] in the following citation: Stufflebeam, D. L., Madaus, G. F. Kellaghan, T. (2000). Evaluation Models: Viewpoints on educational and human services evaluation, 2ed. Boston, MA: Kluwer Academic Publishers). Patton’s chapter is a good summary–still it is 14 pages.

To start, it is important to understand exactly how the word “evaluation” is used in the context of utilization. In the Stufflebeam, Madaus, & Kellaghan publication cited above, Patton (2000, p. 426) describes evaluation as, “the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness and/or inform decisions about future programming. Utilization-focused evaluation (as opposed to program evaluation in general) is evaluation done for and with specific intended primary users for specific, intended uses (emphasis added). ”

There are four different types of use–instrumental, conceptual, persuasive, and process. The interest of potential stakeholders cannot be served well unless the stakeholder(s) whose interests are being served is made explicit.

To understand the types of use, I will quote from a document titled, “Non-formal Educator Use of Evaluation Findings: Factors of Influence” by Sarah Baughman.

“Instrumental use occurs when decision makers use the findings to change or modify the program in some way (Fleisher & Christie, 2009; McCormick, 1997; Shulha & Cousins, 1997). The information gathered is used in a direct, concrete way or applied to a specific decision (McCormick, 1997).

Conceptual use occurs when the evaluation findings help the program staff or key stakeholders understand the program in a new way (Fleisher & Christie, 2009).

Persuasive use has also been called political use and is not always viewed as a positive type of use (McCormick, 1997). Examples of negative persuasive use include using evaluation results to justify or legitimize a decision that is already made or to prove to stakeholders or other administrative decision makers that the organization values accountability (Fleisher & Christie, 2009). It is sometimes considered a political use of findings with no intention to take the actual findings or the evaluation process seriously (Patton, 2008). Recently persuasive use has not been viewed as negatively as it once was.

Process use is the cognitive, behavioral, program, and organizational changes resulting, either directly or indirectly, from engagement in the evaluation process and learning to think evaluatively” (Patton, 2008, p. 109). Process use results not from the evaluation findings but from the evaluation activities or process.”

Before beginning the evaluation, the question, “Who is the primary intended user of the evaluation?” must not only be asked; it also must be answered. What stakeholders need to be at the table? Those are the people who have a stake in the evaluation findings and those stakeholders may be different for each evaluation. They are probably the primary intended users who will determine the evaluations use.

Citations mentioned in the Baughman quotation include:

Fleischer, D. N. & Christie, C. A. (2009). Evaluation use: Results from a survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30(2), 158-175
McCormick, E. R. (1997). Factors influencing the use of evaluation results. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 58, 4187 (UMI 9815051).
Shula, L. M. & Cousins, J. B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208.
Patton, M. Q. (2008). Utilization Focused Evaluation (4th ed.). Thousand Oaks: Sage Publications.

As promised last week, this week is (briefly) on implementation, monitoring, and delivering evaluation.

Implementation. To implement an evaluation, one needs to have a plan, often called a protocol. Typically, this is a step-by-step list of what you will do to present the program to your target audience. In presenting your program to your target audience, you will also include a step-by-step list of how you will gather evaluation information (data). What is important about the plan is that it be specific enough to be replicated by other interested parties. When a plan is developed, there is typically a specific design behind each type of data to be collected. For example, specific knowledge change is often measured by a pretest-posttest design; behavioral change is often measured with a repeated measures design. Campbell and Stanley, in their classic book, Experimental and quasi-experimental designs for research, present a wealth of information about designs that is useful in evaluation (as well as research).

There are numerous designs which will help develop the plan for the implementation of the program AND the evaluation.

Monitoring. Simply put, monitoring is watching to see if what you said would happen, actually does. Some people think of monitoring as . Although monitoring may seem like being watched, it is being watched with a plan. When I first finished my doctorate and became an evaluator, I conceptualized evaluation simply as process, progress, product. This helped stakeholders understand what evaluation was all about. The monitoring part of evaluation was answered when I asked, “Are we making progress? Are we where we said we would be at the time we said we would be there?” This is really important because sometimes, as Jonny Morell points out in his book, evaluation don’t always go as planned, even with the best monitoring system.

Delivering. Delivering is the nuts and bolts of what you are going to do. It addresses the who, what, where, when, how, and why of the implementation plan. All of these questions interrelate–for example, if you do not identify who will conduct the evaluation, often the evaluation is “squeezed in” at the end of a program because it is required.

In addition to answering these questions when delivering the evaluation, one thinks about the models, or evaluation approaches. Stufflebeam, Madaus, and Kellaghan (in Evaluation models: Viewpoints on educational and human services evaluation) discuss various approaches and state that the approach used by the evaluator will provide a framework for conducting an evaluation as well as presenting and using the evaluation results.

Evaluation is an Everyday Activity

Program Evaluation Discussions

More resources

How do you find the answer?

Language…what does it really mean and how do you know?

TIMELY TOPIC: IS IT ALL ABOUT MONEY?

Impact or not?

All of the people all of the time?

Demographics

TIMELY TOPIC: Thinking carefully

Evaluation Use

Implementation, monitoring, and delivering evaluation

Contact Info