evaluation use « Evaluation is an Everyday Activity

I am reading the book, Eaarth, by Bill McKibben (a NY Times review is here). He writes about making a difference in the world on which we live. He provides numerous examples that have all happened in the 21st century, none of them positive or encouraging. He makes the point that the place in which we live today is not, and never will be again, like the place in which we lived when most of us were born. He talks about not saving the Earth for our grandchildren but rather how our parents needed to have done things to save the earth for them–that it is too late for the grandchildren. Although this book is very discouraging, it got me thinking.

Isn’t making a difference what we as Extension professionals strive to do?

Don’t we, like McKibben, need criteria to determine what that difference can/could/would be made and look like?

And if we have that criteria well established, won’t we be able to make a difference, hopefully positive (think hand washing here)? And like this graphic, , won’t that difference be worth the effort we have put into the attempt? Especially if we thoughtfully plan how to determine what that difference is?

We might not be able to recover (according to McKibben, we won’t) the Earth the way it was when most of us were born; I think we can still make a difference–a positive difference–in the lives of the people with whom we work. That is an evaluative opportunity.

A colleague asks for advice on handling evaluation stories, so that they don’t get brushed aside as mere anecdotes. She goes on to say of the AEA365 blog she read, ” I read the steps to take (hot tips), but don’t know enough about evaluation, perhaps, to understand how to apply them.” Her question raises an interesting topic. Much of what Extension does can be captured in stories (i.e., qualitative data) rather than in numbers (i.e., quantitative data). Dick Krueger, former Professor and Evaluation Leader (read specialist) at the University of Minnesota has done a lot of work in the area of using stories as evaluation. Today’s post summarizes his work.

At the outset, Dick asks the following question: What is the value of stories? He provides these three answers:

Stories make information easier to remember
Stories make information more believable
Stories can tap into emotions.

There are all types of stories. The type we are interested in for evaluation purposes are organizational stories. Organizational stories can do the following things for an organization:

Depict culture
Promote core values
Transmit and reinforce the culture
Provide instruction to employees
Motivate, inspire, and encourage

He suggests six common types of organizational stories:

Hero stories (someone in the organization who has done something beyond the normal range of achievement)
Success stories (highlight organizational successes)
Lessons learned stories (what major mistakes and triumphs teach the organization)
“How it works around here” stories (highlight core organizational values reflected in actual practice
“Sacred bundle” stories (a collection of stories that together depict the culture of an organization; core philosophies)
Training and orientation stories (assists new employees in understanding how the organization works)

To use stories as evaluation, the evaluator needs to consider how stories might be used, that is, do they depict how people experience the program? Do they understand program outcomes? Do they get insights into program processes?

You (as evaluator) need to think about how the story fits into the evaluation design (think logic model; program planning). Ask yourself these questions: Should you use stories alone? Should you use stories that lead into other forma of inquiry? Should you use stories that augment/illustrate results from other forms of inquiry?

You need to establish criteria for stories. Rigor can be applied to story even though the data are narrative. Criteria include the following: Is the story authentic–is it truthful? Is the story verifiable–is there a trail of evidence back to the source of the story? Is there a need to consider confidentiality? What was the original intent–purpose behind the original telling? And finally, what does the story represent–other people or locations?

You will need a plan for capturing the stories. Ask yourself these questions: Do you need help capturing the stories? What strategy will you use for collecting the stories? How will you ensure documentation and record keeping? (Sequence the questions; write them down the type–set up; conversational; etc.) You will also need a plan for analyzing and reporting the stories as you, the evaluator, are responsible for finding meaning.

Last week I spoke about thinking like an evaluator by identifying the evaluative questions that you face daily. They are endless…Yet, doing this is hard, like any new behavior. Remember when you first learned to ride a bicycle? You had to practice before you got your balance. You had to practice a lot. The same is true for identifying the evaluative questions you face daily.

So you practice, maybe. You try to think evaluatively. Something happens along the way; or perhaps you don’t even get to thinking about those evaluative questions. That something that interferes with thinking or doing is resistance. Resistance is a Freudian concept that means that you directly or indirectly refuse to change your behavior. You don’t look for evaluative questions. You don’t articulate the criteria for value. Resistance usually occurs with anxiety about a new and strange situation. A lot of folks are anxious about evaluation–they personalize the process. And unless it is personnel evaluation, it is never about you. It is all about the program and the participants in that program.

What is interesting (to me at least) is that there is resistance at many different levels–the evaluator, the participant, the stakeholder (which may include the other two levels as well). Resistance may be active or passive. Resistance may be overt or covert. I’ve often viewed resistance as a 2×2 diagram. The rows are active or passive; the columns are overt or covert. So combining labels, resistance can be active overt, active covert, passive overt, passive covert. Now I know this is an artificial and socially constructed idea and may be totally erroneous. This approach helps me to make sense out of what I see when I go to meetings to help a content team develop their program and try to introduce (or not) evaluation in the process. I imagine you have seen examples of these types of resistance–maybe you’ve even demonstrated them. If so, then you are in good company–most people have demonstrated all of these types of resistance.

I bring up the topic of resistance now for two reasons.

1) Because I’ve just started a 17-month long evaluation capacity building program with 38 participants. Some of those participants were there because they were told to be there, and let me know their feelings about participating–what kind of resistance could they demonstrate? Some of those participants are there because they are curious and want to know–what kind of resistance could that be? Some of the participants just sat there–what kind of resistance could that be? Some of the participants did anything else while sitting in the program–what kind of resistance could that be? and

2) Because I will be delivering a paper on resistance and evaluation at the annual American Evaluation Association meeting in November. This is helping me organize my thoughts.

I would welcome your thoughts on this complex topic.

I was talking with a colleague about evaluation capacity building (see last week’s post) and the question was raised about thinking like an evaluator. Got me thinking about the socialization of professions and what has to happen to build a critical mass of like minded people.

Certainly, preparatory programs in academia conducted by experts, people who have worked in the field a long time–or at least longer than you starts the process. Professional development helps–you know, attending meetings where evaluators meet (like the upcoming AEA conference, U. S. regional affiliates [there are many and they have conferences and meetings, too], and international organizations [increasing in number–which also host conferences and professional development sessions]–let me know if you want to know more about these opportunities). Reading new and timely literature on evaluation provides insights into the language. AND looking at the evaluative questions in everyday activities. Questions such as: What criteria? What standards? Which values? What worth? Which decisions?

The socialization of evaluators happens because people who are interested in being evaluators look for the evaluation questions in everything they do. Sometimes, looking for the evaluative question is easy and second nature–like choosing a can of corn at the grocery store; sometimes it is hard and demands collaboration–like deciding on the effectiveness of an educational program.

My recommendation is start with easy things–corn, chocolate chip cookies, wine, tomatoes; move to harder things with more variables–what to wear when and where, or whether to include one group or another . The choices you make will all depend upon what criteria is set, what standards have been agreed upon, and what value you place on the outcome or what decision you make.

The socialization process is like a puzzle, something that takes a while to complete, something that is different for everyone, yet ultimately the same. The socialization is not unlike evaluation…pieces fitting together–criteria, standards, values, decisions. Asking the evaluative questions is an ongoing fluid process…it will become second nature with practice.

Last week, a colleague and I led two, 20 person cohorts in a two-day evaluation capacity building event. This activity was the launch (without the benefit of champagne ) of a 17-month long experience where the participants will learn new evaluation skills and then be able to serve as resources for their colleagues in their states. This training is the brain-child of the Extension Western Region Program Leaders group. They believe that this approach will be economical and provide significant substantive information about evaluation to the participants.

What Jim and I did last week was work to, hopefully, provide a common introduction to evaluation. The event was not meant to disseminate the vast array of evaluation information. We wanted everyone to have a similar starting place. It was not a train-the-trainer event, so common in Extension. The participants were at different places in their experience and understanding of program evaluation–some were seasoned, long-time Extension faculty, some were mid-career, some were brand new to Extension and the use of evaluation. All were Extension faculty from western states. And although evaluation can involve programs, policies, personnel, products, performance, processes, etc…these two days focused on program evaluation.

It occurred to me that it would be useful to talk about what is evaluation capacity building (ECB) and what resources are available to build capacity. Perhaps, the best place to start is with the Preskill and Russ-Eft book by the same name, Evaluation Capacity Building.

This volume is filled with summaries of evaluation points and there are activities to reinforce those points. Although this is a comprehensive resource, it covers key points briefly and there are other resources s that are valuable to understand the field of capacity building. For example, Don Compton and his colleagues, Michael Baizerman and Stacey Stockdill edited a New Directions in Evaluation volume (No. 93) that addresses the art, craft, and science of ECB. ECB is often viewed as a context-dependent system of processes and practices that help instill quality evaluation skills in an organization and its members. The long term outcome of any ECB is the ability to conduct a rigorous evaluation as part of routine practice. That is our long-term goal–conducting rigorous evaluations as a part of routine practice.

Although not exhaustive, below are some ECB resources and some general evaluation resources (some of my favorites, to be sure).

ECB resources:

Preskill, H. & Russ-Eft, D. (2005). Building Evaluation Capacity. Thousand Oaks, CA: Sage

Compton, D. W. Baizerman, M., & Stockdill, S. H. (Ed.). (2002). The art, craft, and science of evaluation capacity building. New Directions for Evaluation, No. 93. San Francisco: Jossey-Bass.

Preskill, H. & Boyle, S. (2008). A multidisciplinary model of evalluation capacity building. American Journal of Evaluation, 29 (4), 443-459.

General evaluation resources:

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (4th ed.). (2011). Program evaluation: Alternative approaches and practical guidelines. Boston, MA: Pearson.

Scriven, M. (4th ed.). (1991). Evaluation Thesaurus. Newbury Park, CA: Sage.

Patton, M. Q. (4th ed.). (2008). Utilization-focused evaluation. Thousand Oaks, CA: Sage.

Patton, M. Q. (2012). Essentials of utilization-focused evaluation. Thousand Oaks, CA: Sage

Hopefully, the technical difficulties with images is no longer a problem and I will be able to post the answers to the history quiz and the post I had hoped to post last week. So, as promised, here are the answers to the quiz I posted the week of July 5. The keyed responses are in BOLD

1. Michael Quinn Patton, author of Utilization-Focused Evaluation and the new book, Developmental Evaluation and the classic Qualitative Evaluation and Research Methods .

2. Michael Scriven is best known for his concept of formative and summative evaluation. He has also advocated that evaluation is a transdiscipline. He is the author of the Evaluation Thesaurus .

3. Hallie Preskill is the co-author (with Darlene Russ-Eft) of Evaluation Capacity Building.

4. Robert E. Stake has advanced work in case study and is the author of the book Multiple Case Study and The Art of Case Study Research.

5. David M. Fetterman is best known for his advocacy of empowerment evaluation and the book of that name, Foundations of Empowerment Evaluation .

6. Daniel Stufflebeam developed the CIPP (context input process product) model which is discussed in the book Evaluation Models .

7. James W. Altschuldt is the go-to person for needs assessment. He is the editor of the Needs Assessment Kit (or everything you wanted to know about needs assessment and didn’t know where to find the answer). He is also the co-author with Bell Ruth Witkin of two needs assessment books, and .

8. Jennifer C. Greene, the current President of the American Evaluation Association, and the author of a book on mixed methods .

9. Ernest R. House is a leader in the work of evaluation policy and is the author of an evaluation novel, Regression to the Mean .

10. Lee J. Cronbach is a pioneer in education evaluation and the reform of that practice. He co-authored with several associates the book, Toward Reform of Program Evaluation .

11. Ellen Taylor-Powell, the former Evaluation Specialist at University of Wisconsin Extension Service and is credited with developing the logic model later adopted by the USDA for use by the Extension Service. To go to the UWEX site, click on the words “logic model”.

12. Yvonna Lincoln, with her husband Egon Guba (see below) co-authored the book Naturalistic Inquiry . She is the currently co-editor (with Norman K. Denzin) of the Handbook of Qualitative Research .

13. Egon Guba, with his wife Yvonna Lincoln, is the co-author of 4th Generation Evaluation.

14. Blaine Worthen has championed certification for evaluators. He wit h Jody L. Fitzpatrick and James
R. Sanders have co-authored Program Evaluation: Alternative Approaches and Practical Guidelines.

15. Thomas A. Schwandt, a philosopher at heart who started as an auditor, has written extensively on evaluation ethics. He is also the co-author (with Edward S. Halpern) of Linking Auditing and Metaevaluation.

16. Peter H. Rossi, co-author with Howard E. Freeman and Mark E. Lipsey, wrote Evaluation: A Systematic Approach , and is a pioneer in evaluation research.

17. W. James Popham, a leader in educational evaluation, and authored the volume, Educational Evaluation

18. Jason Millman was a pioneer of teacher evaluation and author of Handbook of Teacher Evaluation

19. William R. Shadish co-edited (with Laura C. Leviton and Thomas Cook) of Foundations of Program Evaluation: Theories of Practice . His work in theories of evaluation practice earned him the Paul F. Lazarsfeld Award for Evaluation Theory, from the American Evaluation Association in 1994.

20. Laura C. Leviton (co-editor with Will Shadish and Tom Cook–see above) of Foundations of Program Evaluation: Theories of Practice has pioneered work in participatory evaluation.

Although I’ve only list 20 leaders, movers and shakers, in the evaluation field, there are others who also deserve mention: John Owen, Deb Rog, Mark Lipsey, Mel Mark, Jonathan Morell, Midge Smith, Lois-Ellin Datta, Patricia Rogers, Sue Funnell, Jean King, Laurie Stevahn, John, McLaughlin, Michale Morris, Nick Smith, Don Dillman, Karen Kirkhart, among others.

If you want to meet the movers and shakers, I suggest you attend the American Evaluation Association annual meeting. In 2011, it will be held in Anaheim CA, November 2 – 5; professional development sessions are being offered October 31, November 1 and 2, and also November 6. More conference information can be found here.

Those of you who read this blog know a little about evaluation. Perhaps you’d like to know more? Perhaps not…

I think it would be valuable to know who was instrumental in developing the profession to the point it is today; hence, a little history. This will be fun for those of you who don’t like history. It will be a matching game. Some of these folks have been mentioned in previous posts. I’ll post the keyed responses next week.

Directions: Match the name with the evaluation contribution. I’ve included photos so you know who is who, who you can put with a name and a contribution.

1. 2.

3. 4.

5. 6.

7. 8.

9. 10.

11. 12.

13. 14. 15.

16. 17. 18.

19. 20.

A. Michael Scriven 1. Empowerment Evaluation

B. Michael Quinn Patton 2. Mixed Methods

C. Blaine Worthen 3. Naturalistic Inquiry

D. David Fetterman 4. CIPP

E. Thomas Schwandt 5. Formative/Summative

F. Jennifer Greene 6. Needs Assessment

G. James W. Altschuld 7. Developmental Evaluation

H. Ernie House 8. Case study

I. Yvonna Lincoln 9. Fourth Generation Evaluation

J. Egon Guba 10. Evaluation Capacity Building

K. Lee J. Cronbach 11. Evaluation Research

L. W. James Popham 12. Teacher Evaluation

M. Peter H. Rossi 13. Logic Models

N. Hallie Preskill 14. Educational Evaluation

O. Ellen Taylor-Powell 15. Foundations of Program Evaluation

P. Robert Stake 16. Toward Reform of Program Evaluation

Q. Dan Stufflebeam 17. Participatory Evaluation

R. Jason Millman 18. Evaluation and Policy

S. Will Shadish 19. Evaluation and epistomology

T. Laura Leviton 20. Evaluation Certification

There are others more recent who have made contributions.These represent the folks who did seminal work that built the profession. It also includes some more recent thinkers. Have fun.

One of the opportunities I have as a faculty member at OSU is to mentor students. I get to do this in a variety of ways–sit on committees, provide independent studies, review preliminary proposal, listen…I find it very exciting to see the change and growth in students’ thinking and insights when I work with students. I get some of my best ideas from them. Like today’s post…

I just reviewed several chapters of student dissertation proposals. These students had put a lot of thought and passion into their research questions. To them, the inquiry was important; it could be the impetus to change. Yet, the quality of the writing often detracted from the quality of the question; the importance of the inquiry; the opportunity to make a difference.

How does this relate to evaluation? For evaluations to make a difference, the findings must be used. This does not mean writing the report and giving it to the funder, the principal investigator, the program leader, or other stakeholders. Too many reports have gathered dust on someone shelf because they are not used. In order to be used, the report must be written so that they can be understood. The report needs to be written to a naive audience; as though the reader knows nothing about the topic.

When I taught technical writing, I used the mnemonic of the 5Cs. My experience is that if these concepts (all starting with the letter ) were employed, the report/paper/manuscript would be able to be understood by any reader.

The report needs to be written:

Clearly
Coherently
Concisely
Correctly
Consistently

Clearly means not using jargon; using simple words; explaining technical words.

Coherently means having the sections of the report hang together; not having any (what I call) quantum leaps.

Concisely means using few words; avoiding long meandering paragraphs; avoiding the over use of prepositions (among other things).

Correctly means making sure that grammar and syntax are correct; subject/verb agreements; remembering that the word “data” is a plural word and takes a plural verb and plural articles.

Consistently means using the same word to describe the parts of your research; participants are participants all through the report, not subjects on page 5, respondents on page 11, and students on page 22.

This little mnemonic has helped many students write better papers; I know it can help many evaluators write better reports.

This is no easy task. Writing is hard work; using the 5Cs makes it easier.

I was putting together a reading list for an evaluation capacity building program I’ll be leading come September and was reminded about process evaluation. Nancy Ellen Kiernan has a one page handout on the topic. It is a good place to start. Like everything in evaluation, there is so much more to say. Let’s see what I can say in 440 words or less.

When I first started doing evaluation (back when we beat on hollow logs), I developed a simple approach (call it a model) so I could talk to stakeholders about what I did and what they wanted done. I called it the P3 model–Process, Progress, Product. This is a simple approach that answers the following evaluative questions:

How did I do what I did? (Process)
Did I do what I did in a timely manner? (Progress)
Did I get the outcome I wanted (Product)

It is the “how” question I’m going to talk about today.

Scriven, in the 4th ed of the Evaluation Thesaurus, says that a process evaluation “focuses entirely on the variables between input and output”. It may include input variables. Knowing this helps you know what the evaluative question is for the input and output parts of a logic model (remember there are evaluative questions/activities for each part of a logic model).

When considering evaluating a program, process evaluation is not sufficient; it may be necessary and still not be sufficient. An outcome evaluation must accompany a process evaluation. Evaluating process components of a program involves looking at internal and external communications (think memos, emails, letters, reports, etc.); interface with stakeholders (think meeting minutes); the formative evaluation system of a program (think participant satisfaction); and infrastructure effectiveness (think administrative patterns, implementation steps, corporate responsiveness; instructor availability, etc.).

Scriven provides these examples that suggest the need for program improvement: “…program’s receptionists are rude to most of a random selection of callers; the telephonists are incompetent; the senior staff is unhelpful to evaluators called in by the program to improve it; workers are ignorant about the reasons for procedures that are intrusive to their work patterns; or the quality control system lacks the power to call a halt to the process when it discerns an emergency.” Other examples which demonstrate program success are administrators are transparent about organizational structure; program implementation is inclusive; or participants are encouraged to provide ongoing feedback to program managers. We could then say that a process evaluation assesses the development and actual implementation of a program to determine whether the program was implemented as planned and whether expected output was actually produced.

Gathering data regarding the program as actually implemented assists program planners in identifying what worked and what did not. Some of the components included in a process evaluation are descriptions of program environment, program design, and program implementation plan. Data on any changes to the program or program operations and on any intervening events that may have affected the program should also be included.

Quite likely, these data will be qualitative in nature and will need to be coded using one of the many qualitative data analysis methods.

Hi everybody–it is time for another TIMELY TOPIC. This week’s topic is about using pretest/posttest evaluation or a post-then-pre evaluation.

There are many considerations for using these designs. You have to look at the end result and decide what is most appropriate for your program. Some of the key considerations include:

the length of your program;
the information you want to measure;
the factors influencing participants response; and
available resources.

Before explaining the above four factors, let me urge you to read on this topic. There are a couple of resources (yes, print…) I want to pass your way.

Campbell, D. T. & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Houghton Mifflin Company: Boston, MA. (The classic book on research and evaluation designs.)
Rockwell, S. K., & Kohn, H. (1989). Post-then-pre evaluation. Journal of Extension [On-line]. 27(2). Available at: http://www.joe.org/joe/1989summer/a5.htm (A seminal JoE paper explaining post-then-pre test.)
Nimon, K. Zigarami, D. & Allen, J. (2011). Measures of program effectiveness based on retrospective pretest data: Are all created equal? American Journal of Evaluation, 32, 8 – 28. (A 2011 publication with an extensive bibliography.)

Let’s talk about considerations.

Length of program.

For pre/post test, you want a program that is long. More than a day. Otherwise you risk introducing a desired response bias and the threats to internal validity that Campbell and Stanley identify. Specifically the threats called history, maturation, testing, and instrumentation, also a possible regression to the mean threat, though that is on a possible source of concern. These threats to internal validity assume no randomization and a one group design, typical for Extension programs and other educational programs. Post-then-pre works well for short programs, a day or less, and tend to control for response shift and desired response bias. There may still be threats to internal validity.

Information you want to measure.

If you want to know a participants specific knowledge, a post-then-pre cannot provide you with that information because you can not test something you cannot unknow. The traditional pre/post can focus on specific knowledge, e.g., what food is the highest in Vitamin C in a list that includes apricot, tomato, strawberry cantaloupe. (Answer: strawberry) If you are wanting agreement/disagreement with general knowledge (e.g., I know what the key components of strategic planning are), the post-pre works well. Confidence, behaviors, skills, and attitudes can all be easily measured with a post-then-pre.

Factors influencing participants response.

I mentioned threats to internal validity above. These factors all influence participants responses. If there is a long time between the pretest and the post test, participants can be affected by history (a tornado prevents attendance to the program); maturation (especially true with programs with children–they grow up); testing (having taken the pretest, the post test scores will be better); and instrumentation (the person administering the posttest administers it differently than the pretest was administered). Participants desire to please the program leader/evaluator, called desired response bias, also affects participants response.

Available resources.

Extension programs (as well as many other educational programs) are affected by the availability of resources (time, money, personnel, venue, etc.). If you only have a certain amount of time, or a certain number of people who can administer the evaluation, or a set amount of money, you will need to consider which approach to evaluation you will use.

The idea is to get usable, meaningful data that accurately reflects the work that went into the program.

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: evaluation use

TIMELY TOPIC: Making a difference

Stories as evaluation

Resistance

TIMELY TOPIC: Evaluation mind set

Evaluation Capacity Building

History Redux

A Little History

Technical writing

Process Evaluation

TIMELY TOPIC: PRE-POST or POST THEN PRE?

Contact Info