outcomes « Evaluation is an Everyday Activity

What? So what? Now what?

Sounds like an evaluation problem.

King and Stevahn (in press) tells us the first query requires thoughtful observation of a situation; the second query a discussion of possible options and implications of those options, and the third query calls for the creation of a list of potential next steps.

Yet these are the key words for “adaptive action” (If you haven’t looked at the web site, I suggest you do.) One quote that is reflective of adaptive action is, “Adaptive Action reveals how we can be proactive in managing today and influencing tomorrow.”( David W. Jamieson, University of St. Thomas). Adaptive action can help you

Understand the sources of uncertainty in your chaotic world
Explore opportunities for action and their implications as they occur
Learn a simple process that cuts through complexity
Transform the work of individuals, teams, organizations and communities
Take on any challenge—as large as a strategic plan or small as a messy meeting
Take action to improve productivity, collaboration and sustainability

Evaluation is a proactive (usually) activity (oh, I know that sometimes evaluation is flying by the seat of your pants and is totally reactive). People are now recognizing that evaluation will benefit them, their programs, and their organizations and that it isn’t personal (although that fear is still out there).

Although the site is directed towards leadership in organizations, the key questions are evaluative. You can’t determine “what” without evidence (data); you can’t determine “so what” unless you have a plan (logic model), and you can’t think about “now what” unless you have an outcome that you can move toward. These questions are evaluative in contemporary times because there are no simple problems any more. (Panarchy approaches similar situations using a similar model .) Complex situations are facing program people and evaluators all the time. Using adaptive action may help. Panarchy may help (the book is called Panarchy by Gunderson and Hollings .)

Just think of adaptive action as another model of evaluation.

molly.

Many of you have numerous lists for summer reading (NY Times, NPR, Goodreads, Amazon, others…). My question is what are you reading to further your knowledge about evaluation? Perhaps you are; perhaps you’re not. So I’m going to give you one more list 🙂 …yes, it is evaluative.

If you want something light: Regression to the Mean by Ernest R. House. It is a novel. It is about evaluation. It explains what evaluators do from a political perspective.

If you want something qualitative: Qualitative Data Analysis by Matthew B. Miles, A. Michael Huberman, and Johnny Saldana. It is the new 3rd edition which Sage (the publisher) commissioned. A good thing, too, as both Miles and Huberman are no longer able to do a revision. My new go-to book.

If you want something on needs assessment: Bridging the Gap Between Asset/Capacity Building and Needs Assessment by James W. Altschuld. Bridging the Gap-altschuld Most needs assessments start with what is lacking (i.e., needed); this proposes that an assessment start with what is present (assets) and build from there, and in the process, meeting needs.

If you want something on higher education: College (Un)bound by Jeff Selingo. college unbound by jeffry selingo The state of higher education and some viable alternatives by a contributing editor at the Chronicle of Higher Education. Yes, it is evaluative.

Most of these I’ve mentioned before. I’ve read the above. I recommend them.

Continue reading →

A colleague asked, “How do you design an evaluation that can identify unintended consequences?” This was based on a statement about methodologies that “only measure the extent to which intended results have been achieved and are not able to capture unintended outcomes (see AEA365). (The cartoon is attributed to Rob Cottingham.)

Really good question. Unintended consequences are just that–outcomes which are not what you think will happen with the program you are implementing. This is where program theory comes into play. When you model the program, you think of what you want to happen. What you want to happen is usually supported by the literature, not your gut (intuition may be useful for unintended, however). A logic model lists as outcome the “intended” outcomes (consequences). So you run your program and you get something else, not necessarily bad, just not what you expected; the outcome is unintended.

Program theory can advise you that other outcomes could happen. How do you design your evaluation so that you can capture those. Mazmanian in his 1998 study on intention to change had an unintended outcome; one that has applications to any adult learning experience (1). So what method do you use to get at these? A general question, open ended? Perhaps. Many (most?) people won’t respond to open ended questions–takes too much time. OK. I can live with that. So what do you do instead? What does the literature say could happen? Even if you didn’t design the program for that outcome. Ask that question. Along with the questions about what you expect to happen.

How would you represent this in your logic model–by the ubiquitous “other”? Perhaps. Certainly easy that way. Again, look at program theory. What does it say? Then use what is said there. Or use “other”–then you are getting back to the open ended questions and run the risk of not getting a response. If you only model “other”–do you really know what that “other” is?

I know that I won’t be able to get to world peace, so I look for what I can evaluate and since I doubt I’ll have enough money to actually go and observe behaviors (certainly the ideal), I have to ask a question. In your question asking, you want a response right? Then ask the specific question. Ask it in a way that elicits program influence–how confident the respondent is that X happened? How confident the respondent is that they can do X? How confident is the respondent that this outcome could have happened? You could ask if X happened (yes/no) and then ask the confidence questions (confidence questions are also known as self-efficacy). Bandura will be proud. See OR OR (for discussions of self-efficacy and social learning).

molly.

1. Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., Kantrowitz, M. P. (1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine 73(8), 882-886.

On May 9, 2014, Dr. Don Kirkpatrick died at the age of 90. His approach (called a model) to evaluation was developed in 1954 and has served the training and development arena well since then; it continues to do so.

For those of you who are not familiar with the Kirkpatrick model, here is a primer, albeit short. (There are extensive training programs for getting certified in this model, if you want to know more.)

Don Kirkpatrick, Ph. D. developed the Kirkpatrick model when he was a doctoral student; it was the subject of his dissertation which was defended in 1954. There are four levels (they are color coded on the Kirkpatrick website) and I quote:

Level 1: Reaction

To what degree participants react favorably to the training

Level 2: Learning

To what degree participants acquire the intended knowledge, skills, attitudes, confidence and commitment based on their participation in a training event

Level 3: Behavior

To what degree participants apply what they learned during training when they are back on the job

Level 4: Results

To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement

Sounds simple, right. (Reminiscent of a logic model’s short, medium, and long term outcomes). He was the first to admit that it is difficult to get to level four (no world peace for this guy, unfortunately). We all know that behavior can be observed and reported, although self-report is fraught with problems (self-selection, desired response, other cognitive bias, etc.). Continue reading →

Intention to change

I’ve talked about intention to change and how stating that intention out loud and to others makes a difference. This piece of advice is showing up in some unexpected places and here. If you state your goal, there is a higher likelihood that you will be successful. That makes sense. If you confess publicly (or even to a priest), you are more likely to do the penance/make a change. What I find interesting is that this is so evaluation. What difference did the intervention make? How does that difference relate to the merit, worth, value of the program?

Lent started March 5. That is 40 days of discipline–giving up or taking on. That is a program. What difference will it make? Can you go 40 days without chocolate?

New Topic:

I got my last comment in November, 2013. I miss comments. Sure most of them were check out this other web site. Still there were some substantive comments and I’ve read those and archived them. My IT person doesn’t know what was the impetus for this sudden stop. Perhaps Google changed its search engine optimization code and my key words are no longer in the top. So I don’t know if what I write is meaningful; is worthwhile; or is resonating with you the reader in any way. I have been blogging now for over four years…this is no easy task. Comments and/or questions would be helpful, give me some direction.

New Topic:

Chris Lysy cartoons in his blog. This week he blogged about logic models. He only included logic models that are drawn with boxes. What if the logic model is circular? How would it be different? Can it still lead to outcomes? Non-linear thinkers/cultures would say so. How would you draw it? Given that mind mapping may also be a model, how do they relate?

Have a nice weekend. The sun is shining again!

People often say one thing and do another.

This came home clearly to me with a nutrition project conducted with fifth and sixth grade students over the course of two consecutive semesters. We taught them nutrition and fitness and assorted various nutrition and fitness concepts (nutrient density, empty calories, food groups, energy requirements, etc.). We asked them at the beginning to identify which snack they would choose if they were with their friends (apple, carrots, peanut butter crackers, chocolate chip cookie, potato chips). We asked them at the end of the project the same question. They said they would choose an apple both pre and post. On the pretest, in descending order, the students would choose carrots, potato chips, chocolate chip cookies, and peanut butter crackers. On the post test, in descending order, the students would choose chocolate chip cookies, carrots, potato chips, and peanut butter crackers. (Although the sample sizes were reasonable [i.e., greater than 30], I’m not sure that the difference between 13.0% [potato chips] and 12.7% [peanut butter crackers] was significant. I do not have those data.) Then, we also asked them to choose one real snack. What they said and what they did was not the same, even at the end of the project. Cookies won, hands down in both the treatment and control groups. Discouraging to say the least; disappointing to be sure. What they said they would do and what they actually did were different.

Although this program ran from September through April, and is much longer than the typical professional development conference of a half day (or even a day), what the students said was different from what the students did. We attempted to measure knowledge, attitude, and behavior. We did not measure intention to change.

That experience reminded me of a finding of Paul Mazmanian . (I know I’ve talked about him and his work before; his work bears repeating.) He did a randomized controlled trial involving continuing medical education and commitment to change. After all, any program worth its salt will result in behavior change, right? So Paul Mazmanian set up this experiment involving doctors, the world’s worst folks with whom to try to change behavior.

He found that “…physicians in both the study and the control groups were significantly more likely to change (47% vs 7%, p<0.001) IF they indicated an INTENT (emphasis added in both cases) to change immediately following the lecture ” (i.e., the continuing education program). He did a further study and found that a signature stating that they would change didn’t increase the likelihood that they would change.

Bottom line, measure intention to change in evaluating your programs.

References:

Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowitz, M. P. (August 1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine, 73(8), 882-886.

Mazmanian, P. E., Johnson, R. E., Zhang, A., Boothby, J. & Yeatts, E. J. (June, 2001). Effects of a signature on rates of change: A randomized controlled trial involving continuing education and the commitment-to-change model. Academic Medicine, 76(6), 642-646.

Warning: This post may contain information that is controversial .

Schools (local public schools) were closed (still are).

The University (which never closes) was closed for four days (now open).

The snow kept falling and falling and falling. Snow in corvallis February 2014.jpg (Thank you Sandra Thiesen for the photo.)

Eighteen inches. Then freezing rain. It is a mess (although as I write this, the sun is shining, and it is 39F and supposed to get to 45F by this afternoon).

This is a complex messy system (thank you Dave Bella). It isn’t getting better. This is the second snow Corvallis has experienced in the same number of months, with increasing amounts.

It rains in the valley in Oregon; IT DOES NOT SNOW.

Another example of a complex messy system is what is happening in the UK.

These are examples extreme events; examples of climate chaos.

Evaluating complex messy systems is not easy. There are many parts. If you hold constant one part, what happens to the others? If you don’t hold constant one part, what happens to the rest of the system?. Systems thinking and systems evaluation has come of age with the 21st century; there were always people who viewed the world as a system; one part linked to another, indivisible. Soft systems theory dates back to at least von Bertalanffy who developed general systems theory and published the book by the same name in 1968 (ISBN 0-8076-0453-4).

One way to view systems is in this photo (compliments of Wikipedia) .

Evaluating systems is complicated and complex.

Bob Williams, along with Iraj Imam, edited the volume Systems Concepts in Evaluation (2007), and along with Richard Hummelbrunner, wrote the volume Systems Concepts in Action: A Practitioner’s Toolkit (2010). He is a leader in systems and evaluation.

These two books relate to my political statement at the beginning and complex messy systems. According to Amazon, the second book “explores the application of systems ideas to investigate, evaluate, and intervene in complex and messy situations”.

If you think your program works in isolation, think again. If you think your program doesn’t influence other programs, individuals, stakeholders, think again. You work in a complex messy system. Because you work in a complex messy system, you might want to simplify the situation (I know I do); only you can’t. You have to work within the system.

Might be worth while to get von Bertalanffy’s book; might be worth while to get Williams books; might be worth while to get a copy of Gunderson and Holling book Panarchy: Understanding Transformations in Systems of Humans and Nature.

After all, nature is a complex messy system.

On February 1 at 12:00 pm PT, I will be holding my annual virtual tea party. This is something I’ve been doing since February of 1993. I was in Minnesota and the winter was very cold, and although not as bleak as winter in Oregon, I was missing my friends who did not live near me. I had a tea party for the folks who were local and wanted to think that those who were not local were enjoying the tea party as well. So I created a virtual tea party. At that time, the internet was not available; all this was done in hard copy (to this day, I have one or two friends who do not have internet…sigh…). Today, the internet makes the tea party truly virtual–well the invitation is; you have to have a real cup of tea where ever you are.

How is this evaluative? Gandhi says that only you can be the change you want to see…this is one way you can make a difference. How will you know?

I know because my list of invitees has grown exponentially. And some of them share the invitation. They pass it on. I started with a dozen or so friends. Now my address list is over three pages long. Including my daughters and daughters of my friends (maybe sons, too for that matter…)

Other ways: Design an evaluation plan; develop a logic model; create a metric/rubric. Report the difference. This might be a good place for using an approach other than a survey or Likert scale. Think about it.

Did you know that there are at least 11 winter holidays besides Christmas–many of them related to light or the return of light.

One needs evaluation tools to determine the merit or worth, to evaluate the holiday’s value to you. For me, any that return light are important. So for me, there is Hanukkah (and eight candles), Solstice (and bonfires and yule logs), Christmas (and Advent wreaths with five candles), Kwanzaa ( and kinara seven candles). Sometimes Diwali falls late in November to be included (it is the ancient Hindu festival of lights that is a movable feast like Hanukkah).

I have celebrations for Hanukkah (I have several menorahs), for Solstice (I have two special candelabra that holds 12 candles–a mini-bonfire to be sure), for Advent/Christmas (I make a wreath each year), and for Kwanzaa (a handmade Kinara). And foods for each celebration as well. Because I live in a multicultural household, it is important that everyone understand that no holiday is more important than any other–all talk about returning light (literal or figurative). Sometimes the holidays over lap–Hanukkah, Solstice, Christmas all in the same week…phew, I’m exhausted just thinking about it. Sometimes it seems hard to keep them separate–then I realized that returning the light is not separate; it is light returning. It is an evaluative task.

So well come the new born sun/son…the light returns. Evaluation continues.

Happy Holidays…all of them!

I’m taking two weeks holiday–will see you in the new year.

Variables.

We all know about independent variables, and dependent variables. Probably even learned about moderator variables, control variables and intervening variables. Have you heard of confounding variables? Variables over which you have no (or very little) control. They present as a positive or negative correlation with the dependent and independent variable. This spurious relationship plays havoc with analyses, program outcomes, and logic models. You see them often in social programs.

Ever encounter one? (Let me know). Need an example? Here is one a colleague provided. There was a program developed to assist children removed from their biologic mothers (even though the courts typically favor mothers) to improve the children’s choices and chances of success. The program had included training of key stakeholders (including judges, social service, potential foster parents). The confounding variable that wasn’t taken into account was the sudden appearance of the biological father. Judges assumed that he was no longer present (and most of the time he wasn’t); social service established fostering without taking into consideration the presence of the biological father; potential foster parents were not allerted in their training of the possibility. Needless to say, the program failed. When biologic fathers appeared (as often happened), the program had no control over the effect they had. Fathers had not been included in the program’s equation.

Reviews.

Recently, I was asked to review a grant proposal, the award would result in several hundred thousand dollars (and in today’s economy, no small change). The PI’s passion came through in the proposal’s text. However, the PI and the PI’s colleagues did some major lumping in the text that confounded the proposed outcomes. I didn’t see how what was being proposed would result in what was said to happen. This is an evaluative task. I was charged to with evaluating the proposal on technical merit, possibility of impact (certainly not world peace), and achievability. The proposal was lofty and meant well. The likelihood that it would accomplish what it proposed was unclear, despite the PI’s passion. When reviewing a proposal, it is important to think big picture as well as small picture. Most proposals will not be sustainable after the end of funding. Will the proposed project be able to really make an impact (and I’m not talking here about world peace).

Conversations.

I attended a meeting recently that focused on various aspects of diversity. (Now among the confounding here is what does one mean by diversity; is it only the intersection of gender and race/ethnicity? Or something bigger, more?) One of the presenters talked about how just by entering into the conversation, the participants would be changed. I wondered, how can that change be measured? How would you know that a change took place? Any ideas? Let me know.

Focus groups.

A colleague asked whether a focus group could be conducted via email. I had never heard of such a thing (virtual, yes; email, no). Dick Krueger and Mary Ann Casey only talk about electronic reporting in their 4th edition of their Focus Group book. If I go to Wikipedia (keep in mind it is a wiki…), there is a discussion of online focus groups. Nothing offered about email focus groups. So I ask you, readers, is it a focus group if it is conducted by email?

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: outcomes

Adaptive action–evaluation by another name?

Summer Reading

Unintended? Unanticipated? Un…?

Kirkpatrick Model

Intention to change

Do what I say not what I do

Reality Check–complex messy systems

Virtual Tea Party

There are 11 winter holidays at least…

Variables and other things

Contact Info