Evaluation is an Everyday Activity « Program Evaluation Discussions

What? So what? Now what?

Sounds like an evaluation problem.

King and Stevahn (in press) tells us the first query requires thoughtful observation of a situation; the second query a discussion of possible options and implications of those options, and the third query calls for the creation of a list of potential next steps.

Yet these are the key words for “adaptive action” (If you haven’t looked at the web site, I suggest you do.) One quote that is reflective of adaptive action is, “Adaptive Action reveals how we can be proactive in managing today and influencing tomorrow.”( David W. Jamieson, University of St. Thomas). Adaptive action can help you

Understand the sources of uncertainty in your chaotic world
Explore opportunities for action and their implications as they occur
Learn a simple process that cuts through complexity
Transform the work of individuals, teams, organizations and communities
Take on any challenge—as large as a strategic plan or small as a messy meeting
Take action to improve productivity, collaboration and sustainability

Evaluation is a proactive (usually) activity (oh, I know that sometimes evaluation is flying by the seat of your pants and is totally reactive). People are now recognizing that evaluation will benefit them, their programs, and their organizations and that it isn’t personal (although that fear is still out there).

Although the site is directed towards leadership in organizations, the key questions are evaluative. You can’t determine “what” without evidence (data); you can’t determine “so what” unless you have a plan (logic model), and you can’t think about “now what” unless you have an outcome that you can move toward. These questions are evaluative in contemporary times because there are no simple problems any more. (Panarchy approaches similar situations using a similar model .) Complex situations are facing program people and evaluators all the time. Using adaptive action may help. Panarchy may help (the book is called Panarchy by Gunderson and Hollings .)

Just think of adaptive action as another model of evaluation.

molly.

Many of you have numerous lists for summer reading (NY Times, NPR, Goodreads, Amazon, others…). My question is what are you reading to further your knowledge about evaluation? Perhaps you are; perhaps you’re not. So I’m going to give you one more list 🙂 …yes, it is evaluative.

If you want something light: Regression to the Mean by Ernest R. House. It is a novel. It is about evaluation. It explains what evaluators do from a political perspective.

If you want something qualitative: Qualitative Data Analysis by Matthew B. Miles, A. Michael Huberman, and Johnny Saldana. It is the new 3rd edition which Sage (the publisher) commissioned. A good thing, too, as both Miles and Huberman are no longer able to do a revision. My new go-to book.

If you want something on needs assessment: Bridging the Gap Between Asset/Capacity Building and Needs Assessment by James W. Altschuld. Bridging the Gap-altschuld Most needs assessments start with what is lacking (i.e., needed); this proposes that an assessment start with what is present (assets) and build from there, and in the process, meeting needs.

If you want something on higher education: College (Un)bound by Jeff Selingo. college unbound by jeffry selingo The state of higher education and some viable alternatives by a contributing editor at the Chronicle of Higher Education. Yes, it is evaluative.

Most of these I’ve mentioned before. I’ve read the above. I recommend them.

Continue reading →

Recently I came across some old note of mine, from some meeting several years ago. I though it would be useful in my writing so I saved it; actually there were two notes that were similar in content. They both relate to blogging, although at the time I didn’t know I would be blogging.

I lump them all under the title of taking a stand, although stance would probably be more descriptive.

The notes are these:

Know your audience.
Be proactive to anticipate needs.
Be reactive to meet needs.
Be authentic.
Be direct.
Be unapologetic.

Continue reading →

How do you approach evaluation?

Are you the expert?

Do you work in partnership?

Are you one of the group?

To which question did you answer yes?

If you are the expert and know the most (not everything, no one know everything [although teenagers think they do]), you are probably “doing to”. Extension has been “doing to” for most of its existence. Continue reading →

The US has been a country for 238 years. A long time. Perhaps it is an opportunity to reflect on what are the rights, privileges, and obligations of citizenship. Perhaps it is just another holiday. Perhaps it is just a time for blueberry pie and peach ice cream. Perhaps it is a…fill in the blank.

I’m not feeling particularly patriotic. I am feeling very evaluative. Recently I viewed a map indicating that on a US passport an individual could travel to 172 different countries. The only country passports which were more powerful (i.e., able to visit more countries) were UK, Finland and Sweden. I wonder to where (what country) can’t I travel on my US passport? That question requires evidence. That is evaluative. I value my US passport. My girls and I travel with them even though driver’s license would be easier. (Being able to fly to Paris at a Continue reading →

What makes a blog engaging?

We know that blogs and blogging outreach to community members–those who have subscribed as well as those using various search engines to find a topical response.

Do the various forms of accessing the blog make a difference in whether the reader is engaged?

This is not a casual question, dear Readers. I will be presenting a poster at the Engagement Scholarship Consortium in October (which will be held in Edmonton, Alberta). I want to know. I want to be able to present to the various audiences at that meeting what my readers think. I realize that reading evaluation blogs may yield a response that is different from reading blogs related to food, or sustainability, or food sustainability, or climate chaos, or parenthood, or some other topic. There are enough evaluation blogs populating the internet that I think that there is some interest. I think my readers are engaged. Continue reading →

A colleague asked, “How do you design an evaluation that can identify unintended consequences?” This was based on a statement about methodologies that “only measure the extent to which intended results have been achieved and are not able to capture unintended outcomes (see AEA365). (The cartoon is attributed to Rob Cottingham.)

Really good question. Unintended consequences are just that–outcomes which are not what you think will happen with the program you are implementing. This is where program theory comes into play. When you model the program, you think of what you want to happen. What you want to happen is usually supported by the literature, not your gut (intuition may be useful for unintended, however). A logic model lists as outcome the “intended” outcomes (consequences). So you run your program and you get something else, not necessarily bad, just not what you expected; the outcome is unintended.

Program theory can advise you that other outcomes could happen. How do you design your evaluation so that you can capture those. Mazmanian in his 1998 study on intention to change had an unintended outcome; one that has applications to any adult learning experience (1). So what method do you use to get at these? A general question, open ended? Perhaps. Many (most?) people won’t respond to open ended questions–takes too much time. OK. I can live with that. So what do you do instead? What does the literature say could happen? Even if you didn’t design the program for that outcome. Ask that question. Along with the questions about what you expect to happen.

How would you represent this in your logic model–by the ubiquitous “other”? Perhaps. Certainly easy that way. Again, look at program theory. What does it say? Then use what is said there. Or use “other”–then you are getting back to the open ended questions and run the risk of not getting a response. If you only model “other”–do you really know what that “other” is?

I know that I won’t be able to get to world peace, so I look for what I can evaluate and since I doubt I’ll have enough money to actually go and observe behaviors (certainly the ideal), I have to ask a question. In your question asking, you want a response right? Then ask the specific question. Ask it in a way that elicits program influence–how confident the respondent is that X happened? How confident the respondent is that they can do X? How confident is the respondent that this outcome could have happened? You could ask if X happened (yes/no) and then ask the confidence questions (confidence questions are also known as self-efficacy). Bandura will be proud. See OR OR (for discussions of self-efficacy and social learning).

molly.

1. Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., Kantrowitz, M. P. (1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine 73(8), 882-886.

“Engaged scholarship most commonly refers to a range of collaborative research, teaching, and learning initiatives rooted in sustained community-university partnerships and pursued across various disciplines and social and cultural contexts.” That is a commonly agreed upon definition of engaged scholarship. So let me ask you: Are blogs engaging?

If I tease apart the definition this way:

it is collaborative IF (and only if) you consider that reading a lot of other blogs is collaborative learning (someone writes; someone reads=collaboration);
that is sustained in a “partnership” that consists of other bloggers who are in the community and bloggers who are in the university;
that is pursued across various disciplines (remember, I read a lot of other blogs); and
that are in a variety of social and cultural contexts (remember, I read a lot of other blogs);

then I would say yes…and no…because what is really collaborative learning? Or collaborative teaching? (I’m certainly not doing research.) To me, collaboration is an agreed upon working together in an intellectual effort. I would guess that in this case there is tacit agreement; I write and the reader agrees to read either by subscription or random search engine optimization. Some how that feels really one sided–until I realize that I’m still getting comments on posts that I wrote two-three years ago. So something must be engaging–even if it is random SEO. (though I’m not sure it is collaborative in the usual sense of collaboration).

And since this is a blog about evaluation, my question is, “How do you know?” Chris Lysy has a cartoon about knowing that always makes me smile (actually several). .. Remember, you need a rubric to determine if something works you cannot work on gut impressions; only through rigorous evaluation can you determine if a program has merit, value, worth (the root of evaluation is value); and only if you have evidence do you know it works. Can I use comments (many over the last four years of this blog) to say the blog is engaging, that I have evidence? One comment received in the last week suggested that since folks are still commenting on this post, perhaps that is evidence….don’t know.

Any thoughts–comments are welcome; please.

My .

molly.

On May 9, 2014, Dr. Don Kirkpatrick died at the age of 90. His approach (called a model) to evaluation was developed in 1954 and has served the training and development arena well since then; it continues to do so.

For those of you who are not familiar with the Kirkpatrick model, here is a primer, albeit short. (There are extensive training programs for getting certified in this model, if you want to know more.)

Don Kirkpatrick, Ph. D. developed the Kirkpatrick model when he was a doctoral student; it was the subject of his dissertation which was defended in 1954. There are four levels (they are color coded on the Kirkpatrick website) and I quote:

Level 1: Reaction

To what degree participants react favorably to the training

Level 2: Learning

To what degree participants acquire the intended knowledge, skills, attitudes, confidence and commitment based on their participation in a training event

Level 3: Behavior

To what degree participants apply what they learned during training when they are back on the job

Level 4: Results

To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement

Sounds simple, right. (Reminiscent of a logic model’s short, medium, and long term outcomes). He was the first to admit that it is difficult to get to level four (no world peace for this guy, unfortunately). We all know that behavior can be observed and reported, although self-report is fraught with problems (self-selection, desired response, other cognitive bias, etc.). Continue reading →

Recently, I received the following comment: “In today’s world it’s virtually impossible to keep up with facebook, twitter, news, tv, movies email, texts, etc.”

It was in response to a blog post about making a difference. How do you know? Given that most of what was suggested happens in the virtual world, the play on words is interesting. How is it impossible–because there is too much information? because you are too distracted by the virtual part of all the information and get lost? because virtuality it is not clearly understood? because of something else? I personally find I can get lost when I spend all day on line (virtual). It isn’t real, actually. I have no sense of what is happening and what isn’t happening. Even with the feeds from news lines, I find I have to double check my facts. Yet even as I say this, the virtual is expanding (go here). I have heard about Web 2.0; hadn’t heard about IoE (Internet of Everything)…the CEO of Cisco (John Chambers) stated that the IoE depends on the architecture, the systems integration. Is virtual the way of the world? It certainly isn’t the future any more; it is now. I have to ask, though, what about people…Given that much evaluation is now being done with the use of virtual tools, are we really understanding what difference is being made? Or are there just connections?

The individual continued with the comment by saying, “Keep up your small voice. Some are listening.” Those “listening” are certainly reflected in the number of comments I received on the posts about making a difference in the last two days (over 45). This may certainly be a way of engaging; I know it is outreaching. It is only my small voice; it is rewarding to know that some are listening/reading. Even if they only stay a short while.

My two cents. (my small voice).

molly.

Evaluation is an Everyday Activity

Program Evaluation Discussions

Adaptive action–evaluation by another name?

Summer Reading

Taking a stand

Doing to; doing with; doing as

Tomorrow is July 4th

I want you to tell me…

Unintended? Unanticipated? Un…?

Blogs are engaging?

Kirkpatrick Model

Virtual impossibility

Contact Info