Filed Under (program evaluation) by Molly on 17-11-2017

Good ideas. Maybe.

Did I get good ideas? Maybe.

I recently returned (Saturday,November 11, 2017, late) from the 2017 annual American Evaluation Association conference. This year the meeting was held in Washington, D. C.  (Thank you Lance Wyman, for this photo.) I realize that this is not the iconic view of D.C. that one imagines (like this: .) It was fall and it was mostly clear. I did get to the zoo as part of the conference.

As you know, I determine if a conference is good by seeing three long time friends, meeting three new people I want to see again, and getting three new ideas . This year was bitter sweet. Yes, I did see three long time friends (however, there were only 10). Used to be that I could not go across the lobby without seeing someone I knew well and wanted to see again. This year, many friends (both professional and personal) were not there–they had retired; they were frail and not traveling; they had died and I thought of my own mortality and realized that I had less time to take breaths, even those that take my breath away. I did not meet (although I did interact with young people) three new people I wanted to see again. I think I got only two good ideas–maybe three; hard to say.

Read the rest of this entry »

Filed Under (criteria, program evaluation) by Molly on 13-02-2017


Ah…love. Tomorrow is Valentine’s Day, the day traditionally set aside for lovers–you know the lovey dovey kind. And if you forgot…watch out.

It is the day when Saint Valentine    (officially Saint Valentine of Terni), a widely recognized third-century Roman saint, has his feast day. Since the  High Middle Ages it is associated with a tradition of courtly love. It is said that Valentine’s Day was established to counteract the pagan celebration of Lupercalia. There is much we do not know about St. Valentine.

Not courtly love.

I want to talk about a different kind of love (and I do not mean the various definitions of  that word). I want to talk about  your calling; your passion.

A good friend of mine said:  Know what your calling is, your vocation, and follow it faithfully.

She also said in that same missive: “When you are most disgruntled, take a moment of conscious breath or five moments of conscious play!”

This is the love I’m talking about. The love for your calling; your vocation (passion).

And what to do when you feel disgruntled (breathe/play).


Susan Kistler,  AEA Executive Director Emeritus, shares perhaps an important message about love:

“Success is made manifest in health and happiness, confidence that you are loved and the capacity to love with others.”

That is passion.

How does that relate to evaluation?

We are all evaluators and  live and work by criteria, whether they are implicit or explicit. Our passions are found in the criteria. We continue that passion for long in our lives–some of us because of family responsibilities; some of us because it is fun. When we get tired, we stop. We still have the passion and that passion comes out when we least expect it. Because once an evaluator (whether formally or not), always an evaluator.

So celebrate your passion tomorrow. And remember to breath…or play!

Sheila Robinson has an interesting post which she titled “Outputs are for programs. Outcomes are for people.”  Sounds like a logic model to me.

Evaluating something (a strategic plan, an administrative model, a range management program) can be problematic. Especially if all you do is count. So “Do you want to count?” OR “Do you want to determine what difference you made?” I think it all relates to outputs and outcomes.


Logic model


The model below explains the difference between outputs and outcomes.

.logicmodel (I tried to find a link on the University of Wisconsin website and UNFORTUNATELY it is no longer there…go figure. Thanks to Sheila, I found this link which talks about outputs and outcomes) I think this model makes clear the  difference between Outputs (activities and participation) and Outcomes-Impact (learning, behavior, and conditions). Read the rest of this entry »

Filed Under (criteria, program evaluation) by Molly on 19-08-2016

Probable? Maybe. Making a difference is always possible.

Oxford English Dictionary defines possible as capable of being (may/can exist, be done, or happen). It  defines probable as worthy of acceptance, believable.

Ray Bradbury Ray Bradbury: “I define science fiction as the art of the possible. Fantasy is the art of the impossible.”

Somebody asked me what was the difference between science fiction and fantasy. Certainly the simple approach is that science fiction deals with the possible (if you can think it, it can happen). Fantasy deals with monsters, fairies, goblins, and other mythical creatures, i.e., majic and majical creatures.

(Disclaimer: I personally believe in majic; much of fantasy deals with magic.) I love the Arthurian legend (it could be fantasy; it has endured for so long it is believable). It is full of majic. I especially like  the Marion Zimmer Bradley MarionZimmerBradley book, The Mists of Avalon Mists_of_Avalon-1st_ed. (I find the feminist perspective refreshing.)

Is fantasy always impossible as Bradbury suggests, or is it just improbable?  (Do the rules of physics apply?) This takes me back to Bradbury’s quote and evaluation after the minor digression. Bradbury also says that “Science fiction, again, is the history of ideas, and they’re always ideas that work themselves out and become real and happen in the world.” Not unlike evaluation. Evaluation works itself out and becomes real and happens. Usually.

Evaluation and the possible.

Often, I am invited to be the evaluator of record after the program has started. I sigh. Then I have a lot of work to do. I must teach folks that evaluation is not an “add on” activity. I  must also teach the folks how to identify the difference the program made. Then there is the issue of outputs (activities, participants) vs. outcomes (learning, behavior, conditions). Many principal investigators want to count differences pre-post.

Does the “how many” provide a picture of what difference the program made? If you start with no or few participants  and you end with many participants, have you made a difference? Yes, it is possible to count. Counts often meet reporting requirements. They are possible. So is documenting the change in knowledge, behavior, and conditions. It takes more work and more money. It is possible. Will you get to world peace? Probably not. Even if you can think it. World peace may be probable; it may not be possible (at least in my lifetime).

my two cents.




The WECT program arbitrarily divided the WECT program into four parts. Those “modules” are:

  • Program Planning and Logic Modeling;
  • Program Implementation, Monitoring, and Delivery;
  • Data Management and Analysis (divided into Qualitative data and Quantitative data); and
  • Program Evaluation Utilization

Read the rest of this entry »

Filed Under (program evaluation) by Molly on 17-09-2015

I’ve been stuck.

I haven’t blogged for three weeks. I haven’t blogged because I don’t have a topic. Oh, I’ve plenty to say (I am never for a loss of words… 🙂 ) I want something to relate to evaluation. Relate clearly. Without question. Evaluation.

So after 5 years, I’m going to start over. Evaluation is an everyday activity!

Evaluative thinking is something you do everyday; probably all day. (I don’t know about when you are a sleep, so I said probably.) I think evaluative thinking is one of those skills that everyone needs to learn systematically. I think everyone learns at least a part of evaluative thinking as they grow; the learning may not be systematic. I would put that skill in the same category as critical (not negative but thoughtful) thinking, team building, leadership, communication skills (both verbal and written), technological facility as well as some others which escape me right now. I would add systematic evaluative thinking.

Everyone has criteria on which decisions are based. Look at how you choose a package of cookies or a can of corn can of corn 2 at the grocery store. What criteria do you use for choosing? Yet that wasn’t taught to you; it was just something you developed. Evaluative thinking is more than just choosing what you want for dinner. AARP lists problem solving as part of the critical thinking skills. I think it is more than just problem solving; I do agree that it is a critical thinking skill (see graphic, from Grant Tilus, Rasmussen College).Core Critical Thinking Skills

So you think thoughtfully about most events/activities/things that you do throughout the day. And you learn over time what works and what doesn’t; what has value and what doesn’t. You learn to discern the conditions under which something works; you learn what changes the composition of the outcome. You begin to think evaluatively about most things. One day you realize that you are critically thinking about what you can, will, and need to do. Evaluative thinking has become systematic. You realize that it depends on many factors. You realize that evaluative thinking is a part of who you are. You are an evaluator, even if you are a psychologist or geologist or engineer or educator first.


my two cents.


Filed Under (program evaluation) by Molly on 17-07-2015

“fate is chance; destiny is choice”.destiny-calligraphy-poster-c123312071

Went looking for who said that originally so that I could give credit. Found this as the closest saying: “Destiny is no matter of chance. It is a matter of choice: It is not a thing to be waited for, it is a thing to be achieved.

William Jennings Bryan


Evaluation is like destiny. There are many choices to make. How do you choose? What do you choose?

Would you listen to the dictates of the Principal Investigator even if you know there are other, perhaps better, ways to evaluate the program?

What about collecting data? Are you collecting it because it would be “nice”? OR are you collecting it because you will use the data to answer a question?

What tools do you use to make your choices? What resources do you use?

I’m really curious. It is summer and although I have a list (long to be sure) of reading, I wonder what else is out there, specifically relating to making choices? (And yes, I could use my search engine; I’d rather hear from my readers!)

Let me know. PLEASE!

my two cents.


Filed Under (program evaluation) by Molly on 18-02-2015

social-justice.312132658_stdEarlier this week I attended a meeting of the College of Education (my academic home) Social and Environmental Justice (SJE) Work Group.  This is a loosely organized group of interested faculty and staff, led by an individual who is the ESOL Program Coordinator & Instructor. We had representatives from each of the four program areas (Adult and Higher Education [AHE], Teacher and Counseling Education [TCE], Science, Technology, Engineering, and Math [STEM], and Cultural and Linguistic Diversity [CLD]) in person (AHE, TCE,. CLD) or on paper [STEM]. The intent was to document for the work group, what each program area is doing in the area of social justice. Social Justice is a mandate for the College and OSU. The AHE and the TCE representatives provided us with information. We never did get to the STEM response. Then we got on to a discussion of what exactly is meant by social justice (since AHE has not defined the term specifically). My response was the evaluation response: it depends.

Most of the folks in the group focused on the interface of race and gender. OK. Others focused on the multiple and different voices. OK. Others focused on the advantages and disadvantages experienced. How is that not based in economics? Others focused on power and privileged. (As an accident of birth?) What is social justice exactly? Can you have social justice without environmental justice? How does that fit with the issue of diversity? How does any of this relate to evaluation?

The American Evaluation Association has had in place for a long time (since 1994) a set of five guiding principles (see Background section at the link for a bit of history). The fourth and fifth principles are, respectively, Respect for People and Responsibilities for General and Public Welfare. Respect for people says this:  Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. Responsibilities for the General and Public Welfare says this: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation. Although both talk about parts of social justice that we talked about earlier this week, is this a complete view? Certainly, security, dignity, and self worth and diversity of interests and values approach the discussion we had. Is there still something missing? I think so. Where is fairness addressed?

To me, fairness is the crux of the issue.  For example, it certainly isn’t fair that in the US, 2% of the population has the cumulative wealth of the remaining 98%. (Then we are into economics.) Although Gandhi said “be the change”social justice 4 is that enough? What if that change isn’t fair?  And the question must be addressed, fair to whom? What if that change is only one person? Is that fair?  I always talk about the long term outcome as world peace (not in my lifetime, though). If you work for justice (for me that is fairness) will peace result? I don’t know. justice 3



Tomorrow is the Lunar New Year. It is the year of the goat/sheep/ram. I wish you the best. Eat jiaozi and tangerines (for encouraging wealth), and noodles without breaking/biting them (you do want a long life, right?). Happy New Year.






Filed Under (criteria, program evaluation) by Molly on 25-11-2014



A uniquely American holiday (although it is celebrated in other countries as well-Canada, Liberia, The Netherlands, Norfolk Islands),

filled with too much food (pie any one?) apple pie,

too much football (what is your favorite rivalry?),football rivalry

and too much shopping (black Friday?).black-friday 3


For me it is an opportunity to to be grateful–and I am, more than words can express. I am especially grateful for my daughters, bright, articulate, and caring children (who are also adults). Read the rest of this entry »

From Social Networks: What Maslow Misses | Psychology Today – via @mslogophiliac

“Humans are social animals for good reason. Without collaboration, there is no survival. It was not possible to defeat a Woolley Mammoth, build a secure structure, or care for children while hunting without a team effort. It’s more true now than then. Our reliance on each other grows as societies became more complex, interconnected, and specialized. Connection is a prerequisite for survival, physically and emotionally.”

This statement, which I found on Harold Jarche ‘s blog, applies to evaluation as much as it applies to the example provided by Psychology Today.

Evaluation is a collaborative effort; a team effort, a social effort. Without the collaboration, evaluation lacks much. I’m not sure that survival is dependent on evaluation and collaborative effort; perhaps. The evaluator may know all about evaluation and yet not be able to solve the problem presented by the client because the evaluator doesn’t know about the topic needing to be evaluated. The evaluator may know about something similar to and different from what the client needs and yet, not know about the specific problem. Let me give you an example.

I’ve done a lot of evaluation in the natural resources area and as a result, I’ve learned much about various natural resource topics, including horticulture, plant science, crop science, marine science. I do not know much about potatoes. A while back, a colleague called me and asked if I could/would serve as the evaluator on a ZEBRA CHIP project. Before I said, Sure, I asked about ZEBRA CHIP. Apparently, it is a potato disease transmitted by bacteria carrying psyllid that is causing much economic devastation among growers. It shows up best when the potatoes are made into chips (hence the name). It looks like this: zebra chip. To me, it isn’t particularly stripped like the animal which offers its name, yet it doesn’t look like potato chips I’m used to seeing.  I”m told that there is an unpleasant flavor to the chips as well. I knew a lot about growing things, not about potatoes, even though I’ve worked with potato growers before, just not about this disease.

So, I said sure, only to discover that I have 11 lines in which to write a cogent evaluation section for the work that Extension will be doing (if the grant is funded). If the grant is funded, it will be a five year effort. A continuation actually, which brings me full circle–a collaboration of multiple universities, multiple disciplines, multiple investigators. So what could I say cogently in 11 lines? I suggested that perhaps looking at intention and confidence would be appropriate because we (remember, I said, “Sure”) would not be able to measure actual behavior change. And to overcome the psyllids and eradicate this problem (not unlike the spotted wing drosophila which is affecting the soft fruits of the NW, specifically blueberries), we would need to get as close to behavior change as possible once the teaching has occurred. How can social media be used here? Good question–something to explore. At what level of Maslow’s hierarchy is this collaboration?  Survival, sure. Somehow I don’t think Maslow was focused on economic survival.

my two cents.