Filed Under (criteria, program evaluation) by Molly on 14-09-2016


How do we make decisions when we think none of the choices are good?   decision

(Thank you for this thought, Plexus Institute.)

No, I’m not talking about the current political situation in the US. I’m talking about evaluation.

The lead for this email post was “Fixing the frame alters more than the view“. fixing the frame

Art Markman makes this comment (the “how do we make decisions…” comment) here. He says “If you dislike every choice you’ve got, you’ll look for one to reject rather than one to prefer—subtle difference, big consequences.” He based this opinion on research, saying that the rejection mind-set allows us to focus on negative information about options and fixate on the one with the smallest downside. Read the rest of this entry »

Filed Under (criteria, program evaluation) by Molly on 19-08-2016

Probable? Maybe. Making a difference is always possible.

Oxford English Dictionary defines possible as capable of being (may/can exist, be done, or happen). It  defines probable as worthy of acceptance, believable.

Ray Bradbury Ray Bradbury: “I define science fiction as the art of the possible. Fantasy is the art of the impossible.”

Somebody asked me what was the difference between science fiction and fantasy. Certainly the simple approach is that science fiction deals with the possible (if you can think it, it can happen). Fantasy deals with monsters, fairies, goblins, and other mythical creatures, i.e., majic and majical creatures.

(Disclaimer: I personally believe in majic; much of fantasy deals with magic.) I love the Arthurian legend (it could be fantasy; it has endured for so long it is believable). It is full of majic. I especially like  the Marion Zimmer Bradley MarionZimmerBradley book, The Mists of Avalon Mists_of_Avalon-1st_ed. (I find the feminist perspective refreshing.)

Is fantasy always impossible as Bradbury suggests, or is it just improbable?  (Do the rules of physics apply?) This takes me back to Bradbury’s quote and evaluation after the minor digression. Bradbury also says that “Science fiction, again, is the history of ideas, and they’re always ideas that work themselves out and become real and happen in the world.” Not unlike evaluation. Evaluation works itself out and becomes real and happens. Usually.

Evaluation and the possible.

Often, I am invited to be the evaluator of record after the program has started. I sigh. Then I have a lot of work to do. I must teach folks that evaluation is not an “add on” activity. I  must also teach the folks how to identify the difference the program made. Then there is the issue of outputs (activities, participants) vs. outcomes (learning, behavior, conditions). Many principal investigators want to count differences pre-post.

Does the “how many” provide a picture of what difference the program made? If you start with no or few participants  and you end with many participants, have you made a difference? Yes, it is possible to count. Counts often meet reporting requirements. They are possible. So is documenting the change in knowledge, behavior, and conditions. It takes more work and more money. It is possible. Will you get to world peace? Probably not. Even if you can think it. World peace may be probable; it may not be possible (at least in my lifetime).

my two cents.



Filed Under (criteria, program evaluation) by Molly on 05-08-2016

To handle yourself, use your head; to handle others, use your heart. ~~Eleanor Roosevelt eleanor roosevelt

This quote is often attributed to Eleanor Roosevelt (1884-1962); there are some sites that attribute this quote to Donald Anderson Laird (1897-1969), a psychologist and author (no photo found). Probably, more accurate. I’m not sure that the origin of the saying is really important. It may be enough to keep in mind the saying itself. (I know–how does this relate to evaluation? Trust me, it does.)

Before I was an evaluator, I was a child therapist (I also treated young women). I learned many skills as a therapist that have served me well as an evaluator. Skills like listening, standing up for your self, looking at alternatives. Which leads me to this saying. I had to “handle” others all the time at the same time I had to “handle” my self. I could not “blow up” when reprimanded. I could not become discouraged when someone (the client, the funder) criticized me. I had to learn to laugh when the joke was on me. I had to keep my spirits up when things went wrong. I had to keep cool in emergencies. I had to learn to tune out gossip and negative comments from others. This was a hard time for me. I tend to be passionate when I have an opinion; I have/had opinions (often).

As an evaluator, I am still passionate. Once my evaluation “on” button is pushed, it is hard to turn it off. Yet I still have to handle people. This morning, for example, I met with a fellow faculty member. I had to listen. I had to look for (and at) alternatives. I “handled” with my head; remember, I am passionate about evaluation. I provided her with alternatives and followed through with those alternatives. I handled with my heart.

When others are involved (and in evaluation there are always others), they must be handled with care, with the heart. It goes back to the standards (propriety) The_Program_Evaluation_Standards_3ed and the guiding principles Guiding principles  (integrity/honesty, direct respect for people, and responsibilities for general and public welfare).  In the current times, it is especially important to have direct respect for people. All people. (Regardless of race, ethnicity, religion, gender identity, sex, national origin, veteran status, and disability.) To be honest and have integrity. One way to make sure you have integrity is to handle with your heart.


Filed Under (criteria, program evaluation) by Molly on 13-07-2016

I want to talk about learning. Real learning. This week I am borrowing a blog from another writer intact. I have never done this. True, I have taken parts of blogs and quoted them. This blog post from the blog called “adapting to perpetual beta” by Harold Jarche is applied here in its entirety because I think the topic is important. I have added the visuals except for the Rodin, which was in the original post.

Yes, it relates to evaluation. We learn (those who value evaluation) throughout our careers. The various forms of learning are engaged (see: Edgar Dale who designed the learning cone though not with percentages that are usually attributed to the styles).Cone of learning(This particular version was developed by Bruce Hyland based on Dale’s work.) When you read the post below, think about how you learn. Engages? Reflective?

real learning is not abstract

Posted 2016-06-20

Are we entering an era that heralds ‘The End of Reflection’, as this NY Times article suggests?

Read the rest of this entry »

Filed Under (criteria, program evaluation) by Molly on 01-07-2016

The person without a purpose is like a ship without a rudder. (Thomas Carlyle)

There is much written about finding your purpose Purpose if life.  Songs are written about purpose; self-help books are written about purpose; businesses are devoted to the concept; jewelry, leadership, among other things, all focus on purpose.

So how do you find purpose? How do you know what your are “supposed” to do in this life? How does that relate to evaluation? Finding your purpose can be really confusing. Purpose 2 Let me share a story with you.

I lived in Birmingham, AL in the 80s and 90s. Birmingham is the only place I have lived (and I’ve lived many places) where if you woke up on the first day of spring, EVERYTHING would be in bloom. Everything! In Oregon, spring creeps up on you (a wonderful experience, to be sure). In Minnesota, it feels like it is spring one day and summer the next (or if you are not lucky, winter, again). In Tucson, spring happens in February and if you blink you miss it (well, almost). So I was marveling one day around the first day of spring how wonderful life was and I had an epiphany. I conceptualized what were the three things I wanted to do in this life. I wanted to do good work. I wanted to be a good friend. I wanted to grow spiritually. (I knew that being a boss was not for me, even though it came with perks.)

I had just finished a doctoral program in program evaluation. I realized that I would be “in the trenches” a long time and would spend most of my career doing evaluation work (as opposed to teaching evaluation, researching evaluation, writing about evaluation). I saw that as my purpose. To do good work–good evaluation work.

So what does it mean to do “good evaluation work”? Read the rest of this entry »

Filed Under (criteria, program evaluation) by Molly on 14-01-2016

Recently, I read that 45% of individuals make New Year’s Resolutions and only 8% actually achieve success. Hmmm…not a friendly probability. Perhaps intentions about behavior are indeed more realistic. (Haven’t seen the statistics on that potential change. Mazanian (et al, 1998) pemazman does say stated intention to change is the most significant behavioral indicator.) My intention for 2016 is to provide content related to or about evaluation that provides you with something you didn’t have before you read the post (Point one). Examples follow: Read the rest of this entry »

Filed Under (criteria) by Molly on 18-12-2015

Celebrate! Celebrate! Dance to your music!

The holidays are here:

happy holidays1

This is a wonderful formula for Happiness!


Happy Holidays!

my two cents.


Filed Under (criteria, program evaluation) by Molly on 08-10-2015

First, let me say that getting to world peace will not happen in my lifetime (sigh…) and world peace is the ultimate impact. Everything else is an outcome. It may be a long term outcome, that is a condition change (either social, economic, environmental, or civic), or not. Just because the powers that be use a term doesn’t mean the term is being used correctly!

Then let me say that evaluation is the way to know you got  to that impact…ultimately, world peace. Ultimately. In the mean time, you will need to find approximate (proxy) measures.

Last week, I attended the Engagement Scholarship Consortiumengagement scholarship consortium conference in State College, PA, home of Penn State.penn-state-logo I had the good fortune to see long time friends, meet new people and get a few new ideas. One of the long time friends I was able to visit with was Nancy Franz, Professor Emeritus, Iowa State University. She did a session called “Four steps to measuring and articulating engagement impact”.

Basically she reduced into four steps (hence, the title) program evaluation. And since engagement scholarship is a “program” it needs to be evaluated to make sure it is making a difference. Folks are slowly coming to that idea if the attendance at her session is any indication (full). She used different words than I would have used; I found myself adding parenthetical comments to her words.

I want to share in words what she shared graphically:

  1. In order to be able to conduct these four steps, you need evaluation training, evaluation support, and successful models;
  2. STEP 1: You need to map the intended program (my parenthetical was the “logic model” for which she provided the UWEX web site);
  3. STEP 2: You need to determine what “impact” will be measured (input vs. outcome);
  4. STEP 3: You need to collect and analyze data (qualitative and quantitative);
  5. STEP 4: You need to tell the story (when, what, so what, now what; the public value);
  6. If you do these four steps she believes that you will enhance paid and volunteer staff performance; increase program quality; and improve impact reporting (be persuasive).

She had a few good suggestions; specifically:

  1. Since most people don’t like to analyze data (because they do not know how?), she holds a data party to look at what was found; and
  2. Case studies have value; use them.
  3. I added, “If you aren’t going to use the data, do not collect it. It only obfuscates the impact.”

Think about what you do when you evaluate a program. Do you do these four steps? Do you know what impact you are trying to achieve? And if you can’t get to world peace, that’s OK. Each step will bring you closer.

my two cents.


Filed Under (criteria, Methodology, program evaluation) by Molly on 18-08-2015

I just got back from a road trip across Southern Alabama with my younger daughter.southern alabama We started from Birmingham and drove a very circuitous route ending in Mobile and the surrounding areas, then returned to Birmingham for her to start her second year at Birmingham-Southern College.

As we traveled, I read a book by Bill McKibben (one of many) called Oil and Honey: The Education of an Unlikely Activist. It is a memoir, a personal recounting of the early years of this decade, which corresponded with the years my older daughter was in college (2011-2014). I met Bill McKibben, who, in 2008, is credited with starting the non-profit, 350.0rg, and is currently listed as “senior adviser and co-founder”. He is a passionate, soft-spoken man, who beleives that the world is on a short fuse. He really seems to believe that there is a better way to have a future. He, like Gandhi, is taking a stand.  Oil and Honey puts into action Gandhi’s saying about being the change you want to seegandhi and change. As the subtitle indicates, McKibben is an unlikely activist. He is a self-described non-leader who led and advises the global effort to increase awareness of climate change/chaos. When your belief is on the line, you do what has to be done.

Evaluators are the same way. When your belief is on the line, you do what has to be done. And, hopefully, in the process you are the change that you want to see in the world. But know it cannot happen one pipeline at a time. The fossil fuel industry has too much money. So what do you do? You start a campaign. That is what has done:  “There are currently fossil fuel divestment campaigns at 308 colleges and universities, 105 cities and states, and 6 religious institutions.”(Wikipedia, 350.0rg) (Scroll down to the heading “Fossil Fuel Divestment” to see the complete discussion.) Those are clear numbers, hard data for consumption. (Unfortunately, the  divestment campaign at OSU failed.)

So I see the question as one of impact, though not specifically world peace (my ultimate impact). If there is no planet on which to work for world peace, there in no need for world peace. Evaluators can help. They can look at data critically. They can read the numbers. They can gather the words. This may be the best place for the use of pictures (they are, after all, worth 1000 words).  Perhaps by combining efforts, the outcome will be an impact that benefits all humanity and builds a tomorrow for the babies born today.

my two cents.



Filed Under (criteria) by Molly on 04-08-2015

I keep getting comments about my posts “Does this blog make a difference?”

I want to say thank you for all who read it.



I want to say thank you for all who follow this blog.


Mostly, I am continually amazed that people find what I have to say interesting to come back.

So: Thank you. For reading. For following. For coming back.

I think that is making a difference.

my two cents.


P. S. See you in two weeks!