I predict a bright future for complexity. Have you ever considered how complicated things can get, what with one thing always leading to another?

~~E. B. White, Quo VADIMUS? OR THE CASE FOR THE BICYCLE (Garden City. Publishing 1946)


Every thing is connected.

One thing does lead to another. And connections between them can be drawn. So let’s connect the dots.

In 2002, as AEA president, I chose for the theme of the meeting, Evaluation a Systematic Process that Reforms Systems.

Evaluation doesn’t stand in isolation; it is not something that is added on at the end as an afterthought.

Many program planners see evaluation that way, unfortunately. Only as an add on, at the end.

Contrary to may peoples’ belief, evaluators need to be included at the outset of the program. They also need to be included at each stage thereafter (Program Implementation, Program Monitoring, and Program Delivery; Data Management and Data Analysis; Program Evaluation Utilization).

Systems Concepts.

Shortly after Evaluation 2002, (in 2004) the Systems Evaluation Topical Interest Group was formed.

AEA published (2007) “Systems Concepts in Evaluation: A Expert Anthology” (scroll to the end). It was edited by Bob Williams and Iraj Imam (who died as this volume was going to press). To order, see this link.

This volume does an excellent job of demonstrating how evaluation and systems concepts are related.

It connects the dots.

In that volume, Gerald Midgley writes about “the intellectual development of systems field, how this has influenced practice and importantly the relevance for all this to evaluators and evaluation”. It is the pivotal chapter (according to the editors).

While it is possible to trace the idea to trace the ideas about holistic thinking back to the ancient Greeks, systems thinking is probably best attributed to the ideas of von Bertalanffy [Bertalanffy, L. von. (1950). Theory of open systems in physics and biology. Science, III: 23-29.]

I would argue that the complexity concept will go back to at least Alexander von Humboldt . (Way before von Bertalanffy.) He was an intrepid explorer and created modern environmentalism. Alexander von Humboldt lived between 1769–1859. Although environmentalism is a complex word, it really is a system. With connections. And complexity.

Suffice it to say, there are no easy answers to the problems faced by professionals today. Complex. Complicated. And one thing leading to another.

my .

Filed Under (program evaluation) by Molly on 14-09-2017 and tagged , , , ,

Evaluability assessment.

I get my ideas to blog from a lot of places. One place I get ideas are from other blogs.

The content of this blog was Evaluability Assessment.

The blog author says that evaluability assessments tend to cover the following topics:

  • Clarity of the intervention and its objectives: Is there a logical and clear theory of change that articulates how and under what conditions intervention activities influence particular processes of change?
  • Availability of data: which data are available that can be used in assessing the merit and worth of the intervention (e.g. generated by the intervention, external data sets, policy and academic literature)
  • Stakeholder interest and intended use: to what extent is there a clear interest (and capacity) among stakeholders to use the evaluation’s findings and recommendations in strategic decision-making, program improvement, learning about what works, etc.?

What, you ask, is evaluability assessment?

You certainly can go to the blog and read what it says there. OR…You can go to Scriven’s book and read the history on page 138 .

Suffice it to say that evaluability is the extent to which they (projects and programs) can be evaluated. Scriven goes on to say: “It should be thought of as the first commandment of accountability or as the last refinement of Popper’s (Sir Karl Raimond Popper) requirement of falsifiability.



I learned about evaluability assessment (EA) from Midge Smith (shown here with her husband Carl Wisler) in her book by the same name (published by Springer, search for it by title).  She says that EA is “…a diagnostic and prescriptive tool for improving programs and making evaluations more useful.” Like all tools used in evaluation, it is systematic and describes the structure of a program.

There is a newer volume  of that name. It is by Michael S. Trevisan and Tamara M. Walser (they do an AEA365 blog on that topic). It is not, unfortunately, on my shelf.  The blurb that accompanies the book (by the publisher, Sage) says: “Evaluability assessment (EA) can lead to development of sound program theory, increased stakeholder involvement and empowerment, better understanding of program culture and context, enhanced collaboration and communication, process and findings use, and organizational learning and evaluation capacity building.”

More detail than Midge offers, then her book is copyrighted in 1989.

EA is getting a lot of press lately (you may need to search for evaluability assessment when you go to AEA365).

I find it amazing how previously important things (EA) are now once again in vogue.

my .



Filed Under (program evaluation) by Molly on 06-09-2017 and tagged , , ,


With them, we are amazing!

Without them, we humons are limited.

I can only speak for myself–I do not want to be limited. I am a library champion. (A library champion is someone who fosters public awareness on the extensive range of resources and services available at public, school, academic and special libraries nationwide.)

If you were to see my professional library, perhaps you would understand. (It looks like the photo above, only the shelves are white; I have four bookcases with extensions. The shelves are full.)

If you were to see my personal library, you would understand. Most of it is still in boxes.

Because I believe that literacy is important, When they were young and beginning to read, I gave four of my bookshelves to my daughters (who are now grown, although their bookshelves are still full, mostly, of books). One each is in their room; two are now in the quest room. Hence, my library is mostly still in boxes.

I’ve stopped buying books for personal use (I still get professional ones). I use the library to get hard copy. I have 13 books at home plus two book club books. I have 11 books on hold.

I have an iPad on which I have at least four books and an equal number on hold. (I read a lot.)

But libraries do so much more than provide us with books (still their most important function). They move information in new directions! And they have magic fingers on the keyboard. I would be lost without libraries.

The information to which they have access is astounding.

Libraries and evaluation.

I want to discuss my professional library. I have one whole bookcase (of seven plus shelves) which is filled with books relating to evaluation. Some books are in many editions. And that doesn’t include statistics books or measurement books or the hard-copy journals that have come over the years.

“Why?” you ask, do I have multiple versions of the same (well, almost) book, different editions? Ah. Perhaps one edition will provide the answer (to the puzzle) and the others do not. Does that mean that the answer is not relevant? No. Does that mean that the information is passe? Maybe. Maybe not. The book may be the seminal reference and needs to be sited. It may give a history that isn’t found any place else. It is important to see how the volume changes with each edition. Having multiple volumes adds value. (And the root of evaluation is value.)

Do I need all this? Probably not, especially in the age of the internet and access to all that it provides. Yet, there is something about hard copy; you know a book  (whether a paperback or not), with its binding, its smell, its feel, that cannot be duplicated on-line. Something that cannot be diminished. Something that definitely adds value, merit and worth.


“To live in the world without becoming aware of the meaning of the world is like wandering in a great library without touching the books.” ~~Manly P. Hall

My feeling exactly.




Filed Under (program evaluation) by Molly on 01-09-2017

Get better.

Unless someone like you cares a whole awful lot, nothing is going to get better. It’s not.

~~Theodor Seuss Geisel


Dr. Seuss is a life-time favorite of mine. If he were still alive (unfortunately, he died in 1991), he would still be speaking out (however subtly) and making a difference (he was born in 1904). He opposed discrimination and isolationism and cartooned those issues between 1941-1943.


His comment above reminded me about why I’m an evaluator. I do care. A lot.

Yes, I like to solve puzzles.  I like to figure things out. Yes, I really like to make life better.

I use science. And scientific principles. And data. To get answers.

So I have to wonder, how can anyone go through this life without that support? (You know, the support of science and answers provided by data.)


Yes, I know…statistics lie (this saying “statistics lie” is often attributed to Mark Twain) and liars use statistics, depending on what the statistic is representing and who is using it. Or figures don’t lie; liars do figure may be more accurate (used by Carroll D. Wright ). (Oh and while I’m talking about statistics, Hans Rosling  died in February [2017]; a major loss.)

There is a lot of press about “big data” these days. And artificial intelligence. Is this the future?  Given that I hear what sound like competing stories about the future I can only wonder.

Even though I care. A lot. Perhaps as one source said, “…is it just a bit of social theatre we perform to make ourselves feel virtuous, useful, and in the right?” Maybe I’m just feeling virtuous and useful.

Maybe I only think I can change the world.

Gloria Anzaldua , an American scholar of  Chicana cultural theory who died in 2004, is quoted as saying, “I change myself; I change the world.”

So because I care (Seuss) and I am changing myself (Anzaldua), perhaps life will get better. Perhaps.

I will stick around and find out. And remember.

Filed Under (program evaluation, program planning) by Molly on 25-08-2017

 Key Evaluation Questions.

 My friend and colleague, Patricia Rogers shared Better Evaluation discussion of Key Evaluation Questions. The opening paragraphs were enough to get me reading. She says:

Key Evaluation Questions (KEQs) are the high-level questions that an evaluation is designed to answer. Having an agreed set of KEQs makes it easier to decide what data to collect, how to analyze it, and how to report it.

KEQs should be developed by considering the type of evaluation being done, its intended users, its intended uses (purposes), and the evaluative criteria being used.

Our BetterEvaluation task page ‘Specify Key Evaluation Questions’ runs through some of the main considerations to keep in mind when developing your KEQs, and gives some useful examples and resources to help you.

Choosing Key Evaluation Questions.

How do we know we’re making a difference?
Are the tactics we are using appropriate?
What are the meaningful signs of progress?
How does impact relate?

In the graphic above (for which I am grateful although I could not find a source) the closest these steps come to asking “key evaluation questions” is the step “define evaluation questions”.

I want to reiterate: They are NOT the “not specific questions that are asked in an interview or a questionnaire”.

So if you have done all the steps listed above, who defines the evaluation questions?

Or does it (as Rogers suggests) depend on the type of evaluation being done? (To be fair, she also suggests that the questions make it easier to decide what data to collect, how to analyze it, and how to report it.)

Now I’ve talked about the type of evaluation being done before.


Perhaps the type of evaluation is the key here.


There are seven types of evaluation (formative, process, outcome, economic, impact, goal-based, and summative). Each of those seven types do address a specific activity; each has a specific purpose. So it would make sense that the questions that are asked are different. Does each of those types of evaluation need to answer the above questions?


If we are looking to improve the program (formative, usually) do we really need to know if we are making a difference?
If the question of “how” (process) the program was delivered, what do we really need to know? We do need to know if the tactics (approaches) we are using are appropriate.
For any of the other purposes, we do need if a difference occurs. And the only way to know what difference we made is to ask the participants.

Post Script

Oh, since I last posted, I was able to secure a copy of the third edition of Levin’s book on economic evaluation.  Henry Levin wrote it with Patrick J. McEwan, Clive Belfield, A. Brooks Bowden, and Robert Shand.  It is worth a read (as much has changed since the second edition, specifically with “questions of efficiency and cost-effectiveness”.)


Alternative facts.

Never. Never. has evaluation been questioned with the label of “alternative facts.”

Over the years, I have been very aware that evaluation is a political activity.

I have talked about evaluation being political (here, and here, and here, and here ).

But is it? Is it just another way of making the answer be what we want it to be? A form of alternative fact?

I’ve been an evaluator for a long time. I want to make a difference to the people who experience my programs (or the programs for which I’m consulting as an external evaluator). The thought that I might be presenting “alternative facts” is troublesome.

Did I really determine that outcome? Or is the outcome bogus? Liars use statistics, you know. (This is a paraphrase of a quote that Mark Twain attributed to Benjamin Disraeli.)

Big news brings out the fakers. But are evaluation results “big news”? Or…do people not want to hear what is actually happening, what the outcome really is?

Reminds me of 1984 ( George Orwell): War is peace. Freedom is slavery. Ignorance is strength (the English Socialist Party–aka. INGSOC). Kevin Siers added, in his cartoon of Sean Spicer,  “2017 is 1984”.  Two contradictory ideas existing at the same time as correct.


Statistics is a tool that evaluators use on a regular basis. It allows evaluators to tease apart various aspects of a program. The “who” , the “what”, the “when”, maybe even the “why”. Statistics can certainly help determine if I made a difference  But how I see statistics may not be how you see them, interpret them, use them. Two people can look at a set of statistics and say they do not agree. Is that an example of alternative facts?


Everyone comes to any program with preconceived bias. You, the evaluator, want to see a difference. Preferably a statistically significant difference, not just a practical significance (although that would be nice as well).

Even if you are dealing with qualitative data, and not with quantitative data yielding statistics, you come to the program with bias. Objectivity is not an option. You wouldn’t be doing the program if you didn’t think that the program will make a difference. Yet, the individuals who have funded the program (or in some other way are the folks who get the final report) can (and do) not accept the report as it is written. That is not what they want to see/hear/read. Does that make the report alternative facts? Or is bias speaking without acknowledging that bias?

Perhaps Kierkegaard is right.

There are only two ways you can be fooled.


my .


Filed Under (program evaluation) by Molly on 14-06-2017

A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty.

~~Winston Churchill


“A pessimistpessimist is one who makes difficulties of his opportunities and an optimist optimist 1 is one who makes opportunities of his difficulties.”

~~Harry S. Truman

Two sides of the same coin? A different way to say the same thing?

Pessimist, optimist, realist?

So do you see the glass half empty–or half full?optimist-pessimist

OR are you a realist being able to refill the glass (with ice and a couple of shots of your favorite beverage)?

Evaluation is a field which includes all sorts of folks.

The optimist says that they can find a way.

The pessimist says that it isn’t likely.

The realist says it is possible and probable.

Working it out

One way to work it out is to use a logic model.

Another is to use a  theory of change.

Keep in mind that you might be wrong even if you apply either/both of the above.

(I remember a major professor of mine said that the theory may be wrong; it was.)

So sometimes, using the tools of evaluation may not get you where you want to be.

There are many approaches to learning something. For example: You can test a Hypothesis (Bill Nye the science guy says that a hypo thesis is an idea below). First, develop a plan; then you test it by gathering data. You analyze the data. And yes, there is some support for your hypothesis.

Or, you have an see an occurrence which repeats itself under various conditions.  You find that the emergent idea is dominant. So you control the situation and see if the idea emerges once again. It does!

I’m sure you can do trial and error; you can guess what the outcome will be; or you and follow what your parent did (worked for him/her, should work for you).

If you apply the scientific method  to the learning, you usually will test the hypothesis.

You will deal with humons at some point. The humon situation will be your guide.

But what if…

The people you are working with/for want it their way?

What if the end result is really a power and control issue and not one of transparent findings (good or bad).

How do you, the evaluator, address the implied (or actual) power and control.

Where does Standard III (Propriety) and the Guiding Principle D and E (Respect for People and Responsibilities for General and Public Welfare, respectively) enter into the discussion?

Out of your hands, you say? NO. Not really.

You do have a responsibility.

And an obligation.

You will have an opportunity

Does that make you a pessimist? Or an optimist? Or a realist? Only you will decide.

my  .




Filed Under (program evaluation) by Molly on 08-06-2017

Love. Revisited.

Love is the the most radically subversive activism of all, the only thing that ever changed anyone.

~~Ann Voskamp 


Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven’t found it yet, keep looking. Don’t settle. As with all matters of the heart, you’ll know when you find it.

~~Steve Jobs


Both of these quotes speak to me. So, let me tell you a story. And then how they relate to each other.

Two stories.

I fell in love with a 4 year old autistic boy,  a long time ago. I realized early on that stability and as much certainty as possible were important to him. I arranged to care for him (I was working in a hospital at the time) whenever I was working (stability). I don’t know what happened the two days I didn’t work; I just knew that he needed to “catch up” to where he had been before I didn’t work (as much certainty as possible). Slowly, oh, ever so slowly, he recovered from his illness and he “settled down”.  I like to think he trusted me; trusted me to be there; be there for him. And I was (even though I looked after other people). Then one day he wasn’t there anymore.

I was bereft. I mourned the loss of that delicate child. I could only hope that he “made it”. I do not know.

That was when I realized that I could make a difference in the lives of emotionally disturbed children. I was in love. I would do great work “saving” emotionally disturbed children. (NOT)

Fast forward many years. I was recently out of graduate school. It was spring. And spring had arrived in a burst of color and fragrance. Of course I took advantage of this opportunity and paused; I had an epiphany. I clearly saw three items:  do good work; be a good friend; grow spiritually/personally.  I realized  that the work I was doing filled a large part of my life (although I strived for a balance). I was satisfied. I was in love. Again. And even though it wasn’t with emotionally disturbed children, I was making a difference. 

I had found my passion. I didn’t settle. I found it what mattered to me.



Today (many more years since the epiphany and the love of my life) I find myself wanting to make a difference.  Still.

I certainly can do it with evaluation. And do. Does that program have value? Did it make a difference in the lives of the target audience? I look around at the world, the country, and wonder how can I make a difference; how can I be an activist (subversive or not); how will that change any thing? Yes, I believe love trumps hate. Yes, I believe in making a difference. Yes, I believe in doing good work (or as Steve Jobs said–great work).

Does that mean I need to work at this more OR does it mean I need to give myself permission to walk away from the struggle. To pause. To enjoy the roses now that they are in bloom?


my .



Filed Under (program evaluation) by Molly on 31-05-2017


Recently, I talked about how evaluations have changed.

They are still familiar yet they are different.

I have talked about formative and summative evaluation. (Thank you, Michael Scriven).

Those are two of the evaluation.

The other five are:

  1. Process Evaluation
  2. Outcome Evaluation
  3. Economic Evaluation
  4. Impact Evaluation
  5. Goals-based Evaluation

Yes, this discussion was from another blog.

So let’s discuss these other evaluations (which the author says you need to know to have an effective monitoring and evaluation system).

Choosing the evaluation for your program depends on where you are in the development of your program. If you are in the conceptualization phase there is one evaluation to use; the implementation phase uses others; and the end of the project will use yet a different evaluation or evaluations.

Going through them may help.

Conceptualization phase.

Formative evaluation typically is conducted during the development or improvements phase. By preventing waste and identify potential areas of concerns formative evaluation increases chances of success. It helps improve the program. Formative evaluation is often conducted more than once. It is usually contrasted with summative evaluation. For more information, see ; in fact, see it for all these evaluations, except as noted.

Implementation phase.

Process Evaluation usually refers to an evaluation of the treatment that focus entirely on variables between input and output data. It can also refer to the process component of the evaluation. Process evaluation occurs during the implementation phase.

Outcome Evaluation is often called “payoff evaluation.” Outcomes are often effects during the treatment. We would be wise to distinguish between immediate outcomes, middle (or end of treatment) outcomes, and long term outcomes. Outcome evaluation occurs during the implementation phase.

Economic Evaluation is also known as cost-benefit (or benefit-cost) analysis or cost-effectiveness analysis. For a detailed description of these types of evaluation see  OR . More and more, program designers are asked to do more with fewer resources and want to know what how efficient is the program.

Project closure (end) phase.

Impact Evaluation is an evaluation that focuses on outcomes. It occurs at the end of the project. Although it is desirable to do long term impact, there is often no funding available for that evaluation. The impact evaluation often back off the impact because there is no funding.

Summative Evaluation is conducted after the completion of the program, usually for the benefit of some external audience or funding agency. It should not be confused with outcome evaluation (an evaluation focused on outcomes rather than on process).

Goal-based Evaluation is any type of evaluation based on the goals and objectives of the program. It is done at the end of a program that is not on-going.  It often involves SMART objectives (Specific, Measurable, Attainable, Relavant, and Timely).

So what are you going to use to evaluate your program? You have a choice.

my .




Filed Under (program evaluation, program planning) by Molly on 16-05-2017



You know the old saying about when you assume.

I’ve talked about assumptions here and here. (AEA365 talks about them here.)

Each of those times I was talking about assumptions, though not necessarily from the perspective of today’s post.

I still find that making assumptions is a mistake as well as a cognitive bias. And it does… .

Today, though, I want to talk about assumptions that evaluators can make, and in today’s climate, that is dangerous.

So, let me start with an example. Read the rest of this entry »