Nov
21
Filed Under (program evaluation) by Molly on 21-11-2017

Gratitude.

The day after tomorrow is a national holiday. One of gratitude.

It is the lead up to the end of the year holidays and the long dark.

Yep. Thanksgiving.

(I couldn’t decide which was more representative…certainly the Norman Rockwell painting isn’t; so I didn’t include it.)

Real meaning?

I recently read an article from my Alma mater on the “Real Meaning of Thanksgiving”. What I didn’t know is that “since 1970, Native Americans… commemorate a National Day of Mourning on the US Thanksgiving holiday.”  It is a “reminder of the genocide of millions of indigenous people, the theft of their lands, and the relentless assault on their culture.” This day of remembrance and spiritual connection protests the racism and oppression which the Native Americans continue to experience.  A similar gathering will take place in San Francisco, California, on Alcatraz Island.

This is because of the concept “described by scholars as settler colonialism”.

Ronald Trosper, professor of American Indian studies at the University of Arizona, presents a short quiz about Thanksgiving Day (three questions only). Although he cites the web site  GlobalSocietyTheory.com, that link doesn’t work. He says “Settler colonialism persists in the ongoing elimination of indigenous populations, and the assertion of state sovereignty and juridical control over their lands.” Although Thanksgiving is an US holiday, “…Canada, Australia, New Zealand and South Africa also are examples of countries formed by settler colonialism.”

Criteria.

Everyone has a criteria for determining what Thanksgiving means to them. What is the value, merit, worth of that program.

Is is just a two day break from work? Is it just another holiday? Or is is about the food? For me, I look at the food and I am thankful. I have not gone hungry. Sometimes it is any food; sometimes it is only green food.

This year, I will be celebrating the holiday with my brother, his wife, his son, and his son’s girlfriend. We will feast of foods for which we are thankful. That does not include turkey.

I made a new pie for gathering. Dulce De Leche any one?

 

my.

 

Sep
06
Filed Under (program evaluation) by Molly on 06-09-2017

LIBRARIES 

With them, we are amazing!

Without them, we humons are limited.

I can only speak for myself–I do not want to be limited. I am a library champion. (A library champion is someone who fosters public awareness on the extensive range of resources and services available at public, school, academic and special libraries nationwide.)

If you were to see my professional library, perhaps you would understand. (It looks like the photo above, only the shelves are white; I have four bookcases with extensions. The shelves are full.)

If you were to see my personal library, you would understand. Most of it is still in boxes.

Because I believe that literacy is important, When they were young and beginning to read, I gave four of my bookshelves to my daughters (who are now grown, although their bookshelves are still full, mostly, of books). One each is in their room; two are now in the quest room. Hence, my library is mostly still in boxes.

I’ve stopped buying books for personal use (I still get professional ones). I use the library to get hard copy. I have 13 books at home plus two book club books. I have 11 books on hold.

I have an iPad on which I have at least four books and an equal number on hold. (I read a lot.)

But libraries do so much more than provide us with books (still their most important function). They move information in new directions! And they have magic fingers on the keyboard. I would be lost without libraries.

The information to which they have access is astounding.

Libraries and evaluation.

I want to discuss my professional library. I have one whole bookcase (of seven plus shelves) which is filled with books relating to evaluation. Some books are in many editions. And that doesn’t include statistics books or measurement books or the hard-copy journals that have come over the years.

“Why?” you ask, do I have multiple versions of the same (well, almost) book, different editions? Ah. Perhaps one edition will provide the answer (to the puzzle) and the others do not. Does that mean that the answer is not relevant? No. Does that mean that the information is passe? Maybe. Maybe not. The book may be the seminal reference and needs to be sited. It may give a history that isn’t found any place else. It is important to see how the volume changes with each edition. Having multiple volumes adds value. (And the root of evaluation is value.)

Do I need all this? Probably not, especially in the age of the internet and access to all that it provides. Yet, there is something about hard copy; you know a book  (whether a paperback or not), with its binding, its smell, its feel, that cannot be duplicated on-line. Something that cannot be diminished. Something that definitely adds value, merit and worth.

 

“To live in the world without becoming aware of the meaning of the world is like wandering in a great library without touching the books.” ~~Manly P. Hall

My feeling exactly.

 

 

 

Jun
23
Filed Under (Data Analysis, program evaluation) by Molly on 23-06-2017

Alternative facts.

Never. Never. has evaluation been questioned with the label of “alternative facts.”

Over the years, I have been very aware that evaluation is a political activity.

I have talked about evaluation being political (here, and here, and here, and here ).

But is it? Is it just another way of making the answer be what we want it to be? A form of alternative fact?

I’ve been an evaluator for a long time. I want to make a difference to the people who experience my programs (or the programs for which I’m consulting as an external evaluator). The thought that I might be presenting “alternative facts” is troublesome.

Did I really determine that outcome? Or is the outcome bogus? Liars use statistics, you know. (This is a paraphrase of a quote that Mark Twain attributed to Benjamin Disraeli.)

Big news brings out the fakers. But are evaluation results “big news”? Or…do people not want to hear what is actually happening, what the outcome really is?

Reminds me of 1984 ( George Orwell): War is peace. Freedom is slavery. Ignorance is strength (the English Socialist Party–aka. INGSOC). Kevin Siers added, in his cartoon of Sean Spicer,  “2017 is 1984”.  Two contradictory ideas existing at the same time as correct.

Statistics.

Statistics is a tool that evaluators use on a regular basis. It allows evaluators to tease apart various aspects of a program. The “who” , the “what”, the “when”, maybe even the “why”. Statistics can certainly help determine if I made a difference  But how I see statistics may not be how you see them, interpret them, use them. Two people can look at a set of statistics and say they do not agree. Is that an example of alternative facts?

Bias.

Everyone comes to any program with preconceived bias. You, the evaluator, want to see a difference. Preferably a statistically significant difference, not just a practical significance (although that would be nice as well).

Even if you are dealing with qualitative data, and not with quantitative data yielding statistics, you come to the program with bias. Objectivity is not an option. You wouldn’t be doing the program if you didn’t think that the program will make a difference. Yet, the individuals who have funded the program (or in some other way are the folks who get the final report) can (and do) not accept the report as it is written. That is not what they want to see/hear/read. Does that make the report alternative facts? Or is bias speaking without acknowledging that bias?

Perhaps Kierkegaard is right.

There are only two ways you can be fooled.

 

my .

molly.

Feb
13
Filed Under (criteria, program evaluation) by Molly on 13-02-2017

Love.

Ah…love. Tomorrow is Valentine’s Day, the day traditionally set aside for lovers–you know the lovey dovey kind. And if you forgot…watch out.

It is the day when Saint Valentine    (officially Saint Valentine of Terni), a widely recognized third-century Roman saint, has his feast day. Since the  High Middle Ages it is associated with a tradition of courtly love. It is said that Valentine’s Day was established to counteract the pagan celebration of Lupercalia. There is much we do not know about St. Valentine.

Not courtly love.

I want to talk about a different kind of love (and I do not mean the various definitions of  that word). I want to talk about  your calling; your passion.

A good friend of mine said:  Know what your calling is, your vocation, and follow it faithfully.

She also said in that same missive: “When you are most disgruntled, take a moment of conscious breath or five moments of conscious play!”

This is the love I’m talking about. The love for your calling; your vocation (passion).

And what to do when you feel disgruntled (breathe/play).

Passion.

Susan Kistler,  AEA Executive Director Emeritus, shares perhaps an important message about love:

“Success is made manifest in health and happiness, confidence that you are loved and the capacity to love with others.”

That is passion.

How does that relate to evaluation?

We are all evaluators and  live and work by criteria, whether they are implicit or explicit. Our passions are found in the criteria. We continue that passion for long in our lives–some of us because of family responsibilities; some of us because it is fun. When we get tired, we stop. We still have the passion and that passion comes out when we least expect it. Because once an evaluator (whether formally or not), always an evaluator.

So celebrate your passion tomorrow. And remember to breath…or play!

Jan
16
Filed Under (program evaluation) by Molly on 16-01-2017

Resolutions. Renewal.

Renewal is appropriate for the new year. So are resolutions.new-years-resolutions

It has been over a month since I blogged here. And the longer I wait for inspiration, the harder it is to write.

But I’m waiting for inspiration. Really difficult, to be sure.

We all know that resolutions have a great tendency to fail.

So how can one find renewal in these difficult times?

Perhaps it is time to re-evaluate your priorities.

Priorities can change. Depending on circumstances.

Is this a time for you to be more articulate?

Or a time to be more proactive?

A time to be more (fill in the blank)?

Writer’s block

Sheila Robinson the sometime Saturday contributor for AEA 365 Read the rest of this entry »

Oct
20
Filed Under (program evaluation, program planning) by Molly on 20-10-2016

Evaluation is political. I am reminded of that fact when I least expect it.

In yesterday’s AEA 365 post, I am reminded that social justice and political activity may be (probably are) linked; are probably sharing many common traits.

In that post the author lists some of the principles she used recently:

  1. Evaluation is a political activity.
  2. Knowledge is culturally, socially, and temporally contingent.
  3. Knowledge should be a resource of and for the people who create, hold, and share it.
  4. There are multiple ways of knowing (and some ways are privileged over others).

Evaluation is a trans-discipline, drawing from many many other ways of thinking. We know that politics (or anything political) is socially constructed. We know that ‘doing to’ is inadequate because ‘doing with’ and ‘doing as’ are ways of sharing knowledge. (I would strive for ‘doing as’.) We also know that there are multiple ways of knowing.

(See Belenky belenky, Clinchy [with Belenky] belenkyclinchy_trimmed, Goldberger nancy_goldberger, and Tarulejill-mattuck-tarule, Basic Books, 1986 as one.)

OR

(See: Gilligan carol-gilligan, Harvard University Press, 1982; among others.)

How does evaluation, social justice, and politics relate?

What if you do not bring representation of the participant groups to the table?

If they are not asked to be at the table or for their opinion?

What if you do not ask the questions that need to be asked of that group?

To whom ARE your are your questions being addressed?

Is that equitable?

Being equitable is one aspect of social justice. There are others.

Evaluation needs to be equitable.

 

I will be in Atlanta next week at the American Evaluation Association conference. atlanta-georgia-metropolitan

Maybe I’ll see you there!

my two cents.

molly.

 

 

 

 

Oct
08
Filed Under (program evaluation, program planning) by Molly on 08-10-2016

Process is the “how”.

Recently,  reminded of the fact that process is the “how”, I had the opportunity to help develop a Vision Vision Road Sign with dramatic blue sky and clouds.

and a Mission mission statement.

The person who was facilitating the session provided the group with clear guidelines.

The Vision statement, defined as “the desired future condition”, will happen in 2-5 years (i.e., What will change?). We defined the change occurring (i.e., in the environment, the economy, the people). The group also identified what future conditions would be possible. We would write the vision statement so that it would happen within 2-5 years, be practical, be measurable, and be realistic. OK…

And be short…because that is what vision statements are.

The Mission statement (once the Vision statement was written and accepted) defined “HOW” we would get to the vision statement. This reminded me of process–something that is important in evaluation. So I went to my thesaurus to find out what that source said about process. Scriven  Scriven to the rescue, again.

 

 

Process Evaluation

Scriven, in his Evaluation Thesaurus Scriven book cover defines process as the activity that occurs “…between the input and the output, between the start and finish”. Sounds like “how” to me. Process relates to process evaluation. I suggest you read the section on process evaluation on page 277 in the above mentioned source.

Process evaluation rarely functions as the sole evaluation tool because of weak connections between “output quantity and quality”. Process evaluations will probably not generalize to other situations.

However, PROCESS evaluation “…must be looked at as part of any comprehensive evaluation, not as a substitute for inspection of outcomes…” The factors include “the legality of the process, the morality, the enjoyability, the truth of any claims involved, the implementation…, and whatever clues…” that can be provided.

Describing “how ” something is to be done is not easy. It is not output nor outcome.  Process is the HOW something will be accomplished if you have specific inputs . It happens between the inputs and the outputs.

To me, the group needs to read about process evaluation in crafting the mission statement in order to get to the HOW.

my two  two cents      .

molly.

 


 

Sep
14
Filed Under (criteria, program evaluation) by Molly on 14-09-2016

Decisions

How do we make decisions when we think none of the choices are good?   decision

(Thank you for this thought, Plexus Institute.)

No, I’m not talking about the current political situation in the US. I’m talking about evaluation.

The lead for this email post was “Fixing the frame alters more than the view“. fixing the frame

Art Markman makes this comment (the “how do we make decisions…” comment) here. He says “If you dislike every choice you’ve got, you’ll look for one to reject rather than one to prefer—subtle difference, big consequences.” He based this opinion on research, saying that the rejection mind-set allows us to focus on negative information about options and fixate on the one with the smallest downside. Read the rest of this entry »

Aug
31
Filed Under (program evaluation) by Molly on 31-08-2016

AEA365 is honoring living evaluators for Labor Day (Monday, September 5, 2016).

Some of the living evaluators I know (Jim Altschuld, Tom Chapel, Michael Patton, Karen Kirkhart, Mel Mark, Lois-Ellin Datta, Bob Stake); Some of them I don’t know (Norma Martinez-Rubin, Nora F. Murphy, Ruth P. Saunders, Art Hernandez, Debra Joy Perez). One I’m not sure of at all (Mariana Enriquez).  Over the next two weeks, AEA365 is hosting a recognition of living evaluator luminaries.

The wonderful thing is that this give me an opportunity to check out those I don’t know; to read about how others see them, what makes them special. I know that the relationships that develop over the years are dear, very dear.

I also know that the contributions that  these folks have made to evaluation cannot be captured in 450 words (although we try). They are living giants, legends if you will.

These living evaluators have helped move the field to where it is today. Documenting their contributions to evaluation enriches the field. We remember them fondly.

If you don’t know them, look for them at AEA ’16 in Atlanta atlanta-georgia-skyline. Check out their professional development sessions or their other contributions (paper, poster, round-table, books, etc). Many of them have been significant contributors to AEA; some have only been with AEA since the early part of this century. All have made a meaningful contribution to AEA.

Many evaluators could be mentioned and are not. Sheila B. Robinson suggests that “…we recognize that many, many evaluators could and should be honored as well as the 13 we feature this time, and we hope to offer another invitation next year for those who would like to contribute a post, so look for that around this time next year, and sign up!

Evaluators honored

altschuld       Thomas J. Chapel

James W. Altschuld            Thomas J. Chapel

Norma Martinez-Rubin              Patton

Norma Martinez-Rubin            Michael Quinn Patton

 

       Ruth P. Saunders

Nora F. Murphy                                     Ruth P. Saunders

 

ArthurHernandez                  Kirkhart

Art Hernandez                          Karen Kirkhart

Melvin Mark            Loisellen datta

Mel Mark                                       Lois-Ellin Datta

debra-perez-thumbnail-340x340       bob stake 2

Debra Joy Perez                           Bob Stake

ghost_person_60x60_v1

Mariana Enriquez (Photo not known/found)

my two cents.

molly.

Sheila Robinson has an interesting post which she titled “Outputs are for programs. Outcomes are for people.”  Sounds like a logic model to me.

Evaluating something (a strategic plan, an administrative model, a range management program) can be problematic. Especially if all you do is count. So “Do you want to count?” OR “Do you want to determine what difference you made?” I think it all relates to outputs and outcomes.

 

Logic model

 

The model below explains the difference between outputs and outcomes.

.logicmodel (I tried to find a link on the University of Wisconsin website and UNFORTUNATELY it is no longer there…go figure. Thanks to Sheila, I found this link which talks about outputs and outcomes) I think this model makes clear the  difference between Outputs (activities and participation) and Outcomes-Impact (learning, behavior, and conditions). Read the rest of this entry »