Category Archives: program evaluation

Farewell

Posted on by

Farewell.

Someone asked why I stopped posting (I did wonder if anyone would notice). Here is the reason: I will be retiring from Oregon State University the end of August, 2018 and getting ready for that event will take the next four months.

This is my farewell post.

Plans.

I do not have any plans for this commencement.

That is scary for me as I have always had plans.

My evaluation skills do not help me with formulating a plan.

I hope that you, my readers, will remember that evaluation is an everyday activity.

That you assign value, worth, merit to every activity you do. (Remember that the root of evaluation is value.)

Think evaluatively in every situation.

This is a fond farewell, which is bittersweet.

The bittersweet is because I have enjoyed putting evaluation content out there to the nether.

Yet, providing content weekly was hard, challenging, thought provoking.

Tomorrow?

I will not be going to Cleveland (for the American Evaluation Association conference) or to Minneapolis  (for the Engagement Scholarship Consortium conference). I am sorting through my papers, boxes, and drawers full of papers of which I am throwing most out. I don’t plan to take much with me.

Goals.

I have done good work. I have been a good friend. I understand that this “new job” will/could take about 12 months to get settled.

I will be on another adventure; I don’t know what that will be. I will do what pleases me: sleeping until the light wakes me, reading (so many books, so little time; I’m a slow reader), working in my yard. Maybe I will travel. I might learn a new language (something more than words). Perhaps I will take up an instrument and make lovely music. I have lots of options, some I’ve continued since the early 1980s, some that are providing me with things to think about. I remind myself that this is an adventure.

So, my readers, I wish you well.

Adieu.

Beauty.

Posted on by

Beauty.

Let the beauty of what you love be what you do.

~~Jalāl ad-Dīn Muhammad Balkhī (Rumi)

(If you don’t know about Rumi, you can read all about him here and here.)

Passion.

I’ve talked  about being an evaluator before.  (You can also read about that love here and here.) The word that best describes what I do is passion. I do what I love and there is beauty in that.

There was a time that I could not read enough about evaluation. (I spoke about that here.)

Now, I reflect on the past 30+ years and on that passion, that love, that beauty.

I did/do what I love and love what I did/do.

So I went looking for images that captured that passion. That love.

I found a lot:

Don’t know if I changed the world for the better (that was my goal).

   My passion became my purpose.

   I find passion more than an emotion (certainly emotionality adds to the passion).

   Emotionally disturbed children and evaluation. They each captured my heart. Evaluation caught my heart the longest.

                                       Reading excessively did that.

            I didn’t settle. That is the lesson I taught.

Wisdom.

Mandela was a wise person. As was Gandhi. So were a lot of people. I hope I mirror some of that wisdom.

Evaluation is important. I suppose evaluation could be considered “make work” profession EXCEPT that making a difference is not “make work”; it is important. And if there are scholars, that is good. If there are practitioners, that is good also. There are folks who make a difference in the world. And that it good. Very good. Evaluators make a difference. Wisdom is found there. That is the beauty of the work; that is the love of the work.

Book(s)

Posted on by

Book(s).

A book is a dream that you hold in your hand.

~~Neil Gaiman

Where do you want to go?

What do you want to do?

When is your passion?

Who is your best friend?

Why do you care?

All of these questions have one answer: books.

(or if you choose not to own them–libraries.)

Library.

You can have a personal library made of books. Or a professional library made of books.

Or, if you choose not to own the book, you can use the community library made of books (and DVDs and CDs and other things that I have yet to identify.)  

Books can answer all sorts of questions.  

So what do you need to have on your shelf?

Learning.

Depending on where you are and what you want to learn…

I have a professional library that has leadership, personality development, learning theory/pedagogy,  women’s health, aging, health policy, statistics, measurement, health evaluation, research, evaluation,  methodological books (like scaling, survey design, case study, focus groups, analyzing qualitative data, needs assessment), writing, computer programming, psychology and psychiatry, grant writing, and nutrition. It is my biography of sorts. I can tell you when and what I was doing when I got most of them. Possibly all of them.

At home, I have many women writers, alphabetized. On my bookcases.

I also have a stack (literally) of community library books I will read on the floor by my chair. They are mixed authors.

Much of my library in boxes because I gave the bookcases (which held them) to my daughters and they are now full of books. 

I learn something from each of these; slowly, I am making my way through all of them.

I don’t buy books any more.

I have many. (Some would say too many.) I use the community library.

The adage of “too many books, so little time” is so true. Maybe when I retire…

New topic.

This is the end of 2017. A tough year.

One thing you can do is make a record of “being mindful of the small things you do every day that make you healthy”. No gimmicks, no crazy goals, or fancy gadgets needed (Thank you, Be Orange Challenge)! That could be your resolution for 2018.

That and being active in what ever is your passion.

Happy New Year!

I hope that 2018 will be all that you wish.

my.

molly.

 

Utilization

Posted on by

Another word for use

Another word for utilization is use; how does one use the information gathered? What does one do with what it knows?

I’m reading a book by Warren Bennis, the American scholar who pioneered the contemporary field of leadership studies. He died in July 2014. The book, called Why leaders can’t lead: The unconscious conspiracy continues,  was first written in 1989, and many references are old (read Nixon, Regan). No matter; still relevant, like walking into the world of American politics TODAY (see page 99, specifically on wins and losses).

Bennis says, “The true measure of any society is not what it knows but what it does with what it knows.” (Sounds like use to me.)

 

Use the reports

Now, Michael Quinn Patton (who has written a lot on a lot of topics) writes books on utilization .

The 4th edition of Utilization-Focused Evaluation is 667 pages and the Essentials is 461 pages. (I confess that I’ve only read the preface and scattered other pages of the 667 page version.)

For those of you who do not know Michael, he is the founder and director of Utilization-Focused Evaluation. He says that it is important to use the results of evaluation. Patton advocates that evaluations should be designed with careful consideration of how everything is done.

You (the evaluator) can design evaluations that ensure their usefulness. Long reports may typically never get read or never result in any practical changes.

Utilization-focused evaluation is a process that helps intended (read primary) users make decisions about the evaluation. Patton “support(s) evaluation designed for intended use by intended users.” Continue reading →

Gratitude.

Posted on by

Gratitude.

The day after tomorrow is a national holiday. One of gratitude.

It is the lead up to the end of the year holidays and the long dark.

Yep. Thanksgiving.

(I couldn’t decide which was more representative…certainly the Norman Rockwell painting isn’t; so I didn’t include it.)

Real meaning?

I recently read an article from my Alma mater on the “Real Meaning of Thanksgiving”. What I didn’t know is that “since 1970, Native Americans… commemorate a National Day of Mourning on the US Thanksgiving holiday.”  It is a “reminder of the genocide of millions of indigenous people, the theft of their lands, and the relentless assault on their culture.” This day of remembrance and spiritual connection protests the racism and oppression which the Native Americans continue to experience.  A similar gathering will take place in San Francisco, California, on Alcatraz Island.

This is because of the concept “described by scholars as settler colonialism”.

Ronald Trosper, professor of American Indian studies at the University of Arizona, presents a short quiz about Thanksgiving Day (three questions only). Although he cites the web site  GlobalSocietyTheory.com, that link doesn’t work. He says “Settler colonialism persists in the ongoing elimination of indigenous populations, and the assertion of state sovereignty and juridical control over their lands.” Although Thanksgiving is an US holiday, “…Canada, Australia, New Zealand and South Africa also are examples of countries formed by settler colonialism.”

Criteria.

Everyone has a criteria for determining what Thanksgiving means to them. What is the value, merit, worth of that program.

Is is just a two day break from work? Is it just another holiday? Or is is about the food? For me, I look at the food and I am thankful. I have not gone hungry. Sometimes it is any food; sometimes it is only green food.

This year, I will be celebrating the holiday with my brother, his wife, his son, and his son’s girlfriend. We will feast of foods for which we are thankful. That does not include turkey.

I made a new pie for gathering. Dulce De Leche any one?

 

my.

 

Good ideas.

Posted on by

Good ideas. Maybe.

Did I get good ideas? Maybe.

I recently returned (Saturday,November 11, 2017, late) from the 2017 annual American Evaluation Association conference. This year the meeting was held in Washington, D. C.  (Thank you Lance Wyman, for this photo.) I realize that this is not the iconic view of D.C. that one imagines (like this: .) It was fall and it was mostly clear. I did get to the zoo as part of the conference.

As you know, I determine if a conference is good by seeing three long time friends, meeting three new people I want to see again, and getting three new ideas . This year was bitter sweet. Yes, I did see three long time friends (however, there were only 10). Used to be that I could not go across the lobby without seeing someone I knew well and wanted to see again. This year, many friends (both professional and personal) were not there–they had retired; they were frail and not traveling; they had died and I thought of my own mortality and realized that I had less time to take breaths, even those that take my breath away. I did not meet (although I did interact with young people) three new people I wanted to see again. I think I got only two good ideas–maybe three; hard to say.

Continue reading →

Complexity.

Posted on by

Complexity.

I predict a bright future for complexity. Have you ever considered how complicated things can get, what with one thing always leading to another?

~~E. B. White, Quo VADIMUS? OR THE CASE FOR THE BICYCLE (Garden City. Publishing 1946)

 

Every thing is connected.

One thing does lead to another. And connections between them can be drawn. So let’s connect the dots.

In 2002, as AEA president, I chose for the theme of the meeting, Evaluation a Systematic Process that Reforms Systems.

Evaluation doesn’t stand in isolation; it is not something that is added on at the end as an afterthought.

Many program planners see evaluation that way, unfortunately. Only as an add on, at the end.

Contrary to may peoples’ belief, evaluators need to be included at the outset of the program. They also need to be included at each stage thereafter (Program Implementation, Program Monitoring, and Program Delivery; Data Management and Data Analysis; Program Evaluation Utilization).

Systems Concepts.

Shortly after Evaluation 2002, (in 2004) the Systems Evaluation Topical Interest Group was formed.

AEA published (2007) “Systems Concepts in Evaluation: A Expert Anthology” (scroll to the end). It was edited by Bob Williams and Iraj Imam (who died as this volume was going to press). To order, see this link.

This volume does an excellent job of demonstrating how evaluation and systems concepts are related.

It connects the dots.

In that volume, Gerald Midgley writes about “the intellectual development of systems field, how this has influenced practice and importantly the relevance for all this to evaluators and evaluation”. It is the pivotal chapter (according to the editors).

While it is possible to trace the idea to trace the ideas about holistic thinking back to the ancient Greeks, systems thinking is probably best attributed to the ideas of von Bertalanffy [Bertalanffy, L. von. (1950). Theory of open systems in physics and biology. Science, III: 23-29.]

I would argue that the complexity concept will go back to at least Alexander von Humboldt . (Way before von Bertalanffy.) He was an intrepid explorer and created modern environmentalism. Alexander von Humboldt lived between 1769–1859. Although environmentalism is a complex word, it really is a system. With connections. And complexity.

Suffice it to say, there are no easy answers to the problems faced by professionals today. Complex. Complicated. And one thing leading to another.

my .

Evaluability assessment

Posted on by

Evaluability assessment.

I get my ideas to blog from a lot of places. One place I get ideas are from other blogs.

The content of this blog was Evaluability Assessment.

The blog author says that evaluability assessments tend to cover the following topics:

  • Clarity of the intervention and its objectives: Is there a logical and clear theory of change that articulates how and under what conditions intervention activities influence particular processes of change?
  • Availability of data: which data are available that can be used in assessing the merit and worth of the intervention (e.g. generated by the intervention, external data sets, policy and academic literature)
  • Stakeholder interest and intended use: to what extent is there a clear interest (and capacity) among stakeholders to use the evaluation’s findings and recommendations in strategic decision-making, program improvement, learning about what works, etc.?

What, you ask, is evaluability assessment?

You certainly can go to the blog and read what it says there. OR…You can go to Scriven’s book and read the history on page 138 .

Suffice it to say that evaluability is the extent to which they (projects and programs) can be evaluated. Scriven goes on to say: “It should be thought of as the first commandment of accountability or as the last refinement of Popper’s (Sir Karl Raimond Popper) requirement of falsifiability.

 

Sources.

I learned about evaluability assessment (EA) from Midge Smith (shown here with her husband Carl Wisler) in her book by the same name (published by Springer, search for it by title).  She says that EA is “…a diagnostic and prescriptive tool for improving programs and making evaluations more useful.” Like all tools used in evaluation, it is systematic and describes the structure of a program.

There is a newer volume  of that name. It is by Michael S. Trevisan and Tamara M. Walser (they do an AEA365 blog on that topic). It is not, unfortunately, on my shelf.  The blurb that accompanies the book (by the publisher, Sage) says: “Evaluability assessment (EA) can lead to development of sound program theory, increased stakeholder involvement and empowerment, better understanding of program culture and context, enhanced collaboration and communication, process and findings use, and organizational learning and evaluation capacity building.”

More detail than Midge offers, then her book is copyrighted in 1989.

EA is getting a lot of press lately (you may need to search for evaluability assessment when you go to AEA365).

I find it amazing how previously important things (EA) are now once again in vogue.

my .

 

 

Libraries.

Posted on by

LIBRARIES 

With them, we are amazing!

Without them, we humons are limited.

I can only speak for myself–I do not want to be limited. I am a library champion. (A library champion is someone who fosters public awareness on the extensive range of resources and services available at public, school, academic and special libraries nationwide.)

If you were to see my professional library, perhaps you would understand. (It looks like the photo above, only the shelves are white; I have four bookcases with extensions. The shelves are full.)

If you were to see my personal library, you would understand. Most of it is still in boxes.

Because I believe that literacy is important, When they were young and beginning to read, I gave four of my bookshelves to my daughters (who are now grown, although their bookshelves are still full, mostly, of books). One each is in their room; two are now in the quest room. Hence, my library is mostly still in boxes.

I’ve stopped buying books for personal use (I still get professional ones). I use the library to get hard copy. I have 13 books at home plus two book club books. I have 11 books on hold.

I have an iPad on which I have at least four books and an equal number on hold. (I read a lot.)

But libraries do so much more than provide us with books (still their most important function). They move information in new directions! And they have magic fingers on the keyboard. I would be lost without libraries.

The information to which they have access is astounding.

Libraries and evaluation.

I want to discuss my professional library. I have one whole bookcase (of seven plus shelves) which is filled with books relating to evaluation. Some books are in many editions. And that doesn’t include statistics books or measurement books or the hard-copy journals that have come over the years.

“Why?” you ask, do I have multiple versions of the same (well, almost) book, different editions? Ah. Perhaps one edition will provide the answer (to the puzzle) and the others do not. Does that mean that the answer is not relevant? No. Does that mean that the information is passe? Maybe. Maybe not. The book may be the seminal reference and needs to be sited. It may give a history that isn’t found any place else. It is important to see how the volume changes with each edition. Having multiple volumes adds value. (And the root of evaluation is value.)

Do I need all this? Probably not, especially in the age of the internet and access to all that it provides. Yet, there is something about hard copy; you know a book  (whether a paperback or not), with its binding, its smell, its feel, that cannot be duplicated on-line. Something that cannot be diminished. Something that definitely adds value, merit and worth.

 

“To live in the world without becoming aware of the meaning of the world is like wandering in a great library without touching the books.” ~~Manly P. Hall

My feeling exactly.

 

 

 

Making things (life) better.

Posted on by

Get better.

Unless someone like you cares a whole awful lot, nothing is going to get better. It’s not.

~~Theodor Seuss Geisel

 

Dr. Seuss is a life-time favorite of mine. If he were still alive (unfortunately, he died in 1991), he would still be speaking out (however subtly) and making a difference (he was born in 1904). He opposed discrimination and isolationism and cartooned those issues between 1941-1943.

 

His comment above reminded me about why I’m an evaluator. I do care. A lot.

Yes, I like to solve puzzles.  I like to figure things out. Yes, I really like to make life better.

I use science. And scientific principles. And data. To get answers.

So I have to wonder, how can anyone go through this life without that support? (You know, the support of science and answers provided by data.)

Statistics

Yes, I know…statistics lie (this saying “statistics lie” is often attributed to Mark Twain) and liars use statistics, depending on what the statistic is representing and who is using it. Or figures don’t lie; liars do figure may be more accurate (used by Carroll D. Wright ). (Oh and while I’m talking about statistics, Hans Rosling  died in February [2017]; a major loss.)

There is a lot of press about “big data” these days. And artificial intelligence. Is this the future?  Given that I hear what sound like competing stories about the future I can only wonder.

Even though I care. A lot. Perhaps as one source said, “…is it just a bit of social theatre we perform to make ourselves feel virtuous, useful, and in the right?” Maybe I’m just feeling virtuous and useful.

Maybe I only think I can change the world.

Gloria Anzaldua , an American scholar of  Chicana cultural theory who died in 2004, is quoted as saying, “I change myself; I change the world.”

So because I care (Seuss) and I am changing myself (Anzaldua), perhaps life will get better. Perhaps.

I will stick around and find out. And remember.