May
30
Filed Under (program evaluation) by Molly on 30-05-2012

They [CEO’s] simply expect unpredictability. For them, there is no “new normal.” This is why perpetual Beta is a constant theme here. It is a necessary perspective in dealing with increasing complexity.

This is a piece of interesting information I picked up on a blog about letting go.  I found it interesting reading, especially when viewed wearing my evaluators hat.  Increasingly, the programs we devise address problems which are complex.  No longer are we willing to look at a small slice of a problem find one solution and address it (control, according to Jarche).  Rather, all problems are interrelated; consequently, all solutions must also be interrelated (Jarche talks about letting go [of control]).

Every year, Corvallis Sister Cities-Gondar holds a walk for water.  The idea being to raise funds to support the activities of the Sister Cities programs (in this case clean, potable, available water or planting trees in treeless areas).  I asked the president of the organization if in their teaching activities they were talking about population control.  He answered, sadly, No.  Even though he could see the connection between clean water and population, that step hadn’t been taken in the educational efforts of the program.  But providing funding for individual wells to provide clean water was.  I see this as simplification of a complex problem.  (Your point, Engle?)  Right.  Point.

All the above relates to program planning and developing a logic model for that program.  This tries to remind folks to expect the unexpected in planning the program to address a problem and think about building the unexpected into the model.  Yes, you might get clean water if you teach well digging skills; yet that clean water will reduce infant mortality.  With the reduction of infant mortality, population reaching adulthood will increase.  An increase in population will tax already limited resources (food, water, shelter–not to mention consumer resources); that leads back to the basic problem–no/little access to clean water.  Teaching well drilling skills is only part of the problem; the rest of the problem is unexpected.  By building that into the model, one can see the relation to the bigger picture; the relation to the system.  Even if designing the program will only address that one small part (of the problem), identifying the unexpected, the unpredictable, will help the program planner see clearly.

 


May
23
Filed Under (program evaluation) by Molly on 23-05-2012

Once again, it is the whole ‘balance’ thing…(we) live in ordinary life and that ordinary life is really the only life we have…I’ll take it. It has some great moments…

 

These wise words come from the insights of Buddy Stallings, Episcopal priest in charge of a large parish in a large city in the US.  True, I took them out of context; the important thing is that they resonated with me from an evaluation perspective.

Too often, faculty and colleagues come to me and wonder what the impact is of this or that program.  I wonder, What do they mean?  What do they want to know? Are they only using words they have heard–the buzz words?  I ponder how this fits into their ordinary life. Or are they outside their ordinary life, pretending in a foreign country?

A faculty member at Oregon State University equated history to a foreign country.  I was put in a mind that evaluation is a foreign country to many (most) people, even though everyone evaluates every day, whether they know it or not.  Individuals visit that contry because they are required to visit; to gather information; to report what they discovered.  They do this with out any special preparation.  Visiting a foreign country entails preparation (at least it does for me).  A study of customs, mores, foods, language, behavior, tools (I’m sure I’m missing something important in this list) is needed; not just necessary, mandatory.  Because although the foreign country may be exotic and unique and novel to you, it is ordinary life for everyone who lives there.  The same is true for evaluation.  There are customs; students are socialized to think and act in a certain way.  Mores are constantly being called into question; language, behaviors, tools, which not known to you in your ordinary life, present themselves. You are constantly presented with opportunities to be outside your ordinary life.  Yet, I wonder what are you missing by not seeing the ordinary; by pretending that it is extraordinary?  By not doing the preparation to make evaluation part of your ordinary life, something you do without thinking.

So I ask you, What preparation have you done to visit this foreign country called EVALUATION?  What are you currently doing to increase your understanding of this country?  How does this visit change your ordinary life or can you get those great moments by recognizing that this is truly the only life you have?   So I ask you, What are you really asking when you ask, What are the impacts?

 

All of this has significant implications for capacity building.

May
11
Filed Under (program evaluation) by Molly on 11-05-2012

Today, I’m writing about several random thoughts which have occurred to me in the last two weeks–some prompted by comments; some not.  And before I forget–I was sick last week and didn’t blog (did you miss me?); I’ll be gone next week and won’t blog (will you miss me?).  I’ll be back the week of May 21.

Random thought 1.

I’m doing this evaluation capacity building program for folks who are interested in evaluation.  Several (more than 2) participants have commented to me, in person or electronically, that they are being asked to use their evaluation expertise garnered from this program in projects that are in their content area and are not their programs–serving as evaluation consultants, if you will.  This evaluation capacity building program draws Extension professionals from across the western states and includes folks from natural resources, agriculture, family and community science, nutrition, and 4-H.  It is another example in my life where I put information together (like this blog) and send it out and rarely know what happens at the other end of the send.  Getting comments like this is heart warming. Capturing these stories will be part of our summative evaluation.

Random thought 2.

Although I am a program evaluator, specializing in community-based educational programs, there are other kinds of evaluation. (Note: This is not a discussion of evaluation models, like naturalistic evaluation or developmental evaluation–that is another discussion.)   Evaluations can be conducted on products, processes, policy (including proposals, plans, and possibilities), performance, personnel as well as program.  Michael Scriven, in his Evaluation Thesaurus discusses all these.  The goal of all forms of evaluation is to determine the merit, worth, or effectiveness of the thing being evaluated (evaluand).

Random thought 3.

I love getting comments about what I write.  If you comment, I may not respond.  I always read them.  One comment was about subscribing.  You can subscribe to the blog through an RSS feed button in the upper left side of the blog (under the word, Subscribe).  You can also subscribe by email.  The blog postings are archived (thank you OSU), so you can go back and see what I’ve said.  I try not to repeat myself.

Random thought 4.

Because I blog as part of my work, and because I work at Oregon State University, I use the resources available to me there (like wordpress for this blog).  It is written using Firefox NOT Internet Explorer.  The appearance may be different depending on the browser used.  Try viewing it in a different browser if you are having trouble seeing the blog.

Random thought 5.

Most folks who are program people know a lot about their content area, whether it is invasive species, viticulture, woodlot management, nutrition, or youth development.  What I’m finding is that knowing what to do with the data they have about their programs isn’t as well known.  The data from most of what Extension does can be analyzed using relatively simple statistics, such as chi-square, t-test, ANOVA.  Sometimes using an ANCOVA is needed.  There are many useful resources available for broadening understanding about statistics.  One I really like is a TED Talk by Hans Rosling.