change « Evaluation is an Everyday Activity

Chris Lysy, at Fresh Spectrum, had a guest contributor in his most recent blog, Rakesh Mohan.

Rakesh says “…evaluators forget that evaluation is inherently political because it involves making judgment about prioritization, distribution, and use of resources.”

I agree that evaluators can make judgements about prioritization, distribution and resource use. I wonder if making judgements is built in to the role of evaluator; is even taught to the nascent evaluator? I also wonder if the Principal Investigator (PI) has much to say about the judgements. What if the evaluator interprets the findings one way and the PI doesn’t agree. Is that political? Or not. Does the PI have the final say about what the outcomes mean (the prioritization, distribution, and resource use)? Does the evaluator make recommendations or does the evaluator only draw conclusions? Then where do comments on the prioritization, the distribution, the resource use come into the discussion? Are they recommendations or are they conclusions?

I decided I would see what my library says about politics: Scriven’s Thesaurus* talks about the politics of evaluation; Fitzpatrick, Sanders, and Worthen* have a chapter on “Political, Interpersonal, and Ethical Issues in Evaluation” (chapter 3); Rossi, Lipsey, and Freeman* have a section on political context (pp. 18-20) and a section on political process (pp. 381-393) that includes policy and policy implications. The 1982 Cronbach* volume (Designing Evaluatations of Educational and Social Programs) has a brief discussion (of multiple perspectives) and the classic 1980 volume, Toward Reform of Program Evaluation, also addresses the topic*. Least I neglect to include those authors who ascribe to the naturalistic approaches, Guba and Lincoln talk about the politics of evaluation (pp. 295-299) in their1981 volume, Effective Evaluation . The political aspects of evaluation have been part of the field for a long time.

So–because politics has been and continues to be part of evaluation, perhaps what Mohan says is relevant. When I look at Scriven’s comments in the Thesauras, the comment that stands out is, “Better education for the citizen about –and in–evaluation, may be the best route to improvement, short of a political leader with the charisma to persuade us of anything and the brains to persuade us to imporve our critical thinking.” Since the likelihood that we will see a political leader to persuade us is slim, perhaps education is the best approach. And like Mohan says, invite them to the conference. (After all, education comes in all sizes and experiences.) Perhaps then policy makers, politicians, press, and public will be able understand and make a difference BECAUSE OF EVALUATION!

*Scriven, M. (1991). Evaluation thesaurus. Newbury Park, CA: Sage.

*Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines. (4th ed.) Boston, MA: Pearson

*Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.) Thousand Oaks, CA: Sage.

*Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass Inc. Publishers.

*Cronbach, L. J. et al. (1980). Toward reform of program evaluation. San Francisco, CA: Jossey-Bass Inc. Publishers.

*Guba, E. G. & Lincoln, Y. S. (1981). Effective evaluation. San Francisco, CA: Jossey-Bass Inc. Publishers.

Like many people, I find change hard. In fact, I really don’t like change. I think this is the result of a high school experience; one-third of my classmates left each year. (I was a military off-spring; we changed assignments every three years.)

Yet, in today’s world change is probably the only constant. Does that make it fun? Not necessarily. Does that make it easy? Nope. Does that make it necessary? Yep.

Evaluators deal with change regularly. New programs are required; those must be evaluated. Old programs are revised; those must be evaluated. New approaches are developed and presented to the field. (When I first became an evaluator, there wasn’t a systems approach to evaluation; there wasn’t developmental evaluation; I could continue.) New technologies are available and must be used even if the old one wasn’t broken (even for those of us who are techno-peasants).

I just finished a major qualitative evaluation that involved real-time virtual focus groups. When I researched this topic (virtual focus groups), I found a lot of information about non-synchronous focus groups, focus groups using a conferencing software, even synchronous focus groups without pictures. I didn’t find anything about using real-time synchronous virtual focus groups. Unfortunately, we didn’t have much money even though there are services available. Continue reading →

At a loss for what to write, I once again went to one of my favorite books, Michael Scriven’s Evaluation Thesaurus . This time when I opened the volume randomly, I came upon the entry for meta-evaluation. This is a worthy topic, one that isn’t addressed often. So this week, I’ll talk about meta-evaluation and quote Scriven as I do.

First, what is meta-evaluation? This is an evaluation approach which is the evaluation of evaluations (and “indirectly, the evaluation of evaluators”). Scriven suggests the application of an evaluation-specific checklist or a Key Evaluation Checklist (KEC) (p. 228). Although this approach can be used to evaluate one’s own work, the results are typically unreliable which implies (if one can afford it) to use an independent evaluator to conduct a meta-evaluation of your evaluations.

Then, Scriven goes on to say the following key points:

Meta-evaluation is the professional imperative of evaluation;
Meta-evaluation can be done formatively or summatively or both; and
Use the KEC to generate a new evaluation OR apply the checklist to the original evaluation as a product.

He lists the parts a KEC involved in a meta evaluation; this process includes 13 steps (pp. 230-231).

He gives the following reference:

Stufflebeam, D. (1981). Meta-evaluation: Concepts, standards, and uses. In R. Berk (Ed.), Educational evaluation methodology: The state of the art. Baltimore, MD: Johns Hopkins.

I don’t know what to write today for this week’s post. I turn to my book shelf and randomly choose a book. Alas, I get distracted and don’t remember what I’m about. Mama said there would be days like this…I’ve got writer’s block (fortunately, it is not contagious). (Thank you, Calvin). There is also an interesting (to me at least because I learned a new word–thrisis: a crisis of the thirties) blog on this very topic (here).

So this is what I decided rather than trying to refocus. In the past 48 hours I’ve had the following discussions that relate to evaluation and evaluative thinking.

In a faculty meeting yesterday, there was the discussion of student needs which occur during the students’ matriculation in a program of study. Perhaps it should include assets in addition to needs as students often don’t know what they don’t know and cannot identify needs.
A faculty member wanted to validate and establish the reliability for a survey being constructed. Do I review the survey, provide the reference for survey development, OR give a reference for validity and reliability (a measurement text)? Or all of the above.
There appears to be two virtual focus group transcripts for a qualitative evaluation that have gone missing. How much affect will those missing focus groups have on the evaluation? Will notes taken during the sessions be sufficient?
A candidate came to campus for an assistant professor position who presented a research presentation on the right hand (as opposed to the left hand) [Euphemisms for the talk content to protect confidentiality.] Why even study the right hand when the left hand is what is the assessment?
Reading over a professional development proposal dealing with what is, what could be, and what should be. Are the questions being asked really addressing the question of gaps?

Continue reading →

I was reading Harold Jarche’s blog, Perpetual Beta and he is talking about the interface of the digital and analog worlds; he is talking about connections. I find that that concept applies to evaluators. Here’s how.

I was trained as an evaluator in the 1980s; we didn’t have access to the web, internet, email, FB, or many of the other high technology options available today. I did an NDE (wasn’t called that then) volume (Vol. 62) which was all done by hard copy and the USPS (a tedious and laborious process). I just completed another NDE (now called that) volume (Vol. 145) that was done electronically (no longer tedious, though still a laborious process). This last volume was quick. Although most of the authors entered the field after I did, my co-editor didn’t; he entered before I did. These authors had the luxury of electronics that we didn’t have. I have had to learn how to use electronics; I know my co-editor did, too. (I remember saying to myself and my colleagues, what will my secretary [yes, we used that title] do now that I’m composing on a key-board?) Now I do all my composing and other things on a keyboard; most of my work is augmented with electronics (i.e., the digital world). But I am truly a digital immigrant; learning how to use this new technology, to be in this digital world, is hard for and on me. (My children laugh at me and get exasperated; it is so simple to them.)

Today’s evaluators are highly connected, largely due to the electronic capabilities. Yet there is still evaluator isolation. Makes me wonder if evaluators really connected? Sheila Robinson (the only evaluator in her organization bemoans this fact here). She advocates for connections through EvalTalk and AEA’s LinkedIn account. I still see these as digital, albeit, opportunities to connect. Social media is also mentioned for connections. Still I wonder–are you really connected? With colleagues scattered around the world, this may truly be the only way to stay connected today. Letters and phone calls are truly analog and perhaps passe. Still they are appreciated and sometimes welcome as the only way to connect. What will this world look like if the only way to connect is via digital?

Jarach advocates changing the way we organize. To me that talks about changing the way we work. Maybe evaluators will work in isolation. In order to “see as many possible roads ahead”, perhaps we need to “work in self-managing networks”. “If those who are educated, knowledgeable, and experienced do not push for a better world of work, then who will? An effective knowledge network cultivates the diversity and autonomy of each worker. Knowledge networks function best when each person can choose with whom and when they connect. Solving problems together is becoming the real business challenge.” (From Jarach)

And that affects evaluators!

To whom are you connected? How do you connect? Are you caught in-between?

my.

molly.

Personal and situational bias are forms of cognitive bias and we all have cognitive bias.

When I did my dissertation on personal and situational biases, I was talking about cognitive bias (only I didn’t know it, then).

According to Wikipedia, the term cognitive bias was introduced in 1972 (I defended my dissertation in 1983) by two psychologists Daniel Kahneman and Amos Tversky .

Then, I hypothesized that previous research experience (naive or sophisticated) and the effects of exposure to expected project outcomes (positive, mixed, negative) would affect the participant and make a difference in how the participant would code data. (It did.) The Sadler article which talked about intuitive data processing was the basis for this inquiry. Now many years later, I am encountering cognitive bias again. Sadler says that “…some biases can be traced to a particular background knowledge…”(or possibly–I think–lack of knowledge), “…prior experience, emotional makeup or world view”. bias 4 (This, I think, falls under the category of, according to Tversky and Kahneman, human judgements and it will differ from rational choice theory (often given that label). Continue reading →

It has been about /years since I started this blog (more or less–my anniversary is actually in early December) .

Because I am an evaluator, I have asked several time is this blog making a difference. And those posts, the ones in which I ask “is this blog making a difference”, are the ones which get the most comments. Now, truly, most comments are often either about marketing some product, inviting me to view another blog, mirroring comments made previously, or comments in a language which I cannot read (even with an online translator). Yet, there must be something about “making a difference” that engages viewers and then engages them to make a comment.

Today, I read a comment Continue reading →

I read. A lot.

I also blog. Weekly, unless I’m not in the office.

This past week I read (again) Harold Jarche’s blog. He posts periodically on interesting social media finds. Some of these finds are relevant to evaluation (even if they are not framed that way). His post on October 17 included a post from Kate Pinner called Half-baked ideas (She is found on twitter @kmpinner ). She says, “Just because you know how to do something doesn’t mean you should: It’s rewarding to give other people a chance to shine.”

Pinner’s comment is related to a thought I’ve been mulling for some time now (a couple of years, actually). That is the whole idea of “doing as.”

David Fetterman talks about empowerment evaluation Continue reading →

Having just returned from the annual AEA conference (Evaluation 2014) in Denver, I am taking this moment to reflect, process, and apply.

For years my criteria for a “good” conference was the following

See three long time friends and spend some time catching up;
Meet three people I didn’t know before and would like to continue to know;
Get three new ideas that I can use.

I think this year’s conference was a success (despite the difficulty in identifying who was doing what when because the management corporation minimized the program in an attempt to be ecological, if excluding). If I were to ask my daughters to rate the conference on a scale of one (1) to 10 (ten), one being not “good”, 10 being “good”, I think they would have said an 8 – 8.5. (They have their own following of friends and their own interests.)

I saw and talked to three long time friends, although I missed those who have chosen not to attend AEA any more (I must be getting old) and those with whom I didn’t spend time.

I met more than three people I didn’t know before and I must say, if they are any indication (and I think they are) of the evolution of the association, the association is in good hands (even though I miss the intimacy I “grew up with”). Continue reading →

This summer I spent a lot of time dealing with needs assessments and talking about needs and assets. It occurred to me that the difference between need and wants has a lot to do with evaluation (among other things). So what are needs? What are wants? How does all this relate to evaluation?

Needs.

Maslow spoke eloquently about needs in his hierarchy, and although the hierarchy is often presented as a pyramid, Maslow didn’t present the needs this way. He did present this hierarchy as a set of building blocks with basic needs (physiological) as the foundation, followed by safety, loving/belonging, esteem, and self-actualization. He talks about this theory of motivation in his book, Motivation and Personality (a 3rd edition is available as well). This view of the individual ushered in the humanistic view of psychology (often called the third theory after behaviorism and psychoanalysis). He believed that human could not live without these needs and advocated that they are necessary for survival.

Wants

A “want” is often considered a desire based purely in economic, social, or psychological reality of human existence. It is something that an individual would like to have. (Chocolate, any one?) A want is not essential to human existence; it is only something an individual would like to have. Unfortunately, there are limited resources (as well as a large body of literature) talking about having enough. If you have enough, then wants are few and resources are available for everyone.

Continue reading →

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: change

Evaluation as political.

Change

Meta-evaluation

Blogging and Writer’s Block

Caught between

Personal and Situational Bias-Cognitive Bias by another name?

Update–making a difference

Experts–doing as?

A good conference

Needs, wants, and evaluation

Contact Info