merit « Evaluation is an Everyday Activity

At a loss for what to write, I once again went to one of my favorite books, Michael Scriven’s Evaluation Thesaurus . This time when I opened the volume randomly, I came upon the entry for meta-evaluation. This is a worthy topic, one that isn’t addressed often. So this week, I’ll talk about meta-evaluation and quote Scriven as I do.

First, what is meta-evaluation? This is an evaluation approach which is the evaluation of evaluations (and “indirectly, the evaluation of evaluators”). Scriven suggests the application of an evaluation-specific checklist or a Key Evaluation Checklist (KEC) (p. 228). Although this approach can be used to evaluate one’s own work, the results are typically unreliable which implies (if one can afford it) to use an independent evaluator to conduct a meta-evaluation of your evaluations.

Then, Scriven goes on to say the following key points:

Meta-evaluation is the professional imperative of evaluation;
Meta-evaluation can be done formatively or summatively or both; and
Use the KEC to generate a new evaluation OR apply the checklist to the original evaluation as a product.

He lists the parts a KEC involved in a meta evaluation; this process includes 13 steps (pp. 230-231).

He gives the following reference:

Stufflebeam, D. (1981). Meta-evaluation: Concepts, standards, and uses. In R. Berk (Ed.), Educational evaluation methodology: The state of the art. Baltimore, MD: Johns Hopkins.

About two years ago, I conducted a 17 month hybrid evaluation preparation program for the Western Region Extension Service faculty. There were over 30 individuals involved. I was the evaluation expert; Jim Lindstrom (who was at WSU at the time) was the cheerleader, the encourager, the professional development person. I really couldn’t have done it without him. (Thank you, Jim.) Now, to maximize this program and make it available to others who were not able to participate, I’ve been asked to explore an option for creating an on-line version of the WECT (say west) program. It would be loaded through the OSU professional and continuing education (PACE) venue. To that end, I am calling on those of you who participated in the original program (and any other readers) to provide me with feedback of the following:

What was useful?
What needed to be added?
What could be more in depth?
What could be deleted?
Other comments?

Please be as specific as possible.

I can go to the competency literature (of which there is a lot) and redevelop WECT from those guidelines. (For more information on competencies see: King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22(2), 229-247.) Or I could use the Canadian system as a foundation. (For more information see this link.)

I doubt if I can develop an on-line version that would cover (or do justice) to all those competencies.

So I turn to you my readers. Let me know what you think.

my .

molly.

Last week, I started a discussion on inappropriate evaluations. I was using the Fitzpatrick , Sanders , and Worthen text for the discussion (Program Evaluation: Alternative approaches and practical guidelines, 2011. See here.) There were three other examples given in that text which were:

Evaluation cannot yield useful, valid information;
Type of evaluation is premature for the stage of the program; and
Propriety of evaluation is doubtful.

I will cover them today.

First, if the evaluation doesn’t (or isn’t likely to) produce relevant information, don’t do it. If factors like inadequate resources–personnel, funding, time, lack of administrative support, impossible evaluation tasks, or inaccessible data (which are typically outside the evaluator’s control), give it a pass as all of these factors make the likelihood that the evaluation will yield useful, valid information slim. Fitzpatrick, Sanders, and Worthen say, “A bad evaluation is worse than none at all…”.

Then consider the type of evaluation that is requested. Should you do a formative, a summative, or a developmental evaluation? The tryout phase of a program typically demands a formative evaluation and not a summative evaluations despite the need to demonstrate impact. You may not demonstrate an effect at all because of timing. Consider running the program for a while (more than once or twice in a month). Decide if you are going to use the results for only programmatic improvement or for programmatic improvement AND impact.

Finally consider if the propriety of the evaluation is worthwhile. Propriety is the third standard in the Joint Committee Standards . Propriety helps establish evaluation quality by protecting the rights of those involved in the evaluation–the target audience, the evaluators, program staff, and other stakeholders. If you haven’t read the Standards, I recommend that you do.

New Topic (and timely): Comments.

It has been a while since I’ve commented on any feedback I get in the form of comments on blog posts. I read everyone. I get them both here as I write and as an email. Sometimes they are in a language I don’t read or understand and, unfortunately, the on-line translators don’t always make sense. Sometimes they are encouraging comments (keep writing; keep blogging; thank you; etc.). Sometimes there are substantive comments that lead me to think about things evaluation differently. Regardless of what the message is: THANK YOU! For commenting. Remember, I read each one.

my .

molly.

Can there be inappropriate use of evaluation studies?

Jody Fitzpatrick¹ and her co-authors Jim Sanders and Blaine Worthen, in Program Evaluation: Alternative Approaches and Practical Guidelines (2011) provide several examples of inappropriate evaluation use. Before they give the examples, they share some wise words from Nick Smith² . Nick says there are two broad categories for declining conducting an evaluation. They are “1) when the evaluation could harm the field of evaluation, or 2) when it would fail to support the social good.” Fitzpatrick, Sanders, and Worthen (2011) go on to say that “these problems may arise when it is likely that the ultimate quality of the evaluation will be questionable, major clients would be alienated or misled concerning what evaluation can do, resources will be inadequate, or ethical principles would be violated” (p. 265).

The examples provided are

Evaluation would produce trivial information;
Evaluation results will not be used;
Evaluation cannot yield useful, valid information;
Type of evaluation is premature for the stage of the program; and
Propriety of evaluation is doubtful.

When I study these examples (there may be others; I’m quoting Fitzpatrick, Sanders, and Worthen, 2011), I find that these are examples often found in the published literature. As a reviewer, I find “show and tell” evaluations of little value because they produce trivial information. They report a study that has limited or insufficient impact and that has little or no potential for continuation. The cost of conducting a formal evaluation would easily outweigh the value–if monetized–(merit or worth) of the program and would yield little information useful for others in the field. The intention might be well designed; the product is less than ideal. Continue reading →

I would like to think that the world is a better place than it was 50 years ago. In many ways I suppose it is; I wish that were true in all ways.

Human rights were violated in the name of religious freedom in Indiana last week (not to mention the other 19 states which have Religious Freedom Restoration Act). “The statute shows every sign of having been carefully designed to put new obstacles in the path of equality; and it has been publicly sold with deceptive claims that it is ‘nothing new’.” (Thank you, Garrett Epps). Then there are the those states which follow the Hobby Lobby ruling, a different set.

The eve of Passover is Friday (which also happens to be Good Friday). Passover (or Pesach) is a celebration commemorating the Israelites freedom from slavery imposed by ancient Egypt. Putting an orange on the Seder plate helps remember that liberation specifically the liberation of the marginalized.

Passover is the only holiday that celebrates human rights and individual freedoms.

Does anyone else see the irony with this Indiana law?

This is an evaluation issue. How can you make a difference if you restrict liberation (like the recently passed Indiana law)? What is the merit, the worth, the value of restriction? I don’t think there is any.

my .

molly.

Today is the middle of Spring Break at Oregon State University.

What did you do today that involved thinking evaluatively?

Did you decide to go to work?

Did you decide to go to the beach?

Did you decide you were sick?

Did you decide you would work in the yard/garden?

Did you decide to stop and smell the roses? Continue reading →

A recent blog (not mine) talked about the client’s evaluation use. The author says that she feels “…successful…if the client is using the data…” This statement allowed me to stop and pause and think about data use. The author continues with the comment about the difference between “…facilitating the client’s understanding of the data in order to create plans and telling the client exactly what the data means and what to do with it.”

I work with Extension professionals who may or may not understand the methodology, the data analysis, or the results. How does one communicate with Extension professionals who may be experts in their content area (cereal crops, nutrition, aging, invasive species) and know little about the survey on which they worked? Is my best guess (not knowing the content area) a good guess? Do Extension professionals really use the evaluation findings? If I suggest that the findings could say this, or suggest that the findings could say that, am I preventing a learning opportunity from happening? Continue reading →

This is a link to an editorial in Basic and Applied Social Psychology. It says that inferential statistics are no longer allowed by authors in the journal.

“What?”, you ask. Does that have anything to do with evaluation? Yes and no. Most of my readers will not publish here. They will publish in evaluation journals (of which there are many) or if they are Extension professionals, they will publish in the Journal of Extension. And as far as I know, BASP is the only journal which has established an outright ban on inferential statistics. So evaluation journals and JoE still accept inferential statistics.

Still–if one journal can ban the use, can others?

What exactly does that mean–no inferential statistics? The journal editors define this ban as as “…the null hypothesis significance testing procedure is invalid and thus authors would be not required to perform it.” That means that authors will remove all references to p-values, t-values, F-values, or any reference to statements about significant difference (or lack thereof) prior to publication. The editors go on to discuss the use of confidence intervals (No) and Bayesian methods (case-by case) and what inferential statistical procedures are required by the journal. Continue reading →

Earlier this week I attended a meeting of the College of Education (my academic home) Social and Environmental Justice (SJE) Work Group. This is a loosely organized group of interested faculty and staff, led by an individual who is the ESOL Program Coordinator & Instructor. We had representatives from each of the four program areas (Adult and Higher Education [AHE], Teacher and Counseling Education [TCE], Science, Technology, Engineering, and Math [STEM], and Cultural and Linguistic Diversity [CLD]) in person (AHE, TCE,. CLD) or on paper [STEM]. The intent was to document for the work group, what each program area is doing in the area of social justice. Social Justice is a mandate for the College and OSU. The AHE and the TCE representatives provided us with information. We never did get to the STEM response. Then we got on to a discussion of what exactly is meant by social justice (since AHE has not defined the term specifically). My response was the evaluation response: it depends.

Most of the folks in the group focused on the interface of race and gender. OK. Others focused on the multiple and different voices. OK. Others focused on the advantages and disadvantages experienced. How is that not based in economics? Others focused on power and privileged. (As an accident of birth?) What is social justice exactly? Can you have social justice without environmental justice? How does that fit with the issue of diversity? How does any of this relate to evaluation?

The American Evaluation Association has had in place for a long time (since 1994) a set of five guiding principles (see Background section at the link for a bit of history). The fourth and fifth principles are, respectively, Respect for People and Responsibilities for General and Public Welfare. Respect for people says this: Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. Responsibilities for the General and Public Welfare says this: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation. Although both talk about parts of social justice that we talked about earlier this week, is this a complete view? Certainly, security, dignity, and self worth and diversity of interests and values approach the discussion we had. Is there still something missing? I think so. Where is fairness addressed?

To me, fairness is the crux of the issue. For example, it certainly isn’t fair that in the US, 2% of the population has the cumulative wealth of the remaining 98%. (Then we are into economics.) Although Gandhi said “be the change” is that enough? What if that change isn’t fair? And the question must be addressed, fair to whom? What if that change is only one person? Is that fair? I always talk about the long term outcome as world peace (not in my lifetime, though). If you work for justice (for me that is fairness) will peace result? I don’t know. Maybe.

Tomorrow is the Lunar New Year. It is the year of the goat/sheep/ram. I wish you the best. Eat jiaozi and tangerines (for encouraging wealth), and noodles without breaking/biting them (you do want a long life, right?). Happy New Year.

I don’t know what to write today for this week’s post. I turn to my book shelf and randomly choose a book. Alas, I get distracted and don’t remember what I’m about. Mama said there would be days like this…I’ve got writer’s block (fortunately, it is not contagious). (Thank you, Calvin). There is also an interesting (to me at least because I learned a new word–thrisis: a crisis of the thirties) blog on this very topic (here).

So this is what I decided rather than trying to refocus. In the past 48 hours I’ve had the following discussions that relate to evaluation and evaluative thinking.

In a faculty meeting yesterday, there was the discussion of student needs which occur during the students’ matriculation in a program of study. Perhaps it should include assets in addition to needs as students often don’t know what they don’t know and cannot identify needs.
A faculty member wanted to validate and establish the reliability for a survey being constructed. Do I review the survey, provide the reference for survey development, OR give a reference for validity and reliability (a measurement text)? Or all of the above.
There appears to be two virtual focus group transcripts for a qualitative evaluation that have gone missing. How much affect will those missing focus groups have on the evaluation? Will notes taken during the sessions be sufficient?
A candidate came to campus for an assistant professor position who presented a research presentation on the right hand (as opposed to the left hand) [Euphemisms for the talk content to protect confidentiality.] Why even study the right hand when the left hand is what is the assessment?
Reading over a professional development proposal dealing with what is, what could be, and what should be. Are the questions being asked really addressing the question of gaps?

Continue reading →

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: merit

Meta-evaluation

Preparation of evaluators

More Inappropriate Evaluations

Inappropriate Evaluation?

Liberation

Spring Break

Evaluation use

Inferential statistics

Social Justice

Blogging and Writer’s Block

Contact Info