Apr
17
Filed Under (criteria, program evaluation) by Molly on 17-04-2015

Last week, I started a discussion on inappropriate evaluations. I was using the Fitzpatrick jody fitzpatrick, Sanders Jim Sanders, and Worthen blaine worthen text for the discussion fitzpatrick book 2 (Program Evaluation: Alternative approaches and practical guidelines, 2011. See here.) There were three other examples given in that text which were:

  1. Evaluation cannot yield useful, valid information;
  2. Type of evaluation is premature for the stage of the program; and
  3. Propriety of evaluation is doubtful.

I will cover them today.

First, if the evaluation doesn’t (or isn’t likely to) produce relevant information, don’t do it. If factors like inadequate resources–personnel, funding, time, lack of administrative support, impossible evaluation tasks, or inaccessible data (which are typically outside the evaluator’s control), give it a pass as all of these factors make the likelihood that the evaluation will yield useful, valid information slim. Fitzpatrick, Sanders, and Worthen say, “A bad evaluation is worse than none at all…”.

Then consider the type of evaluation that is requested. Should you do a formative, a summative, or a developmental evaluation? The tryout phase of a program typically demands a formative evaluation and not a summative evaluations despite the need to demonstrate impact. You may not demonstrate an effect at all because of timing. Consider running the program for a while (more than once or twice in a month). Decide if you are going to use the results for only programmatic improvement or for programmatic improvement AND impact.

Finally consider if the propriety of the evaluation is worthwhile. Propriety is the third standard in the Joint Committee Standards The_Program_Evaluation_Standards_3ed. Propriety helps establish evaluation quality by protecting the rights of those involved in the evaluation–the target audience, the evaluators, program staff, and other stakeholders. If you haven’t read the Standards, I recommend that you do.

 

New Topic (and timely): Comments.

It has been a while since I’ve commented on any feedback I get in the form of comments on blog posts. I read everyone. I get them both here as I write and as an email. Sometimes they are in a language I don’t read or understand and, unfortunately, the on-line translators don’t always make sense. Sometimes they are encouraging comments (keep writing; keep blogging; thank you; etc.). Sometimes there are substantive comments that lead me to think about things evaluation differently. Regardless of what the message is: THANK YOU! For commenting. Remember, I read each one.

my two cents.

molly.

Apr
10
Filed Under (criteria, program evaluation) by Molly on 10-04-2015

Can there be inappropriate use of evaluation studies?

Jody Fitzpatrick¹ jody fitzpatrick and her co-authors Jim SandersJim Sanders and Blaine Worthen,blaine worthen in Program Evaluation: Alternative Approaches and Practical Guidelines (2011)fitzpatrick book 2 provide several examples of inappropriate evaluation use. Before they give the examples, they share some wise words from Nick Smith² NickSmith_forweb. Nick says there are two broad categories for declining conducting an evaluation. They are “1) when the evaluation could harm the field of evaluation, or 2) when it would fail to support the social good.” Fitzpatrick, Sanders, and Worthen (2011) go on to say that “these problems may arise when it is likely that the ultimate quality of the evaluation will be questionable, major clients would be alienated or misled concerning what evaluation can do, resources will be inadequate, or ethical principles would be violated” (p. 265).

The examples provided are

  1. Evaluation would produce trivial information;
  2. Evaluation results will not be used;
  3. Evaluation cannot yield useful, valid information;
  4. Type of evaluation is premature for the stage of the program; and
  5. Propriety of evaluation is doubtful.

When I  study these examples (there may be others; I’m quoting Fitzpatrick, Sanders, and Worthen, 2011), I find that these are examples often found in the published literature. As a reviewer, I find “show and tell” evaluations of little value because they produce trivial information. They report a study that has limited or insufficient impact and that has little or no potential for continuation. The cost of conducting a formal evaluation would easily outweigh the value–if monetized–(merit or worth) of the program and would yield little information useful for others in the field. The intention might be well designed; the product is less than ideal. Read the rest of this entry »

Nov
25
Filed Under (criteria, program evaluation) by Molly on 25-11-2014

Thanksgiving

Thanksgiving.

A uniquely American holiday (although it is celebrated in other countries as well-Canada, Liberia, The Netherlands, Norfolk Islands),

filled with too much food (pie any one?) apple pie,

too much football (what is your favorite rivalry?),football rivalry

and too much shopping (black Friday?).black-friday 3

 

For me it is an opportunity to to be grateful–and I am, more than words can express. I am especially grateful for my daughters, bright, articulate, and caring children (who are also adults). Read the rest of this entry »

Nov
19
Filed Under (criteria, program evaluation) by Molly on 19-11-2014

It all depends.

The classic evaluation response. In fact, it is the punch line for one of the few evaluation jokes I can remember (some-timers disease being what it is; if you want to know the joke, ask in your comment).

The response reminds me of something I heard (once again) while I was in Denver. One of the presenters at a session on competencies, certification, credentialing (an indirectly, about accreditation) talked about a criteria for evaluators that is not taught in preparatory programs–the tolerance for ambiguity.  Tolerance for Ambiguity (What do you see in this image?)

What is this tolerance? What is ambiguity?

According to Webster’s Seventh, tolerance is the noun form of the verb “to tolerate” and means “…the relative capacity to endure or adapt physiologically to an unfavorable environmental factor…” also defined as “…sympathy or indulgence for beliefs or practices differing from or conflicting with one’s own; the act of allowing something; allowable deviation from a standard…”.

Using the same source, ambiguity (also a noun) means “…the quality or state of being ambiguous in meaning…” OK. Going on to ambiguous (the root of the word), it  is an adjective meaning “…doubtful or uncertain especially from obscurity or indistinctness…capable of being understood in two or more possible senses…”. Personally, I find the “capable of being understood in two or more possible senses…” relevant to evaluation and to evaluators.

Yet, I have to ask, What does all that mean? It all depends.

Read the rest of this entry »

Nov
05
Filed Under (criteria, program evaluation) by Molly on 05-11-2014

It has been about  five/chinese_symbols_number fiveyears since I started this blog (more or less–my anniversary is actually in early December) .

Because I am an evaluator, I have asked several time is this blog making a difference. And those posts, the ones in which I ask “is this blog making a difference”, are the ones which get the most comments.  Now, truly, most comments are often either about marketing some product, inviting me to view another blog, mirroring comments made previously, or comments in a language which I cannot read (even with an online translator). Yet, there must be something about “making a difference” that engages viewers and then engages them to make a comment.

Today, I read a comment Read the rest of this entry »

Oct
23
Filed Under (criteria, program evaluation) by Molly on 23-10-2014

Having just returned from the annual AEA conference (Evaluation 2014) in Denver, I am taking this moment to reflect, process, and apply.front range of the rocky mountains

For years my criteria for a “good” conference was the following

  • See three long time friends and spend some time catching up;
  • Meet three people I didn’t know before and would like to continue to know;
  • Get three new ideas that I can use.

I think this year’s conference was a success (despite the difficulty in identifying who was doing what when because the management corporation minimized the program in an attempt to be ecological, if excluding). If I were to ask my daughters to rate the conference on a scale of one (1) to 10 (ten), one being not “good”, 10 being “good”, I think they would have said an 8 – 8.5. (They have their own following of friends and their own interests.)

I saw and talked to three long time friends, although I missed those who have chosen not to attend AEA any more (I must be getting old) and those with whom I didn’t spend time.

I met more than three people I didn’t know before and I must say, if they are any indication (and I think they are) of the evolution of the association, the association is in good hands (even though I miss the intimacy I “grew up with”). Read the rest of this entry »

Sep
19
Filed Under (criteria, program evaluation) by Molly on 19-09-2014

I’ve just spent all of August and most of September editing chapters for a volume of New Directions in Evaluation (NDE) on Accreditation, Certification, and Credentialing. These topics all relate to competencies which all relate to building capacity. Now I can site a lot of references for competencies. (For example, Stevahn, King, Ghere, Minnema, 2005, AJE 26(1), pp. 43-59., among others see the work by King and cadre–that one cited just happens to be on my desk right now.) This group has been working on competencies for the last 15 or so years. This is important work–as well as problematic (hence the issue of NDE). I won’t go into details here because the NDE volume pretty much addresses these issues from a variety of perspectives. We (my co-editor, Jim Altschuld and I) have assembled (what I think is) a  stellar collection of writers who have good ideas. Editing an issue of NDE (again) was a valuable experience for me: I learned again why I don’t write the definitive text on anything; I learned again how important Accreditation, Certification, and Credentialing are; I am reminded how complicated it is to assemble a list of competencies that adequately capture what is an evaluator; and I am once again humbled, recognizing that cynicism does not come with the territory–it is acquired.

Now, a bit on competencies and why they are important. Read the rest of this entry »

May
08
Filed Under (criteria, program evaluation) by Molly on 08-05-2014

To quote Annie Leonard, the word sustainability “gets thrown around all the time now and it’s not always clear what is intended.” She goes on to talk about the UN World Commission on Environment and Development definition of sustainable development as “…meeting the needs of the current generation without compromising the ability of future generations to meet their own needs.” That is a good definition, I think. Yet it is missing something which (according to Leonard) are equity and justice. Robert Gilman defines sustainability as “…equity over time”. She says (and I agree), quoting the Center for Sustainable Communities, that sustainability “consider(s) the whole instead of the specific. Sustainability emphasizes relationships rather than pieces in isolation.”

Now, given that evaluation to be effective must look at the whole (here is a good example of when the whole is greater than the sum of its parts); and

given that evaluation works to find out information that will benefit both the current and future generations; and

given that evaluation works to determine what difference was made in people’s lives, it seems to me that there is a relationship here that needs to be acknowledged.

A colleague of mine works in youth development and loves the job. My colleague has to determine the value of the program; the program needs to be evaluated. Yet, if the work is only for the program (i.e., the pieces in isolation) not the whole, what good is it that my colleague loves the job? The relationship between the youth involved and the bigger picture is truly more than can probably be captured in any evaluation. Still, the evaluation needs to be planned to consider that, even if the resources are limited (that is the “probably” above).

So yes, evaluation has something to learn from sustainability. Certainly sustainability can learn from evaluation (and economics, and equity, and ecology…).

NEW TOPIC

I’ve been, once again, getting comments about making a difference. I thought I’d post some of those comments (I’ve copied and pasted comments so the spelling is as it appears in the original text):

  • …every blog post makes a difference in a way or in another. You can answer at your questions just seeing how many comments are here, how many people are interested in answering you. I think you are a good person, and everything said by a good person is always a life’s lesson to keep in mind. Thank you for every helpful information, good job!

  • It may be a temporary difference – i.e. limited on the time, but of course that at least for some seconds your writing are touching the life’s of all your readers.

  • Every blog or article makes a difference to those who read it! They might strongly agree or disagree with what the blogger has wrote, making a difference by reafirming there opinion or being outraged that somebody else looks at ideas different to them! Keep writing Molly, you are making people think, which is always good
  • I think the best measure of the effectiveness of a blog are the number of shares it gets, as people that found something useful in it tend to want to share with others.

  • …I have written quite a bit about this topic and challenge that bloggers face and the bottom line is that you really can’t measure the value.  Sure I think asking for responses like you did might help you see a bit of it, but the reality is 99.9% of people will never comment.  As such, we as bloggers have to remember that each pageview is a real person who was on our site and who was impacted by what we wrote!

  •  Blogs are probably the best tool for engaging a customer in todays times.

My question: are blogs engaging readers or are they only outreach, even if the blog is read?

P.S. I also got a lot of comments about my analytics post…for next time.

 

References:

Leonard, A. (2011). Story of Stuff. NY: Free Press.story of stuff (good book–worth the read)

Report of the World Commission on Environment and Development. (1987). NY: UN World Commission on Environment and Development. http://www.un-documents.net/our-common-future.pdf

Gilman, R., Director, Context Institute.

  • He says:  Sustainability is equity over time.  As a value, it refers to giving equal weight in your decisions to the future as well as the present.  You might think of it as extending the Golden Rule through time, so that you do unto future generations (as well as to your present fellow beings) as you would have them do unto you.

Center for Sustainable Communities is quoted in a variety of places: http://sustainablesonoma.org/keyconcepts/sustainability.html; http://isocs-sustainability.wikispaces.com, among others.

  • The entire definition is: Sustainability is part of a trend to…consider the whole instead of the specific. Sustainability emphasizes relationships rather than pieces in isolation…Sustainability is not about regressing to primitive living conditions. It is about understanding our situation, and developing as communities in ways that are equitable, and make sense ecologically and economically.

 

Dec
02
Filed Under (criteria, program evaluation) by Molly on 02-12-2013

I was reminded about the age of this blog (see comment below).  Then it occurred to me:  I’ve been writing this blog since December 2009.  That is 4 years of almost weekly posts.  And even though evaluation is my primary focus, I occasionally get on my soap box and do something different (White Christmas Pie, anyone?).  My other passion besides evaluation is food and cooking.  I gave a Latke party on Saturday and the food was pretty–and it even tasted good.  I was more impressed by the visual appeal of my table; my guests were more impressed by the array of tastes, flavors, and textures.  I’d say the evening was a success.  This blog is a metaphor for that table.  Sometimes I’m impressed with the visual appeal; sometimes I’m impressed with the content.  Today is an anniversary.  Four years.  I find that amazing (visual appeal).  The quote below (a comment offered by a reader on the post “Is this blog making a difference?”, a post I made a long time ago) is about content.

“Judging just from the age of your blog I must speculate that you’ve done something right. If not then I doubt you’d still be writing regularly. Evaluation of your progress is important but pales in comparison to the importance of writing fresh new content on a regular basis. Content that can be found no place else is what makes a blog truly useful and indeed helps it make a difference.”

Audit or evaluation?

I’m an evaluator; I want to know what difference the “program” is making in the lives of the participants.  The local school district where I live, work, and send my children to school has provided middle school children with iPads iPad.  They want to “audit” their use.  I commend the school district for that initiative (both giving the iPads as well wanting to determine the effectiveness).  I wonder if they really want to know what difference the electronics are making in the lives of the students.  I guess I need to go re-read Tom Schwandt’s 1988 book, “Linking Auditing and Metaevaluation”, a book he wrote with Ed Halpern, Tom Schwandt book  as well as see what has happened in the last 25 years (and it is NOT that I do not have anything else to read…smiley).  I think it is important to note the sentence (taken from the forward), “Nontraditional studies are found not only in education, but also in…divers fields …(and the list they provide is a who’s who in social science).  The problem of such studies is “establishing their merit”.  That is always a problem with evaluation–establishing the merit, worth, value of a program (study).

We could spend a lot of time debating the  merit, worth, value of using electronics in the pursuit of learning.  (In fact, Jeffrey Selingo writes about the need to personalize instruction using electronics in his 2013 book “College (Un)bound”college unbound by jeffry selingo–very readable, recommended.)   I do not think counting the number of apps or the number of page views is going to answer the question posed.  I do not think counting the number of iPads returned in working condition will either.  This is an interesting experiment.  How , reader, would you evaluate the merit, worth, value of giving iPads to middle school children?  All ideas are welcome–let me know because I do not have an answer, only an idea.

Nov
21
Filed Under (criteria, program evaluation) by Molly on 21-11-2013

For the first time in my lifetime the first day of Hanukkah is also Thanksgiving.  The pundits are are sagely calling the event Thanksgivukkah.thanksgivukkah image  According to this referenced source, the first day of Hanukkah will not happen again for over 70,000 years.  However, according to another source, this overlap could happen again in 2070 and 2165.  Although I do not think I’ll be around in 2070, my children could be (they are 17 and 20 of this writing).  I find this phenomenon really interesting–Thanksgiving usually starts the US holiday season and Hanukkah falls later, during Advent.  Not so this year.  I wonder how people combine latkes and Thanksgiving (even without the turkey).  Loaded latkes? Thanksgivukkah latkes (My appreciation to Kia.)

So I’m sure you are wondering, HOW EXACTLY DOES THIS RELATE TO EVALUATION?

I decided that it was time to revisit my blog title, Evaluation is an Everyday Activity. Every day you evaluate something.  Although you do not necessarily articulate out loud the criteria against which you are determining merit, worth, and value, you have those criteria.  I have them for latkes AND Thanksgiving.  Our latkes must be crispy; of winter vegetables including potatoes.  This allows me to use a variety of winter vegetables I may have gotten in my CSA.  (Beet latkes? Sweet potato latkes?  Celeriac latkes?  You bet!)   Our Thanksgiving is to have foods for which we are truly thankful.  That allows us to think about gratitude.  Each year our menu is different because each year we are thankful for different things.  (I must confess, however, we always have pie–pumpkin, which I make from home grown pumpkin/squash, and chocolate pecan, which is an original old family recipe.)  One year when we put all the food on the table, all the food was green.  We didn’t plan it that way; it just happened because they were foods for which we were thankful.  This year, we will have mashed potatoes (by the Queen of mashed potatoes), Celebration Filo, both the gluten-free (made with rice wrappers and no onion, garlic, or dairy) and glutened versions (the version which we renamed and is in the link above), and something else that will probably be green.  This year I’m thankful for my gluten-free; dairy-free friend who will join us for Thanksgiving and I’m working up alternatives to accommodate her and still satisfy the rest of us.

So you see, even when I’m thinking about Thanksgiving, latkes, and gratitude, I’m thinking about evaluation.  What merit does the “program” have?  What is its worth?  What is its value?  Those are all evaluative questions that apply to Thanksgiving (and latkes and gratitude). Thanksgiving 2

So you see, Evaluation is an Everyday Activity.

I won’t be blogging next week.  Enjoy.  Be grateful.Thanksgiving