Apr
24
Filed Under (criteria, program evaluation) by Molly on 24-04-2012

CAVEAT:  This may be too political for some readers.

Sometimes, there are ideas that appear in other blogs that may or may not be directly related to my work in evaluation.  Because I read them, I see evaluative relations and think they are important enough to pass along.  Today is one of those days.  I’ll try to connect the dots  between what I read and share here and evaluation.  (For those of you who are interested in the Connect the Dots, a major event day on climate change and weather on May 5, 2012, go here.)

First, Valerie Williams, AEA365 blog, April18, 2012 says, “…Many environmental education programs struggle with the question of whether environmental education is a means to an end (e.g. increased stewardship) or an end itself. This question has profound implications for how programs are evaluated, and specifically the measures used to determine program success.”

I think that many educational programs (whether environmentally focused or not) struggle with this question.  Is the program a means to an end or the end itself?  I am reminded of programs which are instituted for cost savings and then the program designers want that program evaluated.  Means or end?

Williams also offers comments about evaluability assessment–that evaluation task that helps evaluators decide whether to evaluate a new programs, especially if that new program’s readiness for evaluation is in question. (She provides resources if you are interested.)  She offers reasons for conducting an evaluability assessment.  Specifically:

  • Surfacing disagreements among stakeholders about the program theory, design and/or structure;
  • Highlighting the need for changes in program design; and
  • Clarifying the type of evaluation most helpful to the program.

Evauability assessment is a topic for future discussion.

Second, a colleague offered the following CDC reference and says, “The purpose of this workbook is to help public health program managers, administrators, and evaluators develop an effective evaluation plan in the context of the planning process. It is intended to assist in developing an evaluation plan but is not intended to serve as a complete resource on how to implement program evaluation.”  I offer it here because I know that evaluation plans are often added after the program has been implemented.  Although it has as a focus pubic health programs, one source familiar with this work commented that there is enough in the workbook that can be applied to a variety of settings.  Check it out; the link is below

 

Next, Nigerian novelist Chimamanda Ngozi Adichie is quoted as saying, “The single story creates stereotypes, and the problem with stereotypes is not that they are untrue, but that they are incomplete. They make one story become the only story.”

Given that

  • Extension uses story to evaluate a lot of programs; and
  • Story is used to convince legislators of Extension’s value; and
  • Story, if done right, is a powerful tool;

Then it behooves us all to remember this–are we using the story because it captures the effect or because it is the only story?  If only story, is it promoting a stereotype?  Adichie, though a novelist, may be an evaluator at heart.

Finally, there is this quote, also from an AEA365 blog (Steve Mayer) “There are elements of Justice and Injustice everywhere – in society, in reform efforts, and in the evaluation of reform efforts. The choice of outcomes to be assessed is a political act. “Noticing progress” probably takes us further than “measuring impact,” always being mindful of who benefits.”

We often are stuck on “measuring impact”; after all, isn’t that what everyone wants to know.  If world peace is the ultimate impact, then what is the likelihood of measuring that?  I think that “noticing progress” (i.e., change) will take us much further because of the justice it can capture (or not–and that is telling).  And by capturing “noticing progress”, we can make explicit who benefits.

This runs long today.

 

Apr
20
Filed Under (program evaluation) by Molly on 20-04-2012

I wonder (as y’all know) if anyone reads this; if the blog makes a difference; and should I keep writing (because blogging is hard work).

Over the last two weeks, I’ve received over 50 comments about my posts, from folks who are not subscribed and who read the post.  I don’t know if their search engine has optimized my blog so it pops up or if they are really interested in evaluation.  Some of the comments appear genuine; some seem specious at best.  Please know I read them all.  And I appreciate the feedback.  There were some questions posted in the comments.  Here are some answers, not in any particular order.

  1. AEA365 is a blog sponsored by the American Evaluation Association.  It invited known evaluators who blog (like me) to contribute a post to their AEA365.  Susan Kistler is AEA’s executive director; she has really good ideas.  I wouldn’t be surprised if this was one of them.
  2. To be an evaluator who blogs, you first need to be an evaluator.  You get to be an evaluator by studying evaluation.  There are numerous places to do that–universities, Evaluator’s Institute, AEA’s summer institute, on the job training.  I went to university; I got a Ph.D in program evaluation.  Most people who come to evaluation come through some social science–sociology, psychology, social work, anthropology, other disciplines.  If you want to know more, I’ll be happy to elaborate in a future blog.
  3. When I preview my post, the graphics look fine.  I don’t have to click on them more than once; they just are there.  My IT person says it might be the browser being used.  I use Firefox; I am a PC user.  I don’t know how this looks on a Mac.
  4. Although I try to stay off my political soap box when I post, there are times where the topic (Viktor Frankl, for example) is both political and evaluative.  For those of you new to evaluation, evaluation is a political discipline.  I have a few passions in my life that I return to again and again as they have been with me for a long time (some as long as 50 years).  Evaluatiion is one of those passions even though I’ve been a professional evaluator for only 30 years. (I’ve probably been a lay evaluator for as long as I’ve known my passions.)
  5. 500 words seems to be a good length.

I’m working with my IT person to make this blog better.  Since I’m a technopeasant, learning something new is hard work for me.  Next week I’ll talk about evaluation again.  I promise.  Hopefully, y’all will see the sun where you are. Here in Oregon, we are eager for the sun.   Even if it is the sunset in Florida.

Apr
13
Filed Under (Uncategorized) by Molly on 13-04-2012

Today I’m reporting the results of the survey I ran for two weeks.

I asked five questions:

  1. Is this blog making a difference in what you do?  (answer yes or no)
  2. In which of the following ways is the blog making a difference.  (choose all that apply)
  3. What is your opinion about the length?  (approximately 500 words–too long, too short, just right)
  4. How often do you read the blog?  (weekly; regularly, depending on topic; rarely; archive only; never)
  5. What topic would you like to see discussed (various with an option for other)

I don’t know how many subscribe (I am a technopeasant, after all) and that I blog at all is close to miraculous, so the results I report may or may not be reflective of what is actually happening.

So what are the results?

1.  Of those 22 people responding, 21 people (100% of those responding; one person skipped this question) said that the blog is making a difference in what they do.

2.  Of those 22 people responding, 15 (68.2%) said that they get new ideas; 15 (68.2%) said that they get new perspectives; 8 (36.4%) said that they get old information clarified;  13 (59.1%) said that they learn new information; and 11 (50.0%) said they review previously learned information.  No one responded that the blog has not made a difference. [Phew…:) ] [Keep in mind that percentages will not add to 100% because multiple responses could be selected.)

3.  Everyone who responded (N=22; 100%) said that 500 words is just right in length.

4.  When respondents were asked how often the blog was read by them, 4 (18.2%) said weekly, as it is posted; 17 (77.3%) said regularly, depending on topic; 1 (4.5%) said rarely [although they obviously read it to respond…:) ].

5.  When asked which topic they would like to see addressed in future blogs, 16 (72.7%) said methodology; 9 (40.9%) said quantitative data analysis; 12 (54.5%) said qualitative data analysis; 10 (45.5%) said data collection methods; 15 (68.2%) said survey development; 8 (36.4%) said program evaluation theory; and 8 (36.4%) said program evaluation models.  Three people (13.6%) offered comments.  [Keep in mind that percentages will NOT add to 100% because multiple responses could be selected.]

Comments were about the graphics (hard to read); could be eliminated.  I’ll talk to my tech people about that.  On my reading the graphics are clear.  May be the browser; may be something else, says the technopeasant.  The other comment was about getting new ideas even though the ideas have not been implemented yet.

 

So what does this tell me–given the small sample size, I am cautiously optimistic.   (If I find out  how many are subscribed, I’ll let y’all know.)   I will continue to blog.  I will figure out other ways to determine if I’m making a difference.  And thanks for all of you who took the time to answer the survey and all of you who take the time to read my musings.  Of the several things about which I am passionate, evaluation is close to the top.  (oh, and no graphics this time…)

 

 

Apr
05
Filed Under (Data Analysis) by Molly on 05-04-2012

I had a conversation (ok–an electronic conversation) with colleagues a few weeks ago.  The conversation was about the use of inappropriate analyses for manuscripts being submitted.  The specific question raised was about the t-test and went something like this:

Should a t-test be used on a sample that has not been RANDOMLY drawn from a population.  If the sample is not randomly drawn, or if the entire population is used (as is often the case with Extension evaluations), then a t-test is NOT appropriate.

What exactly is the appropriate test?  First, one has to identify if the underlying assumptions have been met; if not, a parametric, i.e., ttest, is not appropriate.

So what exactly are the assumptions underlying the use of a t-test?

Glass and Stanley (1970, pg. 297) [a classic statistics text in the field of education and psychology and there may be a more recent edition than the one I have on my shelf] say that for dependent samples (the kind most often used by Extension professionals ), the sample:

  1. is normally distributed;
  2. has homogeneous variances (i.e., same spread); and
  3. is RANDOMLY drawn from a population.

Some would add that the scale of measurement needs to result in interval or ratio data (not nominal–ordinal is a questionable case) and have a sample size of 30 or over.

This presents a quandary, to say the least.  Extension professionals know that journals want a probability value (if the study is quantitative) and how do you get a probability value with out a t-test?

Answer:  Use a nonparametric equivalent test.

The nonparametric equivalent for a test of dependent means is a Wilcoxon matched pair test.

Marascuilo and McSweeney (1977, pg 5) say that that the researcher needs to select “…a test for which the power of rejection is maximized when the hypothesis is tested false.”  They go on to say, “If the data adhere to the assumptions required for a classical, normally based t or F test, a researcher would be foolish…not (to) use them, since they are optimum, when justified.”  Justification is the key here.  And the justification is: Do the data meet the assumptions for the test?

Their bottom line is that a researcher should never think that a nonparametric test is exclusively a substitute for a parametric test.  It is not.  Use the right test for the hypothesis being tested.  It may be (probably is) a nonparametric test.

 

Citations

Glass, G. V., & Stanley, J. C. (1970). Statistical Methods in Education and Psychology.  Englewood Cliffs, NJ: Prentice-Hall, Inc.

Marascuilo, L. A. & McSweeney, M. (1977).  Nonparametric and Distribution-free Methods for the Social Sciences. Monterey, CA: Brooks/Cole Publishing.

 

 

Apr
04
Filed Under (program evaluation) by Molly on 04-04-2012

Before Spring break, I blogged about making a difference.  I realize that many who subscribe to this blog were on break last week when the post came.  So I’m sending an extra post this week:  PLEASE COMPLETE THE SURVEY that was posted through an imbedded hyperlink in the post two weeks ago.  I plan to close the survey on Friday, COB.  The URL if the link above doesn’t work is http://www.surveymonkey.com/s/ZD33HFS.  You can copy and paste the URL into your browser.  PLEASE…:)