Hello, readers.  This week I’m doing something different with this blog.  This week, and the third week in each month from now on, I’ll be posting a column called Timely Topic.  This will be a post on a topic that someone (that means you reader) has suggested.  A topic that has been buzzing around in conversations.  A topic that has relevance to evaluation.  This all came about because a colleague from another land grant institution is concerned about the dearth of evaluation skills among Extension colleagues.  (Although this comment makes me wonder to whom this colleague is talking, that question is content for another post, another day.)  So thinking about how to get core evaluation information out to more folks, I decided to devote one post a month to TIMELY TOPICS.  To day’s post is about “THINKING CAREFULLY”.

Recently, I’ve been asked to review a statistics text book for my department. This particular book uses a program that is available on everyone’s computer.  The text has some important points to make and today’s post reflects one of those points.  The point is thinking carefully about using statistics.

As an evaluator–if only the evaluator of your own programs–you must think critically about the “…context of the data, the source of the data, the method used in data collection, the conclusions reached, and the practical implications” (Triola, 2010, p. 18).  The author posits that to understand general methods of using sample data; make inferences about populations; understand sampling and surveys; and important measures of key characteristics of data, as well as the use of valid statistical methods, one must recognize the misuse of statistics.

I’m sure all of you have heard the quote, “Figures don’t lie; liars figure,” which is attributed to Mark Twain.  I’ve always heard the quote as “Statistics lie and liars use statistics.”  Statistics CAN lie.  Liars CAN use statistics.  That is where thinking carefully comes in–to determine if the statistical conclusions being presented are seriously flawed.

As evaluators, we have a responsibility (according to the AEA guiding principles) to conduct systematic, data-based inquiry; provide competent performance; display honesty and integrity…of the entire evaluation process; respect the security, dignity, and self-worth of all respondents; and consider the diversity of the general and public interests and values.  This demands that we think carefully about the reporting of data.  Triola cautions, “Do not use voluntary response sample data for making conclusions about a population.”  How often have you used data from individuals who decide themselves (self-selected) whether to participate in your survey or not?  THINK CAREFULLY about your sample.  These data cannot be generalized to all people like your respondents because of the bias that is introduced by self-selection.

Other examples of misuse of statistics include

• using correlation for concluding causation;
• reporting data that involves a sponsors product;
• identifying respondents inappropriately;
• reporting data that is affected with a desired response bias;
• using small samples to draw conclusions for large groups;
• implying that being precise is being accurate; and
• reporting misleading or unclear percentages. (This cartoon was drawn by Ben Shabad.)

When reporting statistics gathered from your evaluation, THINK CAREFULLY.