Tag Archives: data management

Alternative facts?

Posted on by

Alternative facts.

Never. Never. has evaluation been questioned with the label of “alternative facts.”

Over the years, I have been very aware that evaluation is a political activity.

I have talked about evaluation being political (here, and here, and here, and here ).

But is it? Is it just another way of making the answer be what we want it to be? A form of alternative fact?

I’ve been an evaluator for a long time. I want to make a difference to the people who experience my programs (or the programs for which I’m consulting as an external evaluator). The thought that I might be presenting “alternative facts” is troublesome.

Did I really determine that outcome? Or is the outcome bogus? Liars use statistics, you know. (This is a paraphrase of a quote that Mark Twain attributed to Benjamin Disraeli.)

Big news brings out the fakers. But are evaluation results “big news”? Or…do people not want to hear what is actually happening, what the outcome really is?

Reminds me of 1984 ( George Orwell): War is peace. Freedom is slavery. Ignorance is strength (the English Socialist Party–aka. INGSOC). Kevin Siers added, in his cartoon of Sean Spicer,  “2017 is 1984”.  Two contradictory ideas existing at the same time as correct.

Statistics.

Statistics is a tool that evaluators use on a regular basis. It allows evaluators to tease apart various aspects of a program. The “who” , the “what”, the “when”, maybe even the “why”. Statistics can certainly help determine if I made a difference  But how I see statistics may not be how you see them, interpret them, use them. Two people can look at a set of statistics and say they do not agree. Is that an example of alternative facts?

Bias.

Everyone comes to any program with preconceived bias. You, the evaluator, want to see a difference. Preferably a statistically significant difference, not just a practical significance (although that would be nice as well).

Even if you are dealing with qualitative data, and not with quantitative data yielding statistics, you come to the program with bias. Objectivity is not an option. You wouldn’t be doing the program if you didn’t think that the program will make a difference. Yet, the individuals who have funded the program (or in some other way are the folks who get the final report) can (and do) not accept the report as it is written. That is not what they want to see/hear/read. Does that make the report alternative facts? Or is bias speaking without acknowledging that bias?

Perhaps Kierkegaard is right.

There are only two ways you can be fooled.

 

my .

molly.

Cartoons, data analysis, and infographics

Posted on by

Cartoons

 

Chris Lysy chris lysydraws cartoons.

Evaluation  and research cartoons.http://blogs.oregonstate.edu/programevaluation/files/2014/06/evaluation-and-project-working.jpg

 

http://blogs.oregonstate.edu/programevaluation/files/2014/06/research-v.-evaluation.jpg  http://blogs.oregonstate.edu/programevaluation/files/2014/06/I-have-evidence-cartoon.png

Logic Model cartoons.   http://i2.wp.com/freshspectrum.com/wp-content/uploads/2014/03/Too-complex-for-logic-and-evidence.jpg

Presentation cartoons.BS cartoon from fresh spectrum

 

Data cartoons.  http://i0.wp.com/freshspectrum.com/wp-content/uploads/2013/09/wpid-Photo-Sep-27-2013-152-PM1.jpg

More Cartoons

He has offered an alternative to presenting survey data. He has a wonderful cartoon for this.

Survey results are in. Who's ready to spend the next hour looking at poorly formatted pie charts?

He is a wonderful resource. Use him. You can contact him through his blog, fresh spectrum.

my  two cents   .

molly.

 

 

Inferential statistics

Posted on by

This is a link to an editorial in Basic and Applied Social PsychologyBasic and applied social psychology cover. It says that inferential statistics are no longer allowed by authors in the journal.

“What?”, you ask. Does that have anything to do with evaluation? Yes and no. Most of my readers will not publish here. They will publish in evaluation journals (of which there are many) or if they are Extension professionals, they will publish in the Journal of Extension.JoE logo And as far as I know, BASP is the only journal which has established an outright ban on inferential statistics. So evaluation journals and JoE still accept inferential statistics.

Still–if one journal can ban the use, can others?

What exactly does that mean–no inferential statistics? The journal editors define this ban as as “…the null hypothesis significance testing procedure is invalid and thus authors would be not required to perform it.” That means that authors will remove all references to  p-values, t-values, F-values, or any reference to statements about significant difference (or lack thereof) prior to publication. The editors go on to discuss the use of confidence intervals (No) and Bayesian methods (case-by case) and what inferential statistical procedures are required by the journal. Continue reading →

Capacity building and competencies

Posted on by

Recently, I drafted a paper about a capacity building; I’ll be presenting it at the 2014 AEA conference. The example on which I was reporting was regional and voluntary; it took a dedication, a commitment from participants. During the drafting of that paper, I had think about the parts of the program; what would be necessary for individuals who were interested in evaluation and had did not have a degree. I went back to the competencies listed in the AJE article (March 2005) that I cited in a previous post. I found it interesting to see that the choices I made (after consulting with evaluation colleagues) were listed in the competencies identified by Stevahn et al., yet they list so much more. So the question occurs to me is: To be competent, to build institutional evaluation capacity are all those needed? Continue reading →

Evaluation and the electronic age

Posted on by

Within the last 24 hours I’ve had two experiences that remind me of how tenuous our connection is to others.

  1. Yesterday, I was at the library to return several books and pick up a hold. As I went to check out using the digitally connected self-check out station, I got an “out of service” message. Not thinking much of it, as I had received that message before, I moved to another machine. And got the same message! So I went to the main desk. There was a person in front of me; she was taking a lot of time. Turns out it wasn’t her; it was the internet (or intranet, don’t know which). There was no connection! After several minutes, a paper system was implemented and I was assured that the book would be listed by this evening. That the library had a back up system impressed me; I’ve often wondered what would happen if the electricity went out for a long periods of time since the card catalogs are no longer available.
  2. Also, yesterday, I received a phone call on my office land line (!), which is a rare occurrence these days. On the other end was a long time friend and colleague. We are working feverishly on finishing a NDE volume. We have an August 22 deadline and I will be out of town taking my youngest daughter to college. Family trumps everything. He was calling because the gardeners at his condo had cut the cable to his internet, television, and most importantly, his wi-fi. He couldn’t Skype me (our usual form of communication)! He didn’t expect resumption of service until the next day (August 20 at 9:47am PT he went back on line–he lives in the Eastern Time Zone). Continue reading →

Adaptive action–evaluation by another name?

Posted on by

What? So what? Now what?

Sounds like an evaluation problem.

King and Stevahn (in press) tells us the first query requires thoughtful observation of a situation; the second query a discussion of possible options and implications of those options, and the third query calls for the creation of a list of potential next steps.adaptive_action.wiki

Yet these are the key words for “adaptive action” (If you haven’t looked at the web site, I suggest you do.) One quote that is reflective of adaptive action is, “Adaptive Action reveals how we can be proactive in managing today and influencing tomorrow.”( David W. Jamieson, University of St. Thomas). Adaptive action can help you

  • Understand the sources of uncertainty in your chaotic world
  • Explore opportunities for action and their implications as they occur
  • Learn a simple process that cuts through complexity
  • Transform the work of individuals, teams, organizations and communities
  • Take on any challenge—as large as a strategic plan or small as a messy meeting
  • Take action to improve productivity, collaboration and sustainability

Evaluation is a proactive (usually) activity (oh, I know that sometimes evaluation is flying by the seat of your pantsflying-by-the-seat-of-your-pants-Laurence-Musgrove-with-credit-line  and is totally reactive). People are now recognizing that evaluation will benefit them, their programs, and their organizations and that it isn’t personal (although that fear is still out there).

Although the site is directed towards leadership in organizations, the key questions are evaluative. You can’t determine “what” without evidence (data); you can’t determine “so what” unless you have a plan (logic model), and you can’t think about “now what” unless you have an outcome that you can move toward. These questions are evaluative in contemporary times because there are no simple problems any more. (Panarchy approaches similar situations using a similar model  adaptive-cycle Action.) Complex situations are facing program people and evaluators all the time. Using adaptive action may help. Panarchy may help (the book is called Panarchy by Gunderson and Hollings panarchy .)

Just think of adaptive action as another model of evaluation.

mytwo cents

molly.

Summer Reading

Posted on by

Summer reading 2 Many of you have numerous lists for summer reading (NY Times, NPR, Goodreads, Amazon, others…). My question is what are you reading to further your knowledge about evaluation? Perhaps you are; perhaps you’re not. So I’m going to give you one more list 🙂 …yes, it is evaluative.

If you want something light:  Regression to the Mean by Ernest R. House.house--regression to the mean It is a novel. It is about evaluation. It explains what evaluators do from a political perspective.

If you want something qualitative:  Qualitative Data Analysis by Matthew B. Miles, A. Michael Huberman, and Johnny Saldana.Qualitative data analysis ed. 3 It is the new 3rd edition which Sage (the publisher) commissioned. A good thing, too, as both Miles and Huberman are no longer able to do a revision. My new go-to book.

If you want something on needs assessment: Bridging the Gap Between Asset/Capacity Building and Needs Assessment by James W. Altschuld. Bridging the Gap-altschuld Most needs assessments start with what is lacking (i.e., needed); this proposes that an assessment start with what is present (assets) and build  from there, and in the process, meeting needs.

If you want something on higher education:  College (Un)bound by Jeff Selingo.college unbound by jeffry selingo  The state of higher education and some viable alternatives by a contributing editor at the Chronicle of Higher Education. Yes, it is evaluative.

Most of these I’ve mentioned before. I’ve read the above. I recommend them.

Continue reading →

30 is the rule of thumb…

Posted on by

In a recent post, I said that 30 was the rule of thumb, i.e., 30 cases was the minimum needed in a group to be able to run inferential statistics and get meaningful results.  How do I know, a colleague asked? (Specifically,  “Would you say more about how it takes approximately 30 cases to get meaningful results, or a good place to find out more about that?”) When I was in graduate school, a classmate (who was into theoretical mathematics) showed me the mathematical formula for this rule of thumb. Of course I don’t remember the formula, only the result. So I went looking for the explanation. I found this site. Although my classmate did go into the details of the chi-square distribution and the formula computations, this article doesn’t do that. It even provides an Excel Demo for calculating sample size and verifying this rule of thumb. I am so relieved that there is another source besides my memory.

 

New Topic:

Continue reading →

Survey data

Posted on by

Had a comment a while back on analyzing survey data…hmm…that is a quandary as most surveys are done on line (see Survey monkey, among others).

If you want to reach a large audience (because your population from which you sampled is large), you will probably use an on-line survey. The on-line survey companies will tabulate the data for you. Can’t guarantee that the tabulations you get will be what you want, or will tell you want you want to know. Typically (in my experience), you can get an Excel file which can be imported into a soft ware program and you can run your own analyses, separate from the on line analyses. Continue reading →

There are 11 winter holidays at least…

Posted on by

Did you know that there are at least 11 winter holidays besides Christmas–many of them related to light or the return of light.

One needs evaluation tools to determine the merit or worth, to evaluate the holiday’s value to you.  For me, any that return lightsolstice light are important.  So for me, there is Hanukkah menorah (and eight candles), Solstice solstice bonfire (and bonfires and yule logs), Christmas advent wreath(and Advent wreaths with five candles), Kwanzaa kinara( and kinara seven candles).  Sometimes Diwali Diwali falls late in November to be included (it is the ancient Hindu festival of lights that is a movable feast like Hanukkah).

I have celebrations for Hanukkah  (I have several menorahs), for Solstice  (I have two special candelabra solstice candelabra that holds 12 candles–a mini-bonfire to be sure), for Advent/Christmas (I make a wreath each year), and for Kwanzaa  (a handmade Kinara).  And foods for each celebration as well.  Because I live in a multicultural household, it is important that everyone understand that no holiday is more important than any other–all talk about returning light (literal or figurative).  Sometimes the holidays over lap–Hanukkah, Solstice, Christmas all in the same week…phew, I’m exhausted just thinking about it.  Sometimes it seems hard to keep them separate–then I realized that returning the light is not separate; it is light returning.  It is an evaluative task.

So well come the new born sun/son…the light returns.  Evaluation continues.

Happy Holidays…all of them!

I’m taking two weeks holiday–will see you in the new year.