Data collection « Evaluation is an Everyday Activity

Alternative facts.

Never. Never. has evaluation been questioned with the label of “alternative facts.”

Over the years, I have been very aware that evaluation is a political activity.

I have talked about evaluation being political (here, and here, and here, and here ).

But is it? Is it just another way of making the answer be what we want it to be? A form of alternative fact?

I’ve been an evaluator for a long time. I want to make a difference to the people who experience my programs (or the programs for which I’m consulting as an external evaluator). The thought that I might be presenting “alternative facts” is troublesome.

Did I really determine that outcome? Or is the outcome bogus? Liars use statistics, you know. (This is a paraphrase of a quote that Mark Twain attributed to Benjamin Disraeli.)

Big news brings out the fakers. But are evaluation results “big news”? Or…do people not want to hear what is actually happening, what the outcome really is?

Reminds me of 1984 ( George Orwell): War is peace. Freedom is slavery. Ignorance is strength (the English Socialist Party–aka. INGSOC). Kevin Siers added, in his cartoon of Sean Spicer, “2017 is 1984”. Two contradictory ideas existing at the same time as correct.

Statistics.

Statistics is a tool that evaluators use on a regular basis. It allows evaluators to tease apart various aspects of a program. The “who” , the “what”, the “when”, maybe even the “why”. Statistics can certainly help determine if I made a difference But how I see statistics may not be how you see them, interpret them, use them. Two people can look at a set of statistics and say they do not agree. Is that an example of alternative facts?

Bias.

Everyone comes to any program with preconceived bias. You, the evaluator, want to see a difference. Preferably a statistically significant difference, not just a practical significance (although that would be nice as well).

Even if you are dealing with qualitative data, and not with quantitative data yielding statistics, you come to the program with bias. Objectivity is not an option. You wouldn’t be doing the program if you didn’t think that the program will make a difference. Yet, the individuals who have funded the program (or in some other way are the folks who get the final report) can (and do) not accept the report as it is written. That is not what they want to see/hear/read. Does that make the report alternative facts? Or is bias speaking without acknowledging that bias?

Perhaps Kierkegaard is right.

There are only two ways you can be fooled.

my .

molly.

Cartoons

Chris Lysy draws cartoons.

Evaluation and research cartoons.

Logic Model cartoons.

Presentation cartoons.

Data cartoons.

More Cartoons

He has offered an alternative to presenting survey data. He has a wonderful cartoon for this.

Survey results are in. Who's ready to spend the next hour looking at poorly formatted pie charts?

He is a wonderful resource. Use him. You can contact him through his blog, fresh spectrum.

my .

molly.

The OSU Extension Service conference started today (#OSUExtCon). There are concurrent sessions, plenary sessions, workshops, twitter feeds, (Jeff Hino is Tweeting), tours, receptions, and meal gatherings. There are lots of activities and they cover four days. But I want to talk about conference evaluation.

The thought occurs to me: “What difference is this making?” Ever the evaluator, I realize that the selection will be different next year (it was different last year) so I wonder how valuable is it to evaluate the concurrent sessions? Given that time doesn’t stand still (fortunately {or not, depending}), the plenary sessions will also be different. Basically, the conference this year will be different from the conference the next time. Yes, it will be valuable for the presenters to have feedback on what they have done and it will be useful for conference planners to have feed back on various aspects of the conference. I still have to ask, “Did it make a difference?”

A long time colleague of mine (formerly at Pennsylvania State University ), Nancy Ellen Kiernan proposed a method of evaluating conferences that I think is important to keep and use. She suggested the use of “Listening Post” as an evaluation method. She says, “The “Listening Posts” consisted of a group of volunteer conference participants who agreed beforehand to “post” themselves in the meeting rooms, corridors, and break rooms and record what conferees told them about the conference as it unfolded [Not unlike Twitter, but with value; parenthetical added]. Employing listening posts is an informal yet structured way to get feedback at a conference or workshop without making participants use pencil and paper.” She put it in “Tipsheet #5” and published the method in Journal of Extension (JoE), the peer reviewed monthly on-line publication.

Quoting from the abstract of the JoE article, “Extension agents often ask, “Isn’t there an informal but somewhat structured way to get feedback at a conference or workshop without using a survey?” This article describes the use of ‘Listening Posts’ and the author gives a number of practical tips for putting this qualitative strategy to use. Benefits include: quality feedback, high participation and enthusiastic support from conferees and the chance to build program ownership among conference workers. Deficits: could exclude very shy persons or result in information most salient to participants.”

I’ve used this method. It works. It does solicit information about what difference the conference made, not whether the participants liked or didn’t like the conference. (This is often what is asked in the evaluation.) Nancy Ellen suggests that the listening post collectors ask the following questions:

“What did you think of the idea of …this conference? and
What is one idea or suggestion that you found useful for your professional work? (The value/difference question)
Then, she suggests, that the participant tell anything else about the conference that is important for us to know.

Make sure the data collectors are distinctive. Make sure they do not ask any additional questions. The results will be interesting.

I just got back from a road trip across Southern Alabama with my younger daughter. We started from Birmingham and drove a very circuitous route ending in Mobile and the surrounding areas, then returned to Birmingham for her to start her second year at Birmingham-Southern College.

As we traveled, I read a book by Bill McKibben (one of many) called Oil and Honey: The Education of an Unlikely Activist. It is a memoir, a personal recounting of the early years of this decade, which corresponded with the years my older daughter was in college (2011-2014). I met Bill McKibben, who, in 2008, is credited with starting the non-profit, 350.0rg, and is currently listed as “senior adviser and co-founder”. He is a passionate, soft-spoken man, who beleives that the world is on a short fuse. He really seems to believe that there is a better way to have a future. He, like Gandhi, is taking a stand. Oil and Honey puts into action Gandhi’s saying about being the change you want to see. As the subtitle indicates, McKibben is an unlikely activist. He is a self-described non-leader who led and advises the global effort to increase awareness of climate change/chaos. When your belief is on the line, you do what has to be done.

Evaluators are the same way. When your belief is on the line, you do what has to be done. And, hopefully, in the process you are the change that you want to see in the world. But know it cannot happen one pipeline at a time. The fossil fuel industry has too much money. So what do you do? You start a campaign. That is what 350.org has done: “There are currently fossil fuel divestment campaigns at 308 colleges and universities, 105 cities and states, and 6 religious institutions.”(Wikipedia, 350.0rg) (Scroll down to the heading “Fossil Fuel Divestment” to see the complete discussion.) Those are clear numbers, hard data for consumption. (Unfortunately, the divestment campaign at OSU failed.)

So I see the question as one of impact, though not specifically world peace (my ultimate impact). If there is no planet on which to work for world peace, there in no need for world peace. Evaluators can help. They can look at data critically. They can read the numbers. They can gather the words. This may be the best place for the use of pictures (they are, after all, worth 1000 words). Perhaps by combining efforts, the outcome will be an impact that benefits all humanity and builds a tomorrow for the babies born today.

my .

molly.

The use of a survey is a valuable evaluation tool, especially in the world of electronic media. The survey allows individuals to gather data (both qualitative and quantitative) easily and relatively inexpensively. When I want information about surveys, I turn to the 4th edition of the Dillman book (Dillman, Smyth, & Christian, 2014*). Dillman has advocated the “Tailored Design Method” for a long time. (I first became aware of his method, which he called “Total Design Method,” in his 1978 first edition, a thin, 320 page volume [as opposed to the 509 page fourth edition].)

Today I want to talk about the “Tailored Design” method (originally known as total design method).

In the 4th edition, Dillman et al. say that “…in order to minimize total survey error, surveyors have to customize or tailor their survey designs to their particular situations.” They are quick to point out (through various examples) that the same procedures won’t work for all surveys. The “Tailored Design Method” refers to the customizing survey procedures for each separate survey. It is based upon the topic of the survey and the audience being surveyed as well as the resources available and the time-line in use. In his first edition, Dillman indicated that the TDM (Tailored Design Method) would produce a response rate of 75% for mail surveys and an 80%-90% response rate is possible for telephone surveys. Although I cannot easily find the same numbers in the 4th edition, I can provide an example (from the 4th edition on page 21-22) where the response rate is 77% after a combined contact of mail and email over one month time. They used five contacts of both hard and electronic copy.

This is impressive. (Most surveys I and others I work with conduct have a response rate less than 50%.) Dillman et al. indicate that there are three fundamental considerations in using the TDM. They are:

Reducing four sources of survey error–coverage, sampling, nonresponse, and measurement;
Developing a set of survey procedures that interact and work together to encourage all sample members to respond; and
Taking into consideration elements such as survey sponsorship, nature of survey population, and the content of the survey questions.

The use of a social exchange perspective suggests that respondent behavior is motivated by the return that behavior is expected, and usually does, bring. This perspective affects the decisions made regarding coverage and sampling, the way questions are written and questionnaires are constructed, and determines how contacts will produce the intended sample.

If you don’t have a copy of this book (yes, there are other survey books out there) on your desk, get one! It is well worth the cost ($95.00, Wiley; $79.42, Amazon).

* Dillman, D. A., Smyth, J. D. & Christian, L. M. (2014) Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, N. J.: John Wiley & Sons, Inc.

my .

molly.

Like many people, I find change hard. In fact, I really don’t like change. I think this is the result of a high school experience; one-third of my classmates left each year. (I was a military off-spring; we changed assignments every three years.)

Yet, in today’s world change is probably the only constant. Does that make it fun? Not necessarily. Does that make it easy? Nope. Does that make it necessary? Yep.

Evaluators deal with change regularly. New programs are required; those must be evaluated. Old programs are revised; those must be evaluated. New approaches are developed and presented to the field. (When I first became an evaluator, there wasn’t a systems approach to evaluation; there wasn’t developmental evaluation; I could continue.) New technologies are available and must be used even if the old one wasn’t broken (even for those of us who are techno-peasants).

I just finished a major qualitative evaluation that involved real-time virtual focus groups. When I researched this topic (virtual focus groups), I found a lot of information about non-synchronous focus groups, focus groups using a conferencing software, even synchronous focus groups without pictures. I didn’t find anything about using real-time synchronous virtual focus groups. Unfortunately, we didn’t have much money even though there are services available. Continue reading →

About two years ago, I conducted a 17 month hybrid evaluation preparation program for the Western Region Extension Service faculty. There were over 30 individuals involved. I was the evaluation expert; Jim Lindstrom (who was at WSU at the time) was the cheerleader, the encourager, the professional development person. I really couldn’t have done it without him. (Thank you, Jim.) Now, to maximize this program and make it available to others who were not able to participate, I’ve been asked to explore an option for creating an on-line version of the WECT (say west) program. It would be loaded through the OSU professional and continuing education (PACE) venue. To that end, I am calling on those of you who participated in the original program (and any other readers) to provide me with feedback of the following:

What was useful?
What needed to be added?
What could be more in depth?
What could be deleted?
Other comments?

Please be as specific as possible.

I can go to the competency literature (of which there is a lot) and redevelop WECT from those guidelines. (For more information on competencies see: King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22(2), 229-247.) Or I could use the Canadian system as a foundation. (For more information see this link.)

I doubt if I can develop an on-line version that would cover (or do justice) to all those competencies.

So I turn to you my readers. Let me know what you think.

my .

molly.

This is a link to an editorial in Basic and Applied Social Psychology. It says that inferential statistics are no longer allowed by authors in the journal.

“What?”, you ask. Does that have anything to do with evaluation? Yes and no. Most of my readers will not publish here. They will publish in evaluation journals (of which there are many) or if they are Extension professionals, they will publish in the Journal of Extension. And as far as I know, BASP is the only journal which has established an outright ban on inferential statistics. So evaluation journals and JoE still accept inferential statistics.

Still–if one journal can ban the use, can others?

What exactly does that mean–no inferential statistics? The journal editors define this ban as as “…the null hypothesis significance testing procedure is invalid and thus authors would be not required to perform it.” That means that authors will remove all references to p-values, t-values, F-values, or any reference to statements about significant difference (or lack thereof) prior to publication. The editors go on to discuss the use of confidence intervals (No) and Bayesian methods (case-by case) and what inferential statistical procedures are required by the journal. Continue reading →

Evaluators are often the key people identified to conduct a needs assessment. A needs assessment is identified in the situation that exists before the intervention is designed or implemented. Hopefully. Currently, there is discussion in the field that rather than focusing on needs (i.e., what is missing, needed), there should be discussions of assets (i.e., what is available, strengths). My favorite go-to person on needs assessments is Jim Altschuld who has published a volume that talks about bridging the gap between the two. . In it, he talks about the difference between the two. He says, “Need is a noun, a problem that should be attended to or resolved. It is a gap or discrepancy between the ‘what should be’ and the ‘what is’ conditions”. However, assets/capacity building (emphasis added) refer “…to building a culture in an organization or community so that it can grow and change in accord with its strengths…” Continue reading →

I’ve talked about bias bias 2 before (cognitive bias; personal and situational bias); I’ve probably talked about bias in surveys and sampling. Today I want to talk specifically about self report bias…you know, the bias that exists when people answer questions themselves (as opposed to having their behavior be observed).

First, what is self-report bias (often called self-response bias)? It is the bias that exists when people answer survey questions by themselves. Everyone has this bias; it is unavoidable. It can be seen as social desirability bias (what the the respondent thinks the survey writer wants to hear); self-selection bias (a person decides to respond when invited as opposed to not responding); and what I’m going to call a “clarity bias” (whether the respondent understands the survey content).

I’m finding more and more that the S of good writing are applicable to all writing–fiction, non-fiction, scholarly, SURVEY. If the survey isn’t clear, the respondent isn’t going to be able to answer in a way that is meaningful. If the respondent cannot answer the survey in a way that is meaningful, there will be no meaningful data. If data are not meaningful, then the evaluation will not be able to tell you the value or merit or worth of the project being evaluated.

It is important to

Pilot test the survey before sending it out to the target audience.
Have naive readers read over the survey (different from pilot testing).
Only ask one thing at a time in the questions.

I’m sure there are other things that would help minimize bias–let me know other options used.

Bottom line: Self report bias is always part of evaluation that involves people; it can be minimized.

New topic.

This is the time of year that one thinks about changes and how one will do that in the new year. Yet, those changes often fall by the way side, getting left in the dust (so to speak) of every day life. One way I’ve kept those changes fresh is to follow how the new year presents itself. There is the calendar new year new_year_2015 (on January 1); there is the lunar new year 2015-Chinese-New-Year-Free-Design (this year on February 19, the year of the goat); there is the spring equinox (norooz, the Persian new year); Rosh Hashana (Jewish new year beginning on the evening of September 13); there is the Islamic new year, the Thai new year, the Ethiopian new year, and the list goes on. (What is your favorite new year and new year’s celebration?) By refreshing the year regularly, I can keep my “resolutions” alive all year. My wish for you is a prosperous and healthy new year. Welcome 2015.