Dec
29
Filed Under (Methodology, program evaluation) by Molly on 29-12-2014

I’ve talked about bias bias 2before (cognitive bias; personal and situational bias); I’ve probably talked about bias in surveys and sampling. Today I want to talk specifically about self report bias…you know, the bias that exists when people answer questions themselves (as opposed to having their behavior be observed).

First, what is self-report bias (often called self-response bias)? It is the bias that exists when people answer survey questions by themselves. Everyone has this bias; it is unavoidable. It can be seen as social desirability bias (what the the respondent thinks the survey writer wants to hear); self-selection bias (a person decides to respond when invited as opposed to not responding); and what I’m going to call a “clarity bias” (whether the respondent understands the survey content).

I’m finding more and more that the five the number 2 5Cs-5-CorrectnessS of good writing are applicable to all writing–fiction, non-fiction, scholarly, SURVEY. If the survey isn’t clear, the respondent isn’t going to be able to answer in a way that is meaningful. If the respondent cannot answer the survey in a way that is meaningful, there will be no meaningful data. If data are not meaningful, then the evaluation will not be able to tell you the value or merit or worth of the project being evaluated.

It is important to

  1. Pilot test the survey before sending it out to the target audience.
  2. Have naive readers read over the survey (different from pilot testing).
  3. Only ask one thing at a time in the questions.

I’m sure there are other things that would help minimize bias–let me know other options used.

Bottom line: Self report bias is always part of evaluation that involves people; it can be minimized.

New topic.

This is the time of year that one thinks about changes and how one will do that in the new year. Yet, those changes often fall by the way side, getting left in the dust (so to speak) of every day life. One way I’ve kept those changes fresh is to follow how the new year presents itself. There is the calendar new yearnew_year_2015 (on January 1); there is the lunar new year 2015-Chinese-New-Year-Free-Design(this year on February 19, the year of the goat); there is the spring equinox spring equinox (norooz, the Persian new year); Rosh Hashanarosh_hashanah (Jewish new year beginning on the evening of September 13); there is the Islamic new yearislamic-new-year-1024x768, the Thai new yearThai new year, the Ethiopian new yearethiopian new year, and the list goes on.  (What is your favorite new year and new year’s celebration?) By refreshing the year regularly, I can keep my “resolutions” alive all year. My wish for you is a prosperous and healthy new year. Welcome 2015.

 

mytwo cents.

molly.

Dec
22
Filed Under (program evaluation) by Molly on 22-12-2014

The world tilted yesterday at 3:03pm PST. Today is the first day of winter. The dawn (yes, actually the dawn–the return of the light/sun/longer days) was glorious. For those of us who celebrate such things, the easiest way for me to measure the solstice is if there is sun on the first day of winter. There was.

I think this year is especially wonderful (the end of year is usually so–this year is especially so)–Hanukkahhanukkah lights 2 is still happening (tonight is the seventh night), Solsticesolstice bonfire occurred yesterday, and Christmaschristmas 3 will happen Thursday.

All that light. I can almost forget that it is cold, dark, and rainy here in Oregon. Almost. For the first day of winter, it is remarkably dry (for Oregon in winter) and warm (55° degrees). This bodes well. Last year by this time we had had an enduring snow; the schools were closed for a week (we would have another snow [and the requisite snow days] in February). Although I would be hard pressed to say it is balmy, 55° is hardly cold (sure colder than Florida at 73°). My daughters are rejoicing in the warmth.

This is such a magical time of year. What makes it magical is an evaluative question. Everyone will answer this question differently. For me, it is having my daughterschristmas 2015-2 home–their presence is my present. It will carry me through the new year. Read the rest of this entry »

This will be short.

I showed a revised version of Alkin’s Evaluation Theory Tree in last week’s post. It had leaves. It looked like this:Evaluation theory tree edition 2

It was taken from the second edition of Alkin’s book.

I have had two comments about this tree.

  1. There are few women represented in the tree. (True, especially in the draft version; in version above there are more.)
  2. I was reminded about the  Fred  Carden and Marvin C. Alkin’s article in the Journal of Multidisciplinary Evaluation, 8(17), January 2012. (There are still more leaves and the global south is represented.)

Read the rest of this entry »

Dec
09
Filed Under (Methodology, program evaluation) by Molly on 09-12-2014

Recently, I got a copy of Marvin Alkin’s book, Evaluation Roots (his first edition; eventually, I will get the second edition).

In Chapter Two, he and Tina Christie talk about an evaluation theory tree and presents this idea graphically (all be it in draft form).

Think of your typical tree with three strong branches (no leaves) and two roots. Using this metaphor, the authors explain the development of evaluation theory as it appears in western (read global north) societies.evaluation theory tree

As you can see, the roots are “accountability and control” (positivist paradigm?) and social inquiry (post-positivist paradigm?). Read the rest of this entry »

Personal and situational bias are forms of cognitive bias and we all have cognitive bias.

When I did my dissertation on personal and situational biases, I was talking about cognitive bias (only I didn’t know it, then).

According to Wikipedia, the term cognitive bias was introduced in 1972 (I defended my dissertation in 1983) by two psychologists Daniel Kahneman  and Amos Tversky kahneman-tversky1.

Then, I hypothesized that previous research experience (naive or sophisticated)  and the effects of exposure to expected project outcomes (positive, mixed, negative) would affect the participant and make a difference in how the participant would code data. (It did.)  The Sadler article which talked about intuitive data processing was the basis for this inquiry. Now many years later, I am encountering cognitive bias again. Sadler says that “…some biases can be traced to a particular background knowledge…”(or possibly–I think–lack of knowledge), “…prior experience, emotional makeup or world view”. bias 4 (This, I think, falls under the category of, according to Tversky and Kahneman, human judgements and it will differ from rational choice theory (often given that label). Read the rest of this entry »