I am an evaluator, charter member of the American Evaluation Association, and former member of the forerunner organization, Evaluation Network. When you push my “on ” button, I can talk evaluation until (a lot of metaphors could be used here); and often do. (I can also talk about other things with equal passion, though not professionally.) When my evaluation button is pushed or, for that matter, most of the time, I wonder what difference am I making. In this case, I wonder what difference I am making with this blog.
One of my readers (I have more than I ever imagined) suggested that I develop an “online” survey that I can include regularly in my posts. I thought that was a good idea. I thought I’d go one better and have it be a part of the blog. Then I would tabulate the findings (if there are any ). Just so you know, I DO read all the comments; I get at least six daily. I often do not comment on those, however.
So, reader, here is the making a difference survey . This link will (should) take you to Surveymonkey and the survey. Below, I’ve listed the questions that are in the survey.
Check all that apply.
Reading this blog makes a difference to me by:
Please complete the survey.
I’ve talked about bias before (cognitive bias; personal and situational bias); I’ve probably talked about bias in surveys and sampling. Today I want to talk specifically about self report bias…you know, the bias that exists when people answer questions themselves (as opposed to having their behavior be observed).
First, what is self-report bias (often called self-response bias)? It is the bias that exists when people answer survey questions by themselves. Everyone has this bias; it is unavoidable. It can be seen as social desirability bias (what the the respondent thinks the survey writer wants to hear); self-selection bias (a person decides to respond when invited as opposed to not responding); and what I’m going to call a “clarity bias” (whether the respondent understands the survey content).
I’m finding more and more that the S of good writing are applicable to all writing–fiction, non-fiction, scholarly, SURVEY. If the survey isn’t clear, the respondent isn’t going to be able to answer in a way that is meaningful. If the respondent cannot answer the survey in a way that is meaningful, there will be no meaningful data. If data are not meaningful, then the evaluation will not be able to tell you the value or merit or worth of the project being evaluated.
It is important to
I’m sure there are other things that would help minimize bias–let me know other options used.
Bottom line: Self report bias is always part of evaluation that involves people; it can be minimized.
This is the time of year that one thinks about changes and how one will do that in the new year. Yet, those changes often fall by the way side, getting left in the dust (so to speak) of every day life. One way I’ve kept those changes fresh is to follow how the new year presents itself. There is the calendar new year (on January 1); there is the lunar new year (this year on February 19, the year of the goat); there is the spring equinox (norooz, the Persian new year); Rosh Hashana (Jewish new year beginning on the evening of September 13); there is the Islamic new year, the Thai new year, the Ethiopian new year, and the list goes on. (What is your favorite new year and new year’s celebration?) By refreshing the year regularly, I can keep my “resolutions” alive all year. My wish for you is a prosperous and healthy new year. Welcome 2015.
The world tilted yesterday at 3:03pm PST. Today is the first day of winter. The dawn (yes, actually the dawn–the return of the light/sun/longer days) was glorious. For those of us who celebrate such things, the easiest way for me to measure the solstice is if there is sun on the first day of winter. There was.
I think this year is especially wonderful (the end of year is usually so–this year is especially so)–Hanukkah is still happening (tonight is the seventh night), Solstice occurred yesterday, and Christmas will happen Thursday.
All that light. I can almost forget that it is cold, dark, and rainy here in Oregon. Almost. For the first day of winter, it is remarkably dry (for Oregon in winter) and warm (55° degrees). This bodes well. Last year by this time we had had an enduring snow; the schools were closed for a week (we would have another snow [and the requisite snow days] in February). Although I would be hard pressed to say it is balmy, 55° is hardly cold (sure colder than Florida at 73°). My daughters are rejoicing in the warmth.
This is such a magical time of year. What makes it magical is an evaluative question. Everyone will answer this question differently. For me, it is having my daughters home–their presence is my present. It will carry me through the new year. Read the rest of this entry »
This will be short.
I showed a revised version of Alkin’s Evaluation Theory Tree in last week’s post. It had leaves. It looked like this:
It was taken from the second edition of Alkin’s book.
I have had two comments about this tree.
Personal and situational bias are forms of cognitive bias and we all have cognitive bias.
When I did my dissertation on personal and situational biases, I was talking about cognitive bias (only I didn’t know it, then).
Then, I hypothesized that previous research experience (naive or sophisticated) and the effects of exposure to expected project outcomes (positive, mixed, negative) would affect the participant and make a difference in how the participant would code data. (It did.) The Sadler article which talked about intuitive data processing was the basis for this inquiry. Now many years later, I am encountering cognitive bias again. Sadler says that “…some biases can be traced to a particular background knowledge…”(or possibly–I think–lack of knowledge), “…prior experience, emotional makeup or world view”. (This, I think, falls under the category of, according to Tversky and Kahneman, human judgements and it will differ from rational choice theory (often given that label). Read the rest of this entry »
A uniquely American holiday (although it is celebrated in other countries as well-Canada, Liberia, The Netherlands, Norfolk Islands),
For me it is an opportunity to to be grateful–and I am, more than words can express. I am especially grateful for my daughters, bright, articulate, and caring children (who are also adults). Read the rest of this entry »
Today is Veteran’s Day in the United States.
It is the day celebrated as a federal holiday by libraries, post offices, school districts; not the university. It originated as Armistice Day in celebration of the end of World War I, the war to end all wars, the Great War.
It wasn’t made a national holiday (celebrated by those institutions above) until 1938. The name was changed from Armistice Day to Veterans Day in 1954 after the Korean War to remember all veterans, not just those from WWI.
Yet these women and men often give the ultimate sacrifice and are often not recognized for their service. Metrics do not capture the value, merit, or worth of their service, yet it is usually metrics that is the focus of any evaluation done.
(This cartoon is the segue to the next US holiday.)
It has been about /years since I started this blog (more or less–my anniversary is actually in early December) .
Because I am an evaluator, I have asked several time is this blog making a difference. And those posts, the ones in which I ask “is this blog making a difference”, are the ones which get the most comments. Now, truly, most comments are often either about marketing some product, inviting me to view another blog, mirroring comments made previously, or comments in a language which I cannot read (even with an online translator). Yet, there must be something about “making a difference” that engages viewers and then engages them to make a comment.
Today, I read a comment Read the rest of this entry »
For years my criteria for a “good” conference was the following
I think this year’s conference was a success (despite the difficulty in identifying who was doing what when because the management corporation minimized the program in an attempt to be ecological, if excluding). If I were to ask my daughters to rate the conference on a scale of one (1) to 10 (ten), one being not “good”, 10 being “good”, I think they would have said an 8 – 8.5. (They have their own following of friends and their own interests.)
I saw and talked to three long time friends, although I missed those who have chosen not to attend AEA any more (I must be getting old) and those with whom I didn’t spend time.
I met more than three people I didn’t know before and I must say, if they are any indication (and I think they are) of the evolution of the association, the association is in good hands (even though I miss the intimacy I “grew up with”). Read the rest of this entry »
Recently, I drafted a paper about a capacity building; I’ll be presenting it at the 2014 AEA conference. The example on which I was reporting was regional and voluntary; it took a dedication, a commitment from participants. During the drafting of that paper, I had think about the parts of the program; what would be necessary for individuals who were interested in evaluation and had did not have a degree. I went back to the competencies listed in the AJE article (March 2005) that I cited in a previous post. I found it interesting to see that the choices I made (after consulting with evaluation colleagues) were listed in the competencies identified by Stevahn et al., yet they list so much more. So the question occurs to me is: To be competent, to build institutional evaluation capacity are all those needed? Read the rest of this entry »