Mar
03
Filed Under (Methodology, program evaluation) by Molly on 03-03-2015

This is a link to an editorial in Basic and Applied Social PsychologyBasic and applied social psychology cover. It says that inferential statistics are no longer allowed by authors in the journal.

“What?”, you ask. Does that have anything to do with evaluation? Yes and no. Most of my readers will not publish here. They will publish in evaluation journals (of which there are many) or if they are Extension professionals, they will publish in the Journal of Extension.JoE logo And as far as I know, BASP is the only journal which has established an outright ban on inferential statistics. So evaluation journals and JoE still accept inferential statistics.

Still–if one journal can ban the use, can others?

What exactly does that mean–no inferential statistics? The journal editors define this ban as as “…the null hypothesis significance testing procedure is invalid and thus authors would be not required to perform it.” That means that authors will remove all references to  p-values, t-values, F-values, or any reference to statements about significant difference (or lack thereof) prior to publication. The editors go on to discuss the use of confidence intervals (No) and Bayesian methods (case-by case) and what inferential statistical procedures are required by the journal. Read the rest of this entry »

Feb
18
Filed Under (program evaluation) by Molly on 18-02-2015

social-justice.312132658_stdEarlier this week I attended a meeting of the College of Education (my academic home) Social and Environmental Justice (SJE) Work Group.  This is a loosely organized group of interested faculty and staff, led by an individual who is the ESOL Program Coordinator & Instructor. We had representatives from each of the four program areas (Adult and Higher Education [AHE], Teacher and Counseling Education [TCE], Science, Technology, Engineering, and Math [STEM], and Cultural and Linguistic Diversity [CLD]) in person (AHE, TCE,. CLD) or on paper [STEM]. The intent was to document for the work group, what each program area is doing in the area of social justice. Social Justice is a mandate for the College and OSU. The AHE and the TCE representatives provided us with information. We never did get to the STEM response. Then we got on to a discussion of what exactly is meant by social justice (since AHE has not defined the term specifically). My response was the evaluation response: it depends.

Most of the folks in the group focused on the interface of race and gender. OK. Others focused on the multiple and different voices. OK. Others focused on the advantages and disadvantages experienced. How is that not based in economics? Others focused on power and privileged. (As an accident of birth?) What is social justice exactly? Can you have social justice without environmental justice? How does that fit with the issue of diversity? How does any of this relate to evaluation?

The American Evaluation Association has had in place for a long time (since 1994) a set of five guiding principles (see Background section at the link for a bit of history). The fourth and fifth principles are, respectively, Respect for People and Responsibilities for General and Public Welfare. Respect for people says this:  Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. Responsibilities for the General and Public Welfare says this: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation. Although both talk about parts of social justice that we talked about earlier this week, is this a complete view? Certainly, security, dignity, and self worth and diversity of interests and values approach the discussion we had. Is there still something missing? I think so. Where is fairness addressed?

To me, fairness is the crux of the issue.  For example, it certainly isn’t fair that in the US, 2% of the population has the cumulative wealth of the remaining 98%. (Then we are into economics.) Although Gandhi said “be the change”social justice 4 is that enough? What if that change isn’t fair?  And the question must be addressed, fair to whom? What if that change is only one person? Is that fair?  I always talk about the long term outcome as world peace (not in my lifetime, though). If you work for justice (for me that is fairness) will peace result? I don’t know. Maybe.social justice 3

 

 

Tomorrow is the Lunar New Year. It is the year of the goat/sheep/ram. I wish you the best. Eat jiaozi and tangerines (for encouraging wealth), and noodles without breaking/biting them (you do want a long life, right?). Happy New Year.

 

 

Chinese-Year-of-the-Goat-2015

 

 

Feb
11
Filed Under (program evaluation) by Molly on 11-02-2015

I don’t know what to write today for this week’s post. I turn to my book shelf and randomly choose a book. Alas, I get distracted and don’t remember what I’m about.  Mama said there would be days like this…I’ve got writer’s block (fortunately, it is not contagious).writers-block (Thank you, Calvin). There is also an interesting (to me at least because I learned a new word–thrisis: a crisis of the thirties) blog on this very topic (here).

So this is what I decided rather than trying to refocus. In the past 48 hours I’ve had the following discussions that relate to evaluation and evaluative thinking.

  1. In a faculty meeting yesterday, there was the discussion of student needs which occur during the students’ matriculation in a program of study. Perhaps it should include assets in addition to needs as students often don’t know what they don’t know and cannot identify needs.
  2. A faculty member wanted to validate and establish the reliability for a survey being constructed. Do I review the survey, provide the reference for survey development, OR give a reference for validity and reliability (a measurement text)? Or all of the above.
  3. There appears to be two virtual focus group transcripts for a qualitative evaluation that have gone missing. How much affect will those missing focus groups have on the evaluation? Will notes taken during the sessions be sufficient?
  4. A candidate came to campus for an assistant professor position who presented a research presentation on the right hand (as opposed to the left hand) [Euphemisms for the talk content to protect confidentiality.] Why even study the right hand when the left hand is what is the assessment?
  5. Reading over a professional development proposal dealing with what is, what could be, and what should be. Are the questions being asked really addressing the question of gaps?

Read the rest of this entry »

Feb
04
Filed Under (program evaluation) by Molly on 04-02-2015

There has been a somewhat lengthy discussion regarding logic models on EvalTalk,listserv an evaluation listserv sponsored by the American Evaluation AssociationAEA logo. (Check out the listserv archivesEVALTALK Archives.)  This discussion has been called in the subject line, “Logic model for the world?” The discussion started on January 26, 2015. The most telling (at least to me) was a statement that appeared January 30, 2015:

“The problem is not the instrument. All instruments can be mastered as a matter of technique. The problem is that logic models mistake the nature of evaluative knowledge – which is neither linear nor rational.” (Saville Kushner, EvalTalk, January 30, 2015).

The follow-up of this discussion talks about tools, specifically hammers (Bill Fear, EvalTalk, January 30, 2015). Fear says, “Logic is only a tool. It does not exist outside of the construction of the mind.” Read the rest of this entry »

Jan
28
Filed Under (program evaluation) by Molly on 28-01-2015

making a difference 5I am an evaluator, charter member of the American Evaluation Association, and former member of the forerunner organization, Evaluation Network. When you push my “on ” button, I can talk evaluation until (a lot of metaphors could be used here); and often do. (I can also talk about other things with equal passion, though not professionally.) When my evaluation button is pushed or, for that matter, most of the time, I wonder what differencemake a difference am I making. In this case, I wonder what difference I am making with this blog.

One of my readers (I have more than I ever imagined) suggested that I develop an “online” survey that I can include regularly in my posts. I thought that was a good idea. I thought I’d go one better and have it be a part of the blog. Then I would tabulate the findings (if there are any :) ). Just so you know, I DO read all the comments; I get at least six daily. I often do not comment on those, however.

So, reader, here is the making a difference survey . This link will (should) take you to Surveymonkey and the survey. Below, I’ve listed the questions that are in the survey.

Check all that apply.

Reading this blog makes a difference to me by:

  1. _____ Giving me a voice to follow
  2. _____ Providing interesting content
  3. _____ Providing content I can use in my work
  4. _____ Providing dependable post
  5. _____ Providing me with information to share
  6. _____ Building my skills in evaluation
  7. _____ Showing me that there are others in the world concerned with similar things
  8. _____ Offering me good reading about an interesting weekly topic
  9. _____ Offering me content of value to me
  10. _____ Other. Please specify in comment

Please complete the survey.

my two cents.

molly.

 

 

Dec
29
Filed Under (Methodology, program evaluation) by Molly on 29-12-2014

I’ve talked about bias bias 2before (cognitive bias; personal and situational bias); I’ve probably talked about bias in surveys and sampling. Today I want to talk specifically about self report bias…you know, the bias that exists when people answer questions themselves (as opposed to having their behavior be observed).

First, what is self-report bias (often called self-response bias)? It is the bias that exists when people answer survey questions by themselves. Everyone has this bias; it is unavoidable. It can be seen as social desirability bias (what the the respondent thinks the survey writer wants to hear); self-selection bias (a person decides to respond when invited as opposed to not responding); and what I’m going to call a “clarity bias” (whether the respondent understands the survey content).

I’m finding more and more that the five the number 2 5Cs-5-CorrectnessS of good writing are applicable to all writing–fiction, non-fiction, scholarly, SURVEY. If the survey isn’t clear, the respondent isn’t going to be able to answer in a way that is meaningful. If the respondent cannot answer the survey in a way that is meaningful, there will be no meaningful data. If data are not meaningful, then the evaluation will not be able to tell you the value or merit or worth of the project being evaluated.

It is important to

  1. Pilot test the survey before sending it out to the target audience.
  2. Have naive readers read over the survey (different from pilot testing).
  3. Only ask one thing at a time in the questions.

I’m sure there are other things that would help minimize bias–let me know other options used.

Bottom line: Self report bias is always part of evaluation that involves people; it can be minimized.

New topic.

This is the time of year that one thinks about changes and how one will do that in the new year. Yet, those changes often fall by the way side, getting left in the dust (so to speak) of every day life. One way I’ve kept those changes fresh is to follow how the new year presents itself. There is the calendar new yearnew_year_2015 (on January 1); there is the lunar new year 2015-Chinese-New-Year-Free-Design(this year on February 19, the year of the goat); there is the spring equinox spring equinox (norooz, the Persian new year); Rosh Hashanarosh_hashanah (Jewish new year beginning on the evening of September 13); there is the Islamic new yearislamic-new-year-1024x768, the Thai new yearThai new year, the Ethiopian new yearethiopian new year, and the list goes on.  (What is your favorite new year and new year’s celebration?) By refreshing the year regularly, I can keep my “resolutions” alive all year. My wish for you is a prosperous and healthy new year. Welcome 2015.

 

mytwo cents.

molly.

Dec
22
Filed Under (program evaluation) by Molly on 22-12-2014

The world tilted yesterday at 3:03pm PST. Today is the first day of winter. The dawn (yes, actually the dawn–the return of the light/sun/longer days) was glorious. For those of us who celebrate such things, the easiest way for me to measure the solstice is if there is sun on the first day of winter. There was.

I think this year is especially wonderful (the end of year is usually so–this year is especially so)–Hanukkahhanukkah lights 2 is still happening (tonight is the seventh night), Solsticesolstice bonfire occurred yesterday, and Christmaschristmas 3 will happen Thursday.

All that light. I can almost forget that it is cold, dark, and rainy here in Oregon. Almost. For the first day of winter, it is remarkably dry (for Oregon in winter) and warm (55° degrees). This bodes well. Last year by this time we had had an enduring snow; the schools were closed for a week (we would have another snow [and the requisite snow days] in February). Although I would be hard pressed to say it is balmy, 55° is hardly cold (sure colder than Florida at 73°). My daughters are rejoicing in the warmth.

This is such a magical time of year. What makes it magical is an evaluative question. Everyone will answer this question differently. For me, it is having my daughterschristmas 2015-2 home–their presence is my present. It will carry me through the new year. Read the rest of this entry »

This will be short.

I showed a revised version of Alkin’s Evaluation Theory Tree in last week’s post. It had leaves. It looked like this:Evaluation theory tree edition 2

It was taken from the second edition of Alkin’s book.

I have had two comments about this tree.

  1. There are few women represented in the tree. (True, especially in the draft version; in version above there are more.)
  2. I was reminded about the  Fred  Carden and Marvin C. Alkin’s article in the Journal of Multidisciplinary Evaluation, 8(17), January 2012. (There are still more leaves and the global south is represented.)

Read the rest of this entry »

Personal and situational bias are forms of cognitive bias and we all have cognitive bias.

When I did my dissertation on personal and situational biases, I was talking about cognitive bias (only I didn’t know it, then).

According to Wikipedia, the term cognitive bias was introduced in 1972 (I defended my dissertation in 1983) by two psychologists Daniel Kahneman  and Amos Tversky kahneman-tversky1.

Then, I hypothesized that previous research experience (naive or sophisticated)  and the effects of exposure to expected project outcomes (positive, mixed, negative) would affect the participant and make a difference in how the participant would code data. (It did.)  The Sadler article which talked about intuitive data processing was the basis for this inquiry. Now many years later, I am encountering cognitive bias again. Sadler says that “…some biases can be traced to a particular background knowledge…”(or possibly–I think–lack of knowledge), “…prior experience, emotional makeup or world view”. bias 4 (This, I think, falls under the category of, according to Tversky and Kahneman, human judgements and it will differ from rational choice theory (often given that label). Read the rest of this entry »

Nov
25
Filed Under (criteria, program evaluation) by Molly on 25-11-2014

Thanksgiving

Thanksgiving.

A uniquely American holiday (although it is celebrated in other countries as well-Canada, Liberia, The Netherlands, Norfolk Islands),

filled with too much food (pie any one?) apple pie,

too much football (what is your favorite rivalry?),football rivalry

and too much shopping (black Friday?).black-friday 3

 

For me it is an opportunity to to be grateful–and I am, more than words can express. I am especially grateful for my daughters, bright, articulate, and caring children (who are also adults). Read the rest of this entry »