Mar
25
Filed Under (program evaluation) by Molly on 25-03-2015 and tagged , , ,

Today is the middle of Spring Break at Oregon State University.

What did you do today that involved thinking evaluatively?

Did you decide to go to work?work-clip art

Did you decide to go to the beach?oregon beach

Did you decide you were sick?sick clip art

Did you decide you would work in the yard/garden?garden clip art

Did you decide to stop and smell the roses?smell the roses Read the rest of this entry »

Mar
19
Filed Under (program evaluation) by Molly on 19-03-2015 and tagged ,

How many of you are planning on attending the American Evaluation Association (AEA)AEA logo conference in Chicago this November? AEA just closed its call for proposals on Monday, March 16. Hopefully, you were able to submit prior to the deadline. Notifications of acceptance will be announced in July. It is a lot of work to review those proposals, schedule those proposals, and make sure that there is a balance of topics and presentation types across the week.

I hope anyone (everyone) interested in program evaluation and all the evaluation permutations (of which there are many) will make an effort to attend. I plan to be there.

AEA is my professional home. The first meeting I attended was in 1981 in Austin, Texas. I was a graduate student; several of us drove from Tucson to Austin.(Let me tell you West-TexasWest Texas west-texas-desert is quite an experience; certainly a bucket list opportunity.) That meeting was a combined meeting of the Evaluation Research Society and Evaluation Network. It had about 200 attendees. Quite a difference from meetings experienced in the 21st century. AEA (the name and the organization) became official with the vote of the membership in 1986. Who would have thought that AEA would be the leading evaluation association in the country, possibly in the world? The membership page says that there are members who come from 60 foreign countries. I have met marvelous folks there. I count some of my best friends as AEA members. Certainly the landscape of attendees has changed regularly over the years. For a founding member, that evolution has been interesting to watch. As a board member and as a past-president (among other roles), being part of the organizational change has been exciting. I urge you to attend; I urge you to get involved.

Hope to see you in Chicago in November.

 

If you haven’t taken the my survey, please do. It is found here.

my two cents.

molly.

Mar
12
Filed Under (program evaluation) by Molly on 12-03-2015 and tagged , , , ,

A recent blog (not mine) talked about the client’s evaluation use.evaluation cycle and use The author says that she feels “…successful…if the client is using the data…” This statement allowed me to stop and pause and think about data use. The author continues with the comment about the difference between “…facilitating the client’s understanding of the data in order to create plans and telling the client exactly what the data means and what to do with it.”

I work with Extension professionals who may or may not understand the methodology, the data analysis, or the results. How does one communicate with Extension professionals who may be experts in their content area (cereal crops, nutrition, aging, invasive species) and know little about the survey on which they worked? Is my best guess (not knowing the content area) a good guess? Do Extension professionals really use the evaluation findings?  If I suggest that the findings could say this, or suggest that the findings could say that, am I preventing a learning opportunity from happening? Read the rest of this entry »

This is a link to an editorial in Basic and Applied Social PsychologyBasic and applied social psychology cover. It says that inferential statistics are no longer allowed by authors in the journal.

“What?”, you ask. Does that have anything to do with evaluation? Yes and no. Most of my readers will not publish here. They will publish in evaluation journals (of which there are many) or if they are Extension professionals, they will publish in the Journal of Extension.JoE logo And as far as I know, BASP is the only journal which has established an outright ban on inferential statistics. So evaluation journals and JoE still accept inferential statistics.

Still–if one journal can ban the use, can others?

What exactly does that mean–no inferential statistics? The journal editors define this ban as as “…the null hypothesis significance testing procedure is invalid and thus authors would be not required to perform it.” That means that authors will remove all references to  p-values, t-values, F-values, or any reference to statements about significant difference (or lack thereof) prior to publication. The editors go on to discuss the use of confidence intervals (No) and Bayesian methods (case-by case) and what inferential statistical procedures are required by the journal. Read the rest of this entry »

Evaluators are often the key people identified to conduct a needs assessment. A needs assessment is identified in the situation that exists before the intervention is designed or implemented. Hopefully.  Currently, there is discussion in the field that rather than focusing on needs (i.e., what is missing, needed), there should be discussions of assets (i.e., what is available, strengths). My favorite go-to person on needs assessments is Jim Altschuld altschuld who has published a volume that talks about bridging the gap between the two. Bridging the Gap-altschuld. In it, he talks about the difference between the two. He says, “Need is a noun, a problem that should be attended to or resolved. It is a gap or discrepancy between the ‘what should be’ and the ‘what is’ conditions”. However, assets/capacity building (emphasis added) refer “…to building a culture in an organization or community so that it can grow and change in accord with its strengths…” Read the rest of this entry »

Feb
18
Filed Under (program evaluation) by Molly on 18-02-2015 and tagged , , , ,

social-justice.312132658_stdEarlier this week I attended a meeting of the College of Education (my academic home) Social and Environmental Justice (SJE) Work Group.  This is a loosely organized group of interested faculty and staff, led by an individual who is the ESOL Program Coordinator & Instructor. We had representatives from each of the four program areas (Adult and Higher Education [AHE], Teacher and Counseling Education [TCE], Science, Technology, Engineering, and Math [STEM], and Cultural and Linguistic Diversity [CLD]) in person (AHE, TCE,. CLD) or on paper [STEM]. The intent was to document for the work group, what each program area is doing in the area of social justice. Social Justice is a mandate for the College and OSU. The AHE and the TCE representatives provided us with information. We never did get to the STEM response. Then we got on to a discussion of what exactly is meant by social justice (since AHE has not defined the term specifically). My response was the evaluation response: it depends.

Most of the folks in the group focused on the interface of race and gender. OK. Others focused on the multiple and different voices. OK. Others focused on the advantages and disadvantages experienced. How is that not based in economics? Others focused on power and privileged. (As an accident of birth?) What is social justice exactly? Can you have social justice without environmental justice? How does that fit with the issue of diversity? How does any of this relate to evaluation?

The American Evaluation Association has had in place for a long time (since 1994) a set of five guiding principles (see Background section at the link for a bit of history). The fourth and fifth principles are, respectively, Respect for People and Responsibilities for General and Public Welfare. Respect for people says this:  Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders. Responsibilities for the General and Public Welfare says this: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation. Although both talk about parts of social justice that we talked about earlier this week, is this a complete view? Certainly, security, dignity, and self worth and diversity of interests and values approach the discussion we had. Is there still something missing? I think so. Where is fairness addressed?

To me, fairness is the crux of the issue.  For example, it certainly isn’t fair that in the US, 2% of the population has the cumulative wealth of the remaining 98%. (Then we are into economics.) Although Gandhi said “be the change”social justice 4 is that enough? What if that change isn’t fair?  And the question must be addressed, fair to whom? What if that change is only one person? Is that fair?  I always talk about the long term outcome as world peace (not in my lifetime, though). If you work for justice (for me that is fairness) will peace result? I don’t know. Maybe.social justice 3

 

 

Tomorrow is the Lunar New Year. It is the year of the goat/sheep/ram. I wish you the best. Eat jiaozi and tangerines (for encouraging wealth), and noodles without breaking/biting them (you do want a long life, right?). Happy New Year.

 

 

Chinese-Year-of-the-Goat-2015

 

 

I don’t know what to write today for this week’s post. I turn to my book shelf and randomly choose a book. Alas, I get distracted and don’t remember what I’m about.  Mama said there would be days like this…I’ve got writer’s block (fortunately, it is not contagious).writers-block (Thank you, Calvin). There is also an interesting (to me at least because I learned a new word–thrisis: a crisis of the thirties) blog on this very topic (here).

So this is what I decided rather than trying to refocus. In the past 48 hours I’ve had the following discussions that relate to evaluation and evaluative thinking.

  1. In a faculty meeting yesterday, there was the discussion of student needs which occur during the students’ matriculation in a program of study. Perhaps it should include assets in addition to needs as students often don’t know what they don’t know and cannot identify needs.
  2. A faculty member wanted to validate and establish the reliability for a survey being constructed. Do I review the survey, provide the reference for survey development, OR give a reference for validity and reliability (a measurement text)? Or all of the above.
  3. There appears to be two virtual focus group transcripts for a qualitative evaluation that have gone missing. How much affect will those missing focus groups have on the evaluation? Will notes taken during the sessions be sufficient?
  4. A candidate came to campus for an assistant professor position who presented a research presentation on the right hand (as opposed to the left hand) [Euphemisms for the talk content to protect confidentiality.] Why even study the right hand when the left hand is what is the assessment?
  5. Reading over a professional development proposal dealing with what is, what could be, and what should be. Are the questions being asked really addressing the question of gaps?

Read the rest of this entry »

Feb
04

There has been a somewhat lengthy discussion regarding logic models on EvalTalk,listserv an evaluation listserv sponsored by the American Evaluation AssociationAEA logo. (Check out the listserv archivesEVALTALK Archives.)  This discussion has been called in the subject line, “Logic model for the world?” The discussion started on January 26, 2015. The most telling (at least to me) was a statement that appeared January 30, 2015:

“The problem is not the instrument. All instruments can be mastered as a matter of technique. The problem is that logic models mistake the nature of evaluative knowledge – which is neither linear nor rational.” (Saville Kushner, EvalTalk, January 30, 2015).

The follow-up of this discussion talks about tools, specifically hammers (Bill Fear, EvalTalk, January 30, 2015). Fear says, “Logic is only a tool. It does not exist outside of the construction of the mind.” Read the rest of this entry »

making a difference 5I am an evaluator, charter member of the American Evaluation Association, and former member of the forerunner organization, Evaluation Network. When you push my “on ” button, I can talk evaluation until (a lot of metaphors could be used here); and often do. (I can also talk about other things with equal passion, though not professionally.) When my evaluation button is pushed or, for that matter, most of the time, I wonder what differencemake a difference am I making. In this case, I wonder what difference I am making with this blog.

One of my readers (I have more than I ever imagined) suggested that I develop an “online” survey that I can include regularly in my posts. I thought that was a good idea. I thought I’d go one better and have it be a part of the blog. Then I would tabulate the findings (if there are any :) ). Just so you know, I DO read all the comments; I get at least six daily. I often do not comment on those, however.

So, reader, here is the making a difference survey . This link will (should) take you to Surveymonkey and the survey. Below, I’ve listed the questions that are in the survey.

Check all that apply.

Reading this blog makes a difference to me by:

  1. _____ Giving me a voice to follow
  2. _____ Providing interesting content
  3. _____ Providing content I can use in my work
  4. _____ Providing dependable post
  5. _____ Providing me with information to share
  6. _____ Building my skills in evaluation
  7. _____ Showing me that there are others in the world concerned with similar things
  8. _____ Offering me good reading about an interesting weekly topic
  9. _____ Offering me content of value to me
  10. _____ Other. Please specify in comment

Please complete the survey.

my two cents.

molly.

 

 

Jan
23
Filed Under (program evaluation) by Molly on 23-01-2015 and tagged ,

I was reading Harold Jarche’s blog, Perpetual Beta and he is talking about the interface of the digital and analog worlds; he is talking about connections. connections 2 I find that that concept applies to evaluators. Here’s how.

I was trained as an evaluator in the 1980s; we didn’t have access to the web, internet, email, FB, or many of the other high technology options available today. I did an NDE (wasn’t called that then) volume (Vol. 62) which was all done by hard copy and the USPS (a tedious and laborious process). I just completed another NDE (now called that) volume (Vol. 145) that was done electronically (no longer tedious, though still a laborious process). This last volume was quick. Although most of the authors entered the field after I did, my co-editor didn’t; he entered before I did. These authors had the luxury of electronics that we didn’t have. I have had to learn how to use electronics; I know my co-editor did, too. (I remember saying to myself and my colleagues, what will my secretary [yes, we used that title] do now that I’m composing on a key-board?) Now I do all my composing and other things on a keyboard; most of my work is augmented with electronics (i.e., the digital world). But I am truly a digital immigrant; learning how to use this new technology, to be in this digital world, is hard for and on me. (My children laugh at me and get exasperated; it is so simple to them.)

Today’s evaluators are highly connected, largely due to the electronic capabilitieselectronic connections. Yet there is still evaluator isolation. Makes me wonder if evaluators really connected? Sheila Robinson (the only evaluator in her organization bemoans this fact here). She advocates for connections through EvalTalk and AEA’s LinkedIn account. I still see these as digital, albeit, opportunities to connect. Social media is also mentioned for connections. Still I wonder–are you really connected? With colleagues scattered around the world, this may truly be the only way to stay connected today. Letters and phone calls are truly analog and perhaps passe. Still they are appreciated and sometimes welcome as the only way to connect. What will this world look like if the only way to connect is via digital?

Jarach advocates changing the way we organize. To me that talks about changing the way we work. Maybe evaluators will work in isolation. In order to “see as many possible roads ahead”, perhaps we need to “work in self-managing networks”. networks 2 “If those who are educated, knowledgeable, and experienced do not push for a better world of work, then who will? An effective knowledge network cultivates the diversity and autonomy of each worker. Knowledge networks function best when each person can choose with whom and when they connect. Solving problems together is becoming the real business challenge.” (From Jarach)

And that affects evaluators!

To whom are you connected? How do you connect? Are you caught in-between?

mytwo cents.

molly.