Celebrate! Celebrate! Dance to your music!
The holidays are here:
This is a wonderful formula for Happiness!
Happy Holidays!
my 
.
molly.
Author Archives: Molly
Conference evaluation
The OSU Extension Service
conference started today (#OSUExtCon). There are concurrent sessions, plenary sessions, workshops, twitter feeds, (Jeff Hino
is Tweeting), tours, receptions, and meal gatherings. There are lots of activities and they cover four days. But I want to talk about conference evaluation.
The thought occurs to me: “What difference is this making?” Ever the evaluator, I realize that the selection will be different next year (it was different last year) so I wonder how valuable is it to evaluate the concurrent sessions? Given that time doesn’t stand still (fortunately {or not, depending}), the plenary sessions will also be different. Basically, the conference this year will be different from the conference the next time. Yes, it will be valuable for the presenters to have feedback on what they have done and it will be useful for conference planners to have feed back on various aspects of the conference. I still have to ask, “Did it make a difference?”
A long time colleague of mine (formerly at Pennsylvania State University 
), Nancy Ellen Kiernan
proposed a method of evaluating conferences that I think is important to keep and use. She suggested the use of “Listening Post” as an evaluation method. She says, “The “Listening Posts” consisted of a group of volunteer conference participants who agreed beforehand to “post” themselves in the meeting rooms, corridors, and break rooms and record what conferees told them about the conference as it unfolded [Not unlike Twitter, but with value; parenthetical added]. Employing listening posts is an informal yet structured way to get feedback at a conference or workshop without making participants use pencil and paper.” She put it in “Tipsheet #5” and published the method in Journal of Extension (JoE), the peer reviewed monthly on-line publication.
Quoting from the abstract of the JoE article, “Extension agents often ask, “Isn’t there an informal but somewhat structured way to get feedback at a conference or workshop without using a survey?” This article describes the use of ‘Listening Posts’ and the author gives a number of practical tips for putting this qualitative strategy to use. Benefits include: quality feedback, high participation and enthusiastic support from conferees and the chance to build program ownership among conference workers. Deficits: could exclude very shy persons or result in information most salient to participants.”
I’ve used this method. It works. It does solicit information about what difference the conference made, not whether the participants liked or didn’t like the conference. (This is often what is asked in the evaluation.) Nancy Ellen suggests that the listening post collectors ask the following questions:
- “What did you think of the idea of …this conference? and
- What is one idea or suggestion that you found useful for your professional work? (The value/difference question)
- Then, she suggests, that the participant tell anything else about the conference that is important for us to know.
Make sure the data collectors are distinctive. Make sure they do not ask any additional questions. The results will be interesting.
Thankfulness (evaluation)
The US just celebrated Thanksgiving, the annual day of thankfulness. 
Canada celebrated in mid October (October 12). Although other countries celebrate versions of the holiday, originally the US and Canada celebrated in honor of the previous harvest.
Certainly, the Guiding Principles 
and the Program Evaluation Standards 
provide evaluators with a framework to conduct evaluation
work. The work for which I am thankful.
AEA and Evaluation 2015
I got back to the office Monday after spending last week in Chicago at the AEA annual conference, Evaluation 2015.
Next year AEA will be in Atlanta, October 24-29, 2016. 
Mark your calendars!
I am tired. I take a breath (many breaths), try to catch up (I don’t), and continue to read my email (hundreds of email). I’m sure there are some I will miss–I always do. In the meantime, I process what I experienced. And pass the conference through my criteria for a successful conference: Did I
- See three (and visit with) long time friends: yes.
- Get three new ideas: maybe.
- Meet three new people I’d like to add to my “friendlies” category: maybe.
Why three. Seemed like a good number; more than one (not representative) and less than five (too hard to remember). Continue reading
November 2015 AEA Conference
November 9, 2015. Chicago. (they are everywhere…everywhere).
AEA 
will be holding the annual Evaluation conference there starting November 9, 2015. The theme this year put forward by President Stewart Donaldson 
is . And as always, much thought and preparation has gone into planning this conference. Will you be there? I will. Continue reading
Context and learning
I was reading a blog post by Harold Jarche 
who stated that “Donald Taylor notes that, ‘everyone has a memory that is particularly attuned to learning some things very easily’. In his post, Donald says that the context in which learn something, as well as how it is presented and received, are all important aspects of whether we will remember something.”
Jennifer Greene 
, a long-time colleague currently at the University of Illinois Urbana-Champaign, addresses context when she says, “We all know that the contexts in which our evaluands* take place are inextricably intertwined with the program as envisioned, implemented, experienced, and judged. And regarding this program context, Saville Kushner 
has profoundly challenged us to ask not, “how well are participants doing in the program?” but rather “how well does the program serve, respect, and respond to these participants’ needs, hopes, and dreams in this place?” Continue reading
Cognitive bias and evaluation
My friend and colleague, Patricia Rogers, says of cognitive bias , “It would be good to think through these in terms of systematic evaluation approaches and the extent to which they address these.” This was in response to the article here. 
The article says that the human brain is capable of 10 to the 16th power (a big number) processes per second. Despite being faster than a speeding bullet, etc., the human brain has ” annoying glitches (that) cause us to make questionable decisions and reach erroneous conclusions.”
Bias is something that evaluators deal with all the time. There is desired response bias, non-response bias, recency and immediacy bias, measurement bias, and…need I say more? Isn’t evaluation and aren’t evaluators supposed to be “objective”? That we as evaluators behave in an ethical manner? That we have dealt with potential bias and conflicts of interest. That is where cognitive bias appear. And you might not know it at all. Continue reading
KASA
KASA. You’ve heard the term many times. Have you really stopped to think about what it means? What evaluation approach you will use if you want to determine a difference in KASA? What analyses you will use? How you will report the findings?
Probably not. You just know that you need to measure KNOWLEDGE, ATTITUDE, SKILLS, and ASPIRATIONS.
The Encyclopedia of Evaluation (edited by Sandra Mathison
) says that they influence the adoption of selected practices and technologies (i.e., programs). Claude Bennett 
uses KASA in his TOP model 
.I’m sure there are other sources. Continue reading
Four steps…impact
First, let me say that getting to world peace will not happen in my lifetime (sigh…) and world peace is the ultimate impact. Everything else is an outcome. It may be a long term outcome, that is a condition change (either social, economic, environmental, or civic), or not. Just because the powers that be use a term doesn’t mean the term is being used correctly!
Then let me say that evaluation is the way to know you got to that impact…ultimately, world peace. Ultimately. In the mean time, you will need to find approximate (proxy) measures.
Last week, I attended the Engagement Scholarship Consortium
conference in State College, PA, home of Penn State. I had the good fortune to see long time friends, meet new people and get a few new ideas. One of the long time friends I was able to visit with was Nancy Franz, Professor Emeritus, Iowa State University. She did a session called “Four steps to measuring and articulating engagement impact”.
Basically she reduced into four steps (hence, the title) program evaluation. And since engagement scholarship is a “program” it needs to be evaluated to make sure it is making a difference. Folks are slowly coming to that idea if the attendance at her session is any indication (full). She used different words than I would have used; I found myself adding parenthetical comments to her words.
I want to share in words what she shared graphically:
- In order to be able to conduct these four steps, you need evaluation training, evaluation support, and successful models;
- STEP 1: You need to map the intended program (my parenthetical was the “logic model” for which she provided the UWEX web site);
- STEP 2: You need to determine what “impact” will be measured (input vs. outcome);
- STEP 3: You need to collect and analyze data (qualitative and quantitative);
- STEP 4: You need to tell the story (when, what, so what, now what; the public value);
- If you do these four steps she believes that you will enhance paid and volunteer staff performance; increase program quality; and improve impact reporting (be persuasive).
She had a few good suggestions; specifically:
- Since most people don’t like to analyze data (because they do not know how?), she holds a data party to look at what was found; and
- Case studies have value; use them.
- I added, “If you aren’t going to use the data, do not collect it. It only obfuscates the impact.”
Think about what you do when you evaluate a program. Do you do these four steps? Do you know what impact you are trying to achieve? And if you can’t get to world peace, that’s OK. Each step will bring you closer.
my .
molly.
It depends
I’ve been stuck.
I haven’t blogged for three weeks. I haven’t blogged because I don’t have a topic. Oh, I’ve plenty to say (I am never for a loss of words… 🙂 ) I want something to relate to evaluation. Relate clearly. Without question. Evaluation.
So after 5 years, I’m going to start over. Evaluation is an everyday activity!
Evaluative thinking is something you do everyday; probably all day. (I don’t know about when you are a sleep, so I said probably.) I think evaluative thinking is one of those skills that everyone needs to learn systematically. I think everyone learns at least a part of evaluative thinking as they grow; the learning may not be systematic. I would put that skill in the same category as critical (not negative but thoughtful) thinking, team building, leadership, communication skills (both verbal and written), technological facility as well as some others which escape me right now. I would add systematic evaluative thinking.
Everyone has criteria on which decisions are based. Look at how you choose a package of cookies 
or a can of corn 
at the grocery store. What criteria do you use for choosing? Yet that wasn’t taught to you; it was just something you developed. Evaluative thinking is more than just choosing what you want for dinner. AARP lists problem solving as part of the critical thinking skills. I think it is more than just problem solving; I do agree that it is a critical thinking skill (see graphic, from Grant Tilus, Rasmussen College).
So you think thoughtfully about most events/activities/things that you do throughout the day. And you learn over time what works and what doesn’t; what has value and what doesn’t. You learn to discern the conditions under which something works; you learn what changes the composition of the outcome. You begin to think evaluatively about most things. One day you realize that you are critically thinking about what you can, will, and need to do. Evaluative thinking has become systematic. You realize that it depends on many factors. You realize that evaluative thinking is a part of who you are. You are an evaluator, even if you are a psychologist or geologist or engineer or educator first.
my .
molly.