We all have an audience (unless we are a reclusive hermit). The issue is what kind an audience.
Reminds me of the saying, “You can please some of the people, some of the time; all of the people some of the time and not all of the people all of the time.”
So when I teach evaluation (or try to teach evaluation–evaluation is a scary concept), I begin by saying that we are all evaluators, that we evaluate daily. It is truly an everyday activity.
Some people engage at that point. Others are skeptical. Others are deniers.
Like the old saying–some of the people…
Unconsciously (maybe consciously) you identify criteria that will help you make a decision.
That decision will provide you with a decision tree, to be able to make decisions throughout the day.
The other day, I had the opportunity to explore a “teacher-made” test; to determine if it had face validity. The audience was similar to and different from the audience who will ultimately take the test (survey, actually). I needed to capture the concerns about the instrument; to determine if it did what it was supposed to do; and wording issues.
Then a decision needed to be made about its use. We were using the TOP model (Bennett and Rockwell, 1995*). Specifically the KASA portion of the model. This model can be used for program planning/development AND program evaluation/performance. Just one example of a logic model.
There were concerns that the “knowledge” wasn’t specific enough; that the words used were not clear; that “skills” were not specific enough; that not all response options were included in the response set. Some KASA questions could be easily fixed (the response set); some were not (the specificity issue). I sent the compiled responses to the people in charge of the project. I did not make a decision on the specificity. Perhaps if I knew more about the topic, I could have. Obviously, the survey needed changing.
Is all evaluation about making decisions? It all depends.
Evaluation is determining the merit, value, worth of a program or project.
If you can answer that, you have captured your audience.
*Bennett, C. & Rockwell, K. (1995, December). Targeting outcomes of programs (TOP): An integrated approach to planning and evaluation. Unpublished manuscript. Lincoln, NE: University of Nebraska.
It is the day when Saint Valentine (officially Saint Valentine of Terni), a widely recognized third-century Roman saint, has his feast day. Since the High Middle Ages it is associated with a tradition of courtly love. It is said that Valentine’s day was established to counteract the pagan celebration of Lupercalia. There is much we do not know about St. Valentine.
I want to talk about a different kind of love (and I do not mean the various definitions of that word). I want to talk about your calling; your passion.
A good friend of mine said: Know what your calling is, your vocation, and follow it faithfully.
She also said in that same missive: “When you are most disgruntled, take a moment of conscious breath or five moments of conscious play!”
This is the love I’m talking about. The love for your calling; your vocation (passion).
And what to do when you feel disgruntled (breathe/play).
Susan Kistler, AEA Executive Director Emeritus, shares perhaps an important message about love:
“Success is made manifest in health and happiness, confidence that you are loved and the capacity to love with others.”
That is passion.
How does that relate to evaluation?
We are all evaluators and live and work by criteria, whether they are implicit or explicit. Our passions are found in the criteria. We continue that passion for long in our lives–some of us because of family responsibilities; some of us because it is fun. When we get tired, we stop. We still have the passion and that passion comes out when we least expect it. Because once an evaluator (whether formally or not), always an evaluator.
So celebrate your passion tomorrow. And remember to breath…or play!
Trustworthiness. An interesting topic.
Today is November 9, 2016. An auspicious day, to be sure. (No, I’m not going to rant about November 8, 2016; just post this and move on with my living.) Keep in mind trustworthiness, I remind myself.
I had the interesting opportunity to review a paper recently that talked about trustworthiness. This caused me much thought as I was troubled by what was written. I decided to go to my source on “Naturalistic Inquiry” . Given that the paper used a qualitative design, employed a case study method, and talked about trustworthiness, I wanted to find out more. This book was written by two of my long time evaluation guides, Yvonna Lincoln and Egon Guba. (Lincoln’s name may be familiar to you from the Sage Handbook of Qualitative Research which she co-edited with Norman Denzin.)
On page 218, they talk about trustworthiness. About the conventional criteria for trustworthiness (internal validity, external validity, reliability, and objectivity). They talk about the questions underlying those criteria (see page 218).
They talk about how the criteria formulated by conventional inquirers are not appropriate for naturalistic inquiry. Guba (1981a) offers four new terms as they have “…a better fit with naturalistic epistemology.” These four terms and the terms they propose to replace are: Read the rest of this entry »
(Thank you for this thought, Plexus Institute.)
No, I’m not talking about the current political situation in the US. I’m talking about evaluation.
Art Markman makes this comment (the “how do we make decisions…” comment) here. He says “If you dislike every choice you’ve got, you’ll look for one to reject rather than one to prefer—subtle difference, big consequences.” He based this opinion on research, saying that the rejection mind-set allows us to focus on negative information about options and fixate on the one with the smallest downside. Read the rest of this entry »
Oxford English Dictionary defines possible as capable of being (may/can exist, be done, or happen). It defines probable as worthy of acceptance, believable.
Somebody asked me what was the difference between science fiction and fantasy. Certainly the simple approach is that science fiction deals with the possible (if you can think it, it can happen). Fantasy deals with monsters, fairies, goblins, and other mythical creatures, i.e., majic and majical creatures.
(Disclaimer: I personally believe in majic; much of fantasy deals with magic.) I love the Arthurian legend (it could be fantasy; it has endured for so long it is believable). It is full of majic. I especially like the Marion Zimmer Bradley book, The Mists of Avalon . (I find the feminist perspective refreshing.)
Is fantasy always impossible as Bradbury suggests, or is it just improbable? (Do the rules of physics apply?) This takes me back to Bradbury’s quote and evaluation after the minor digression. Bradbury also says that “Science fiction, again, is the history of ideas, and they’re always ideas that work themselves out and become real and happen in the world.” Not unlike evaluation. Evaluation works itself out and becomes real and happens. Usually.
Often, I am invited to be the evaluator of record after the program has started. I sigh. Then I have a lot of work to do. I must teach folks that evaluation is not an “add on” activity. I must also teach the folks how to identify the difference the program made. Then there is the issue of outputs (activities, participants) vs. outcomes (learning, behavior, conditions). Many principal investigators want to count differences pre-post.
Does the “how many” provide a picture of what difference the program made? If you start with no or few participants and you end with many participants, have you made a difference? Yes, it is possible to count. Counts often meet reporting requirements. They are possible. So is documenting the change in knowledge, behavior, and conditions. It takes more work and more money. It is possible. Will you get to world peace? Probably not. Even if you can think it. World peace may be probable; it may not be possible (at least in my lifetime).
This quote is often attributed to Eleanor Roosevelt (1884-1962); there are some sites that attribute this quote to Donald Anderson Laird (1897-1969), a psychologist and author (no photo found). Probably, more accurate. I’m not sure that the origin of the saying is really important. It may be enough to keep in mind the saying itself. (I know–how does this relate to evaluation? Trust me, it does.)
Before I was an evaluator, I was a child therapist (I also treated young women). I learned many skills as a therapist that have served me well as an evaluator. Skills like listening, standing up for your self, looking at alternatives. Which leads me to this saying. I had to “handle” others all the time at the same time I had to “handle” my self. I could not “blow up” when reprimanded. I could not become discouraged when someone (the client, the funder) criticized me. I had to learn to laugh when the joke was on me. I had to keep my spirits up when things went wrong. I had to keep cool in emergencies. I had to learn to tune out gossip and negative comments from others. This was a hard time for me. I tend to be passionate when I have an opinion; I have/had opinions (often).
As an evaluator, I am still passionate. Once my evaluation “on” button is pushed, it is hard to turn it off. Yet I still have to handle people. This morning, for example, I met with a fellow faculty member. I had to listen. I had to look for (and at) alternatives. I “handled” with my head; remember, I am passionate about evaluation. I provided her with alternatives and followed through with those alternatives. I handled with my heart.
When others are involved (and in evaluation there are always others), they must be handled with care, with the heart. It goes back to the standards (propriety) and the guiding principles (integrity/honesty, direct respect for people, and responsibilities for general and public welfare). In the current times, it is especially important to have direct respect for people. All people. (Regardless of race, ethnicity, religion, gender identity, sex, national origin, veteran status, and disability.) To be honest and have integrity. One way to make sure you have integrity is to handle with your heart.
The person without a purpose is like a ship without a rudder. (Thomas Carlyle)
There is much written about finding your purpose if life. Songs are written about purpose; self-help books are written about purpose; businesses are devoted to the concept; jewelry, leadership, among other things, all focus on purpose.
So how do you find purpose? How do you know what your are “supposed” to do in this life? How does that relate to evaluation? Finding your purpose can be really confusing. Let me share a story with you.
I lived in Birmingham, AL in the 80s and 90s. Birmingham is the only place I have lived (and I’ve lived many places) where if you woke up on the first day of spring, EVERYTHING would be in bloom. Everything! In Oregon, spring creeps up on you (a wonderful experience, to be sure). In Minnesota, it feels like it is spring one day and summer the next (or if you are not lucky, winter, again). In Tucson, spring happens in February and if you blink you miss it (well, almost). So I was marveling one day around the first day of spring how wonderful life was and I had an epiphany. I conceptualized what were the three things I wanted to do in this life. I wanted to do good work. I wanted to be a good friend. I wanted to grow spiritually. (I knew that being a boss was not for me, even though it came with perks.)
I had just finished a doctoral program in program evaluation. I realized that I would be “in the trenches” a long time and would spend most of my career doing evaluation work (as opposed to teaching evaluation, researching evaluation, writing about evaluation). I saw that as my purpose. To do good work–good evaluation work.
So what does it mean to do “good evaluation work”? Read the rest of this entry »
Recently, I read that 45% of individuals make New Year’s Resolutions and only 8% actually achieve success. Hmmm…not a friendly probability. Perhaps intentions about behavior are indeed more realistic. (Haven’t seen the statistics on that potential change. Mazanian (et al, 1998) does say stated intention to change is the most significant behavioral indicator.) My intention for 2016 is to provide content related to or about evaluation that provides you with something you didn’t have before you read the post (Point one). Examples follow: Read the rest of this entry »