difference « Evaluation is an Everyday Activity

To quote Annie Leonard, the word sustainability “gets thrown around all the time now and it’s not always clear what is intended.” She goes on to talk about the UN World Commission on Environment and Development definition of sustainable development as “…meeting the needs of the current generation without compromising the ability of future generations to meet their own needs.” That is a good definition, I think. Yet it is missing something which (according to Leonard) are equity and justice. Robert Gilman defines sustainability as “…equity over time”. She says (and I agree), quoting the Center for Sustainable Communities, that sustainability “consider(s) the whole instead of the specific. Sustainability emphasizes relationships rather than pieces in isolation.”

Now, given that evaluation to be effective must look at the whole (here is a good example of when the whole is greater than the sum of its parts); and

given that evaluation works to find out information that will benefit both the current and future generations; and

given that evaluation works to determine what difference was made in people’s lives, it seems to me that there is a relationship here that needs to be acknowledged.

A colleague of mine works in youth development and loves the job. My colleague has to determine the value of the program; the program needs to be evaluated. Yet, if the work is only for the program (i.e., the pieces in isolation) not the whole, what good is it that my colleague loves the job? The relationship between the youth involved and the bigger picture is truly more than can probably be captured in any evaluation. Still, the evaluation needs to be planned to consider that, even if the resources are limited (that is the “probably” above).

So yes, evaluation has something to learn from sustainability. Certainly sustainability can learn from evaluation (and economics, and equity, and ecology…).

NEW TOPIC

I’ve been, once again, getting comments about making a difference. I thought I’d post some of those comments (I’ve copied and pasted comments so the spelling is as it appears in the original text):

…every blog post makes a difference in a way or in another. You can answer at your questions just seeing how many comments are here, how many people are interested in answering you. I think you are a good person, and everything said by a good person is always a life’s lesson to keep in mind. Thank you for every helpful information, good job!
It may be a temporary difference – i.e. limited on the time, but of course that at least for some seconds your writing are touching the life’s of all your readers.
Every blog or article makes a difference to those who read it! They might strongly agree or disagree with what the blogger has wrote, making a difference by reafirming there opinion or being outraged that somebody else looks at ideas different to them! Keep writing Molly, you are making people think, which is always good
I think the best measure of the effectiveness of a blog are the number of shares it gets, as people that found something useful in it tend to want to share with others.
…I have written quite a bit about this topic and challenge that bloggers face and the bottom line is that you really can’t measure the value. Sure I think asking for responses like you did might help you see a bit of it, but the reality is 99.9% of people will never comment. As such, we as bloggers have to remember that each pageview is a real person who was on our site and who was impacted by what we wrote!
Blogs are probably the best tool for engaging a customer in todays times.

My question: are blogs engaging readers or are they only outreach, even if the blog is read?

P.S. I also got a lot of comments about my analytics post…for next time.

References:

Leonard, A. (2011). Story of Stuff. NY: Free Press. (good book–worth the read)

Report of the World Commission on Environment and Development. (1987). NY: UN World Commission on Environment and Development. http://www.un-documents.net/our-common-future.pdf

Gilman, R., Director, Context Institute.

He says: Sustainability is equity over time. As a value, it refers to giving equal weight in your decisions to the future as well as the present. You might think of it as extending the Golden Rule through time, so that you do unto future generations (as well as to your present fellow beings) as you would have them do unto you.

Center for Sustainable Communities is quoted in a variety of places: http://sustainablesonoma.org/keyconcepts/sustainability.html; http://isocs-sustainability.wikispaces.com, among others.

The entire definition is: Sustainability is part of a trend to…consider the whole instead of the specific. Sustainability emphasizes relationships rather than pieces in isolation…Sustainability is not about regressing to primitive living conditions. It is about understanding our situation, and developing as communities in ways that are equitable, and make sense ecologically and economically.

Having just read Harold Jarche’s April 27, 2014 blog, making sense of the network era, about personal knowledge mastery (PKM), I am once again reminded about the challenge of evaluation. I am often asked, “Do you have a form I could use about…?” My nutrition and exercise questions notwithstanding (I do have notebooks of those), this makes evaluation sound like it is routine, standardized, or prepackaged rather than individualized, customized, or specific. For me, evaluation is about the exceptions to the rule; how the evaluation this week may have similarities to something I’ve done before (after all this time, I would hope so…), yet is so different; unique, specific.

You can’t expect to find a pre-made form for your individual program (unless, of course you are replicating a previously established program). Evaluations are unique and the evaluation approach needs to match that unique program specialness. Whether the evaluation uses a survey, a focus group, or an observation (or any other data gathering approach), that approach to gathering data needs to focus on the evaluation question you want answered. You can start with “What difference did the program make?” Only you, the evaluator, can determine if you have enough resources to conduct the evaluation to answer the specific questions that result from what difference did the program make. You probably do not have enough resources to determine if the program led your target audience to world peace; you might have enough resources to determine if the intention to do something different is there. You probably have enough resources to decide how to use your findings. It is so important that the findings be used; use may be how world peace may be accomplished.

There are a few commonalities in data collection; those are the demographics, the data that tell you what your target audience looks like. Things like gender, age, marital status, education level, SES, probably a few other things depending on the program. Make sure when you ask demographic information that a “choose not to answer” option is provided in the survey. Sometimes you have to ask; observations don’t always provide the answer. You need to make sure you include demographics in your survey as most journals want to know what the target audience looked like.

Readers, what makes your evaluations different, unique, special? I’d like to hear about that. Oh and while you are at it…like and share this post, if you do.

A reader commented that I need to be attending to my analytics not just reading my comments. Hmmm…

My question is: what do analytics tell me about making a difference–by providing an educational forum that changes people am I making a difference? Keep in mind that I am an evaluator and that the root for the word evaluation is “value”. So I wonder, do the analytics tell me about the merit, worth, value of this educational intervention?

What will the analytics really tell me about the readers? What will the comments tell me that the analytics don’t? Will the analytics tell me what difference this blog has made in the readers. Will analytics tell me about intention to change? How will analytics help me write posts to which more people will respond; make me more of an authority in my posting?

I DO NOT KNOW.

If someone, any one out in cyber space knows the answers (readers?), I’d love to hear from you. I blog weekly; sometimes more than weekly (like this week because, although I had the post written, I didn’t get it posted before I left the office so I posted it when I came back). I check my blog regularly for comments. I approve those which provide thoughtful meaningful responses for other readers as well as for me.

Another reader suggests that I look at the number of readers who have established an RSS feed or established a subscription. Hmmm…Not sure what that will tell me. I’ll talk to the IT folks for an answer to that question.

I would certainly appreciate any thoughts from readers.

What follows is a primer, one of the first things evaluators learn when developing a program. This is something that cannot be said enough. Program evaluation is about the program. NOT about the person who leads the program; NOT about the policy about the program; NOT about the people who are involved in the program. IT IS ABOUT THE PROGRAM!

Phew. Now that I’ve said that. I’ll take a deep breath and elaborate.

“Anonymity, or at least a lack of face-to-face dialogue, leads people to post personal attacks…” (This was said by Nina Bahadur, Associate Editor, HuffPost Women.) Although she was speaking about blogs, not specifically program evaluation, this applies to program evaluations. Evaluations are handed out at the end of a program. Program evaluations do not ask for identifying information and often lead to personal attacks. Personal attacks are not helpful to the program lead, the program, or the participants learning.

The program lead really wants to know ABOUT THE PROGRAM, not slams about what s/he did or didn’t do; say or didn’t say. There are some things about a program over which the program lead doesn’t have any control–the air handling at the venue; the type of chairs used; the temperature of the room; sometimes, even the venue. The program lead does have control over the choice of venue (usually), the caterer (if food is offered), the materials (the program) offered to the participants, how s/he looks (grumpy or happy; serious or grateful)–I’ve just learned that how the “teacher” looks at the class makes a big difference in participants learning.

What a participant must remember is that they agreed to participate. It may have been a requirement of their job; it may have been encouraged by their boss; it may have been required by their boss. What ever the reason, they agreed to participate. They must be accountable for their participation. Commenting on those things over which the program lead has no control may make then feel better in the short run; it doesn’t do any good to improve the program or to determine if the program made a difference–that is had merit, worth, value. (Remember the root word of evaluation is VALUE.)

Personal grousing doesn’t add to the program’s value. The question that must be remembered when filling out an evaluation is, “Would this comment be said in real life (not on paper)? Would you tell the person this comment?” If not, it doesn’t belong in your evaluation. Program leads want to build a good and valuable program. The only way they can do is to receive critical feedback about the program. So if the food stinks and the program lead placed the order with the caterer, tell the program lead not to use the caterer again, don’t tell the program lead that her/his taste in food is deplorable–how does that improve the program? If the chairs are uncomfortable, tell the program lead to tell the venue that the chairs were found by participants to be uncomfortable as the program lead didn’t deliberately make the chairs uncomfortable. If there wasn’t enough time for sharing, tell the program lead to increase the sharing time because sometimes sharing of personal experiences is just what is needed to make the program meaningful to participants.

I know–how does this relate to evaluation? Although I think it is obvious, perhaps it isn’t.

I’ll start with a little background. In 1994, M. Scott Peck published A World Waiting To Be Born: Civility Rediscovered. In that book he defined a problem (and there are many) facing the then 20th century person ( I think it applies to the 21st century person as well). That problem was incivility or the “…morally destructive patterns of self-absorption, callousness, manipulativeness, and materialism so ingrained in our routine behavior that we do not even recognize them.” He wrote this in 1994–well before the advent of the technology that has enabled humon to disconnect from fellow humon while being connected. Look about you and count the folks with smart phones. Now, I’ll be the first to agree that technology has enabled a myriad of activities that 20 years ago (when Peck was writing this book) were not even conceived by ordinary folks. Then technology took off…and as a result, civility, community, and, yes, even compassion went by the way.

Self-absorption, callousness, manipulativeness, materialism are all characteristics of the lack of, not only civility (as Peck writes), also loss of community and lack of compassion. If those three (civility, community, compassion) are lost–where is there comfort? Seems to me that these three are interrelated.

To expand–How many times have you used your smart phone to text someone across the room? (Was it so important you couldn’t wait until you could talk to him/her in person–face-to-face?) How often have you thought to yourself how awful an event is and didn’t bother to tell the other person? How often did you say the good word? The right thing? That is evaluation–in the everyday sense. Those of us who call ourselves evaluators are only slightly different from those of you who don’t. Although evaluators do evaluation for a living, everyone does it because evaluation is part of what gets us all through the day.

Ask your self as an evaluative task–was I nice or was I mean? This reflects civility, compassion, and even community.–even very young children know that difference. Civility and compassion can be taught to kindergarteners–ask the next five year old you see–was it nice or was it mean? They will tell you. They don’t lie. Lying is a learned behavior–that, too, is evaluative.

You can ask your self guiding questions about community; about compassion; about comfort. They are all evaluative questions because you are trying to determine if you have made a difference. You CAN be the change you want to see in the world; you can be the change you want to be. That, too is evaluative. Civility. Compassion. Community. Comfort.

Recently, I came across a blog post by Daniel Green, who is the head of strategic media partnerships at the Bill and Melinda Gates Foundation. He coauthored this post with Mayur Patel, vice president of strategy and assessment at the Knight Foundation. I mention this because those two foundations have contributed $3.25 million in seed funding “…to advance a better understanding of audience engagement and media impact…”. They are undertaking an ambitious project to develop a rubric (of sorts) to determine “…how media influences the ways people think and act, and contributes to broader societal changes…”. Although it doesn’t specifically say, I include social media in the broad use of “media”. The blog post talks about broader agenda–that of informed and engaged communities. These foundations believe that an informed and engaged communities will strengthen “… democracy and civil society to helping address some of the world’s most challenging social problems.”

Or in other words, what difference is being made, which is something I wonder about all the time. (I’m an evaluator, after all, and I want to know what difference is made.)

Although there are strong media forces out there (NYTimes, NPR, BBC, the Guardian, among others), I wonder about the strength and effect of social media (FB, Twitter, LinkedIn, blogs, among others). Anecdotally, I can tell you that social media is everywhere and IS changing the way people think and act. I watch my now 17 y/o who uses the IM feature on her social media to communicate with her friends, set up study dates, find out homework assignments, not the phone like I did. I watch my now 20 y/o multitask–talk to me on Skype and read and respond to her FB entry. She uses IM as much as her sister. I know that social media was instrumental in the Arab spring. I know that major institutions have social media connections (FB, Twitter, LinkedIn, etc.). Social media is everywhere. And we have no good way to determine if it is making a difference and what that difference is.

For something so ubiquitous (social media), why is there no way to evaluate social media other than through the use of analytics? I’ve been asking that question since I first posted my query “Is this blog making a difference?” back in March 2012. Since I’ve been posting since December 2009, that gave me over 2 years from which to gather data. That is a luxury when it comes to programming, especially when many programs often are a few hours in duration and an evaluation is expected.

I hope that this project provides useful information for those of us who have come kicking and screaming to social media and have seen the light. Even though they are talking about the world of media, I’m hoping that they can come up with measures that address the social aspect of media. The technology provided IS useful; the question is what difference is it making?

We are four months into 2013 and I keep asking the question “Is this blog making a difference?” I’ve asked for an analytic report to give me some answers. I’ve asked you readers for your stories.

Let’s hear it for SEOs and how they pick up that title–I credit that with the number of comments I’ve gotten. I AM surprised at the number of comments I have gotten since January (hundreds, literally). Most say things like, “of course it is making a difference.” Some compliment me on my writing style. Some are in a foreign language which I cannot read (I am illiterate when it comes to Cyrillic, Arabic, Greek, Chinese, and other non-English alphabets). Some are marketing–wanting ping backs to their recently started blogs for some product. Some have commented specifically on the content (sample size and confidence intervals); some have commented on the time of year (vernal equinox). Occasionally, I get a comment like the comment below and I keep writing.

The questions of all questions… Do I make a difference? I like how you write and let me answer your question. Personally I was supposed to be dead ages ago because someone tried to kill me for the h… of it … Since then (I barely survived) I have asked myself the same question several times and every single time I answer with YES. Why? Because I noticed that whatever you do, there is always someone using what you say or do to improve their own life. So, I can answer the question for you: Do you make a difference? Yes, you do, because there will always be someone who uses your writings to do something positive with it. So, I hope I just made your day! 🙂 And needless to say, keep the blog posts coming!

Enough update. New topic: I just got a copy of the third edition of Miles and Huberman (my to go reference for qualitative data analysis). Wait you say–Miles and Huberman are dead–yes, they are. Johnny Saldana (there needs to be a~ above the “n” in his name only I don’t know how to do that with this keyboard) was approached by Sage to be the third author and revise and update the book. A good thing, I think. Miles and Huberman’s second edition was published in 1994. That is almost 20 years. I’m eager to see if it will hold as a classic given that there are many other books on qualitative coding in press currently. (The spring research flyer from Gilford lists several on qualitative inquiry and analysis from some established authors.)

I also recently sat in on a research presentation of a candidate for a tenure track position here at OSU who talked about how the analysis of qualitative data was accomplished. Took me back to when I was learning–index cards and sticky notes. Yes, there are marvelous software programs out there (NVivo, Ethnograph, N*udist); I will support the argument that the best way to learn about your qualitative data is to immerse yourself in it with color coded index cards and sticky notes. Then you can use the software to check your results. Keep in mind, though, that you are the PI and you will bring many biases to the analysis of your data.

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: difference

Sustainability relates to evaluation

Routine vs. Exceptions

Making a difference–analytics?

Program Evaluation

Civility. Compassion. Community. Comfort.

Making a difference

April update on making a difference

Contact Info