value « Evaluation is an Everyday Activity

I was reminded about the age of this blog (see comment below). Then it occurred to me: I’ve been writing this blog since December 2009. That is 4 years of almost weekly posts. And even though evaluation is my primary focus, I occasionally get on my soap box and do something different (White Christmas Pie, anyone?). My other passion besides evaluation is food and cooking. I gave a Latke party on Saturday and the food was pretty–and it even tasted good. I was more impressed by the visual appeal of my table; my guests were more impressed by the array of tastes, flavors, and textures. I’d say the evening was a success. This blog is a metaphor for that table. Sometimes I’m impressed with the visual appeal; sometimes I’m impressed with the content. Today is an anniversary. Four years. I find that amazing (visual appeal). The quote below (a comment offered by a reader on the post “Is this blog making a difference?”, a post I made a long time ago) is about content.

“Judging just from the age of your blog I must speculate that you’ve done something right. If not then I doubt you’d still be writing regularly. Evaluation of your progress is important but pales in comparison to the importance of writing fresh new content on a regular basis. Content that can be found no place else is what makes a blog truly useful and indeed helps it make a difference.”

Audit or evaluation?

I’m an evaluator; I want to know what difference the “program” is making in the lives of the participants. The local school district where I live, work, and send my children to school has provided middle school children with iPads . They want to “audit” their use. I commend the school district for that initiative (both giving the iPads as well wanting to determine the effectiveness). I wonder if they really want to know what difference the electronics are making in the lives of the students. I guess I need to go re-read Tom Schwandt’s 1988 book, “Linking Auditing and Metaevaluation”, a book he wrote with Ed Halpern, as well as see what has happened in the last 25 years (and it is NOT that I do not have anything else to read…). I think it is important to note the sentence (taken from the forward), “Nontraditional studies are found not only in education, but also in…divers fields …(and the list they provide is a who’s who in social science). The problem of such studies is “establishing their merit”. That is always a problem with evaluation–establishing the merit, worth, value of a program (study).

We could spend a lot of time debating the merit, worth, value of using electronics in the pursuit of learning. (In fact, Jeffrey Selingo writes about the need to personalize instruction using electronics in his 2013 book “College (Un)bound”–very readable, recommended.) I do not think counting the number of apps or the number of page views is going to answer the question posed. I do not think counting the number of iPads returned in working condition will either. This is an interesting experiment. How , reader, would you evaluate the merit, worth, value of giving iPads to middle school children? All ideas are welcome–let me know because I do not have an answer, only an idea.

For the first time in my lifetime the first day of Hanukkah is also Thanksgiving. The pundits are are sagely calling the event Thanksgivukkah. According to this referenced source, the first day of Hanukkah will not happen again for over 70,000 years. However, according to another source, this overlap could happen again in 2070 and 2165. Although I do not think I’ll be around in 2070, my children could be (they are 17 and 20 of this writing). I find this phenomenon really interesting–Thanksgiving usually starts the US holiday season and Hanukkah falls later, during Advent. Not so this year. I wonder how people combine latkes and Thanksgiving (even without the turkey). Loaded latkes? (My appreciation to Kia.)

So I’m sure you are wondering, HOW EXACTLY DOES THIS RELATE TO EVALUATION?

I decided that it was time to revisit my blog title, Evaluation is an Everyday Activity. Every day you evaluate something. Although you do not necessarily articulate out loud the criteria against which you are determining merit, worth, and value, you have those criteria. I have them for latkes AND Thanksgiving. Our latkes must be crispy; of winter vegetables including potatoes. This allows me to use a variety of winter vegetables I may have gotten in my CSA. (Beet latkes? Sweet potato latkes? Celeriac latkes? You bet!) Our Thanksgiving is to have foods for which we are truly thankful. That allows us to think about gratitude. Each year our menu is different because each year we are thankful for different things. (I must confess, however, we always have pie–pumpkin, which I make from home grown pumpkin/squash, and chocolate pecan, which is an original old family recipe.) One year when we put all the food on the table, all the food was green. We didn’t plan it that way; it just happened because they were foods for which we were thankful. This year, we will have mashed potatoes (by the Queen of mashed potatoes), Celebration Filo, both the gluten-free (made with rice wrappers and no onion, garlic, or dairy) and glutened versions (the version which we renamed and is in the link above), and something else that will probably be green. This year I’m thankful for my gluten-free; dairy-free friend who will join us for Thanksgiving and I’m working up alternatives to accommodate her and still satisfy the rest of us.

So you see, even when I’m thinking about Thanksgiving, latkes, and gratitude, I’m thinking about evaluation. What merit does the “program” have? What is its worth? What is its value? Those are all evaluative questions that apply to Thanksgiving (and latkes and gratitude).

So you see, Evaluation is an Everyday Activity.

I won’t be blogging next week. Enjoy. Be grateful.

Variables.

We all know about independent variables, and dependent variables. Probably even learned about moderator variables, control variables and intervening variables. Have you heard of confounding variables? Variables over which you have no (or very little) control. They present as a positive or negative correlation with the dependent and independent variable. This spurious relationship plays havoc with analyses, program outcomes, and logic models. You see them often in social programs.

Ever encounter one? (Let me know). Need an example? Here is one a colleague provided. There was a program developed to assist children removed from their biologic mothers (even though the courts typically favor mothers) to improve the children’s choices and chances of success. The program had included training of key stakeholders (including judges, social service, potential foster parents). The confounding variable that wasn’t taken into account was the sudden appearance of the biological father. Judges assumed that he was no longer present (and most of the time he wasn’t); social service established fostering without taking into consideration the presence of the biological father; potential foster parents were not allerted in their training of the possibility. Needless to say, the program failed. When biologic fathers appeared (as often happened), the program had no control over the effect they had. Fathers had not been included in the program’s equation.

Reviews.

Recently, I was asked to review a grant proposal, the award would result in several hundred thousand dollars (and in today’s economy, no small change). The PI’s passion came through in the proposal’s text. However, the PI and the PI’s colleagues did some major lumping in the text that confounded the proposed outcomes. I didn’t see how what was being proposed would result in what was said to happen. This is an evaluative task. I was charged to with evaluating the proposal on technical merit, possibility of impact (certainly not world peace), and achievability. The proposal was lofty and meant well. The likelihood that it would accomplish what it proposed was unclear, despite the PI’s passion. When reviewing a proposal, it is important to think big picture as well as small picture. Most proposals will not be sustainable after the end of funding. Will the proposed project be able to really make an impact (and I’m not talking here about world peace).

Conversations.

I attended a meeting recently that focused on various aspects of diversity. (Now among the confounding here is what does one mean by diversity; is it only the intersection of gender and race/ethnicity? Or something bigger, more?) One of the presenters talked about how just by entering into the conversation, the participants would be changed. I wondered, how can that change be measured? How would you know that a change took place? Any ideas? Let me know.

Focus groups.

A colleague asked whether a focus group could be conducted via email. I had never heard of such a thing (virtual, yes; email, no). Dick Krueger and Mary Ann Casey only talk about electronic reporting in their 4th edition of their Focus Group book. If I go to Wikipedia (keep in mind it is a wiki…), there is a discussion of online focus groups. Nothing offered about email focus groups. So I ask you, readers, is it a focus group if it is conducted by email?

What follows is a primer, one of the first things evaluators learn when developing a program. This is something that cannot be said enough. Program evaluation is about the program. NOT about the person who leads the program; NOT about the policy about the program; NOT about the people who are involved in the program. IT IS ABOUT THE PROGRAM!

Phew. Now that I’ve said that. I’ll take a deep breath and elaborate.

“Anonymity, or at least a lack of face-to-face dialogue, leads people to post personal attacks…” (This was said by Nina Bahadur, Associate Editor, HuffPost Women.) Although she was speaking about blogs, not specifically program evaluation, this applies to program evaluations. Evaluations are handed out at the end of a program. Program evaluations do not ask for identifying information and often lead to personal attacks. Personal attacks are not helpful to the program lead, the program, or the participants learning.

The program lead really wants to know ABOUT THE PROGRAM, not slams about what s/he did or didn’t do; say or didn’t say. There are some things about a program over which the program lead doesn’t have any control–the air handling at the venue; the type of chairs used; the temperature of the room; sometimes, even the venue. The program lead does have control over the choice of venue (usually), the caterer (if food is offered), the materials (the program) offered to the participants, how s/he looks (grumpy or happy; serious or grateful)–I’ve just learned that how the “teacher” looks at the class makes a big difference in participants learning.

What a participant must remember is that they agreed to participate. It may have been a requirement of their job; it may have been encouraged by their boss; it may have been required by their boss. What ever the reason, they agreed to participate. They must be accountable for their participation. Commenting on those things over which the program lead has no control may make then feel better in the short run; it doesn’t do any good to improve the program or to determine if the program made a difference–that is had merit, worth, value. (Remember the root word of evaluation is VALUE.)

Personal grousing doesn’t add to the program’s value. The question that must be remembered when filling out an evaluation is, “Would this comment be said in real life (not on paper)? Would you tell the person this comment?” If not, it doesn’t belong in your evaluation. Program leads want to build a good and valuable program. The only way they can do is to receive critical feedback about the program. So if the food stinks and the program lead placed the order with the caterer, tell the program lead not to use the caterer again, don’t tell the program lead that her/his taste in food is deplorable–how does that improve the program? If the chairs are uncomfortable, tell the program lead to tell the venue that the chairs were found by participants to be uncomfortable as the program lead didn’t deliberately make the chairs uncomfortable. If there wasn’t enough time for sharing, tell the program lead to increase the sharing time because sometimes sharing of personal experiences is just what is needed to make the program meaningful to participants.

People often ask me what is a good indicator of impact…I usually answer world peace…then I get serious.

I won’t get into language today. Impact–long term outcome. For purposes of today, they are both the same: CHANGE in the person or change in the person’s behavior.

Paul Mazmanian, a medical educator at Virginia Commonwealth University School of Medicine, wanted to determine whether practicing physicians who received only clinical information at a traditional continuing medical education lecture would alter their clinical behavior at the same rate as physicians who received clinical information AND information about barriers to behavioral change. What he found is profound. Information about barriers to change did not change the physician’s clinical behavior. That is important. Sometimes research yields information that is very useful. This is the case here. Mazmanian, etal. (see complete citation below) found (drum roll, please) that both groups of physicians were statistically significantly MORE likely to change their clinical behavior if they indicated their INTENT TO CHANGE their behavior immediately following the lecture they received.

The authors concluded that stated intention to change was important in changing behavior.

We as evaluators can ask the same question: Do you intend to make a behavior change and if so, what specific change.

Albert Bandura talks about self-efficacy. That is often measured by an individual’s confidence to be able to implement a change. By pairing the two questions (How confident are you that…and Do you intend to make a change…) evaluators can often capture an indicator of behavior change; that indicator of behavior change is often the best case for long-term outcome.

I’ll be at AEA this week. Next week, I’m moving offices. I won’t be blogging.

Citation:

Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowitz, M. P. (1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine, 73(8), 882-886.

There has been quite a bit written about data visualization, a topic important to evaluators who want their findings used. Michael Patton talks about evaluation use in his 4th edition of utilization-focused evaluation. He doesn’t however list data visualization in the index; so he may talk about it somewhere–it isn’t obvious.

The current issue of New Directions for Evaluation is devoted to data visualization and it is the first part (implying, I hope, for at least a part 2). Tarek Azzam and Stephanie Evergreen are the guest editors. This volume (the first on this topic in 15 years) sets the stage (chapter 1) and talks about quantitative data visualization and quantitative data visualization. The last chapter talks about the tools that are available to the evaluator and there are many and they are various. I cannot do them justice in this space; read about them in the NDE volume. (If you are an AEA member, the volume is available on line.)

freshspectrum, a blog by Chris Lysy, talks about INTERACTIVE data visualization with illustrations.

Stephanie Evergreen, the co-guest editor of the above NDE, also blogs and in her October 2 post, talks about “Design for Federal Proposals (aka Design in a Black & White Environment)”. More on data visualization.

The data visualizer that made the largest impact on me was Hans Rosling in his TED talks. Certainly the software he uses makes the images engaging. If he didn’t understand his data the way he does, he wouldn’t be able to do what he does.

Data visualization is everywhere. There will be multiple sessions at the AEA conference next week. If you can, check them out–get there early as they will fill quickly.

When I did my dissertation, there were several soon-to-be-colleagues who were irate that I did a quantitative study on qualitative data. (I was looking at cognitive bias, actually.) I needed to reduce my qualitative data so that I could represent it quantitatively. This approach to coding is called magnitude coding. Magnitude coding is just one of the 25 first cycle coding methods that Johnny Saldaña (2013) talks about in his book, The coding manual for qualitative researchers (see pages 72-77). (I know you cannot read the cover title–this is just to give you a visual; if you want to order it, which I recommend, go to Sage Publishers, Inc.) Miles and Huberman (1994) also address this topic.

So what is magnitude coding? It is a form of coding that “consists of and adds a supplemental alphanumeric or symbolic code or sub-code to an existing coded datum…to indicate its intensity, frequency, direction, presence , or evaluative content” (Saldaña, 2013, p. 72-73). It could also indicate the absence of the characteristic of interest. Magnitude codes can be qualitative or quantitative and/or nominal. These codes enhance the description of your data.

Saldaña provides multiple examples that cover many different approaches. Magnitude codes can be words or abbreviations that suggest intensity or frequency or codes can be numbers which do the same thing. These codes can suggest direction (i.e., positive or negative, using arrows). They can also use symbols like a plus (+) or a minus (–), or other symbols indicating presence or absence of a characteristic. One important factor for evaluators to consider is that magnitude coding also suggests evaluative content, that is , did the content demonstrate merit, worth, value? (Saldaña also talks about evaluation coding; see page 119.)

Saldaña gives an example of analysis showing a summary table. Computer assisted qualitative data analysis software (CAQDAS) and Microsoft Excel can also provide summaries. He notes “that is very difficult to sidestep quantitative representation and suggestions of magnitude in any qualitative research” (Saldaña, 2013, p. 77). We use quantitative phrases all the time–most, often, extremely, frequently, seldom, few, etc. These words tend “to enhance the ‘approximate accuracy’ and texture of the prose” (Saldaña, 2013, p. 77).

Making your qualitative data quantitative is only one approach to coding, an approach that is sometimes very necessary.

Before you know it, Evaluation ’13 will be here and thousands of evaluators will converge on Washington DC, the venue for this year’s AEA annual meeting.

The Local Arrangements Working Group (LAWG) is blogging this week in AEA365. (You might want to check out all the posts this week.) There are A LOT of links in these posts (including related past posts) that are worth checking. For those who have not been to AEA before or for those who have recently embraced evaluation, reading their posts are a wealth of information.

What I want to focus on today is the role of the local arrangements working group. The Washington Evaluators group is working in tandem with AEA to organize the local part of the conference. These folks live locally and know the area. Often they include graduate students as well as seasoned evaluators. (David Bernstein and Valerie Caracelli are the co-chairs of this year’s LAWG .) They have a wealth of information in their committee. (Scroll down to the “Please Check Back for Periodic Updates” to see the large committee–it really does take a village!) They only serve for the current year and are truly local. Next year in Denver, there will be a whole new LAWG.

Some things that the committee do include identifying (and evaluating) local restaurants, things to do in DC, and getting around DC. Although these links provide valuable information, there are those of us (me… ) who are still technopeasants and do not travel with a smart phone, tablet, computer, or other electronic connectivity and would like hard copy of pertinent information. (I want to pay attention to real people in real time–I acknowledge that I am probably an artifact, certainly a technology immigrant–see previous blog about civility.)

Restaurants change quicker than I can keep track–although I’m sure that there are still some which existed when I was in DC last for business. I’m sure that today, most restaurants provide vegetarian, vegan, gluten-free options (it is, after all, the current trend). That is very different from when I was there for the last AEA in 2002. I did a quick search for vegetarian restaurants using the search options available at the LAWG/Washington Evaluators’ site–there were several…I also went to look at reviews…I wonder about the singular bad (very) review…was it just an off night or a true reflection?

There are so many things to do in DC…please take a day–the newer monuments are amazing–see them.

Getting around DC…use the Metro–it gets you to most places; it is inexpensive; it is SAFE! It has been expanded to reach beyond the DC boundaries. If nothing else, ride the Metro–you will be able to see a lot of DC. You can get from Reagan-Washington NationalAirport to the conference venue (yes, you will have to walk 4 blocks and there may be some problem with a receipt–put the fare plus $0.05 on the Metro card and turn in the card).

The LAWG has done a wonderful job providing information to evaluators…check out their site. See you in DC.

I blogged earlier this week on civility, community, compassion, and comfort. I indicated that these are related to evaluation because it is part of the values of evaluation (remember the root of evaluation is value)–is it mean or is it nice…Harold Jarche talked today about these very issues phrasing it as doing the right thing…if you do the right thing, it is nice. His blog post only reinforces the fact that evaluation is an everyday activity and that you (whether you are an evaluator or not) are the only person who can make a difference. Yes, it usually takes a village. Yes, you usually cannot see the impact of what you do (we can’t get easily to world peace). Yes, you can be the change you want to see. Yes, evaluation is an every day activity. Make nice, folks. Try a little civility; expand your community; remember compassion. Comfort is the outcome. Comfort seems like a good outcome. So does doing the right thing.

I know–how does this relate to evaluation? Although I think it is obvious, perhaps it isn’t.

I’ll start with a little background. In 1994, M. Scott Peck published A World Waiting To Be Born: Civility Rediscovered. In that book he defined a problem (and there are many) facing the then 20th century person ( I think it applies to the 21st century person as well). That problem was incivility or the “…morally destructive patterns of self-absorption, callousness, manipulativeness, and materialism so ingrained in our routine behavior that we do not even recognize them.” He wrote this in 1994–well before the advent of the technology that has enabled humon to disconnect from fellow humon while being connected. Look about you and count the folks with smart phones. Now, I’ll be the first to agree that technology has enabled a myriad of activities that 20 years ago (when Peck was writing this book) were not even conceived by ordinary folks. Then technology took off…and as a result, civility, community, and, yes, even compassion went by the way.

Self-absorption, callousness, manipulativeness, materialism are all characteristics of the lack of, not only civility (as Peck writes), also loss of community and lack of compassion. If those three (civility, community, compassion) are lost–where is there comfort? Seems to me that these three are interrelated.

To expand–How many times have you used your smart phone to text someone across the room? (Was it so important you couldn’t wait until you could talk to him/her in person–face-to-face?) How often have you thought to yourself how awful an event is and didn’t bother to tell the other person? How often did you say the good word? The right thing? That is evaluation–in the everyday sense. Those of us who call ourselves evaluators are only slightly different from those of you who don’t. Although evaluators do evaluation for a living, everyone does it because evaluation is part of what gets us all through the day.

Ask your self as an evaluative task–was I nice or was I mean? This reflects civility, compassion, and even community.–even very young children know that difference. Civility and compassion can be taught to kindergarteners–ask the next five year old you see–was it nice or was it mean? They will tell you. They don’t lie. Lying is a learned behavior–that, too, is evaluative.

You can ask your self guiding questions about community; about compassion; about comfort. They are all evaluative questions because you are trying to determine if you have made a difference. You CAN be the change you want to see in the world; you can be the change you want to be. That, too is evaluative. Civility. Compassion. Community. Comfort.

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: value

Anniversary

Thanksgivukkah

Variables and other things

Program Evaluation

Intention to change

Data visualization

Magnitude coding

Evaluation ’13

Doing the right thing or doing the thing right?

Civility. Compassion. Community. Comfort.

Contact Info