September « 2013 « Evaluation is an Everyday Activity

Before you know it, Evaluation ’13 will be here and thousands of evaluators will converge on Washington DC, the venue for this year’s AEA annual meeting.

The Local Arrangements Working Group (LAWG) is blogging this week in AEA365. (You might want to check out all the posts this week.) There are A LOT of links in these posts (including related past posts) that are worth checking. For those who have not been to AEA before or for those who have recently embraced evaluation, reading their posts are a wealth of information.

What I want to focus on today is the role of the local arrangements working group. The Washington Evaluators group is working in tandem with AEA to organize the local part of the conference. These folks live locally and know the area. Often they include graduate students as well as seasoned evaluators. (David Bernstein and Valerie Caracelli are the co-chairs of this year’s LAWG .) They have a wealth of information in their committee. (Scroll down to the “Please Check Back for Periodic Updates” to see the large committee–it really does take a village!) They only serve for the current year and are truly local. Next year in Denver, there will be a whole new LAWG.

Some things that the committee do include identifying (and evaluating) local restaurants, things to do in DC, and getting around DC. Although these links provide valuable information, there are those of us (me… ) who are still technopeasants and do not travel with a smart phone, tablet, computer, or other electronic connectivity and would like hard copy of pertinent information. (I want to pay attention to real people in real time–I acknowledge that I am probably an artifact, certainly a technology immigrant–see previous blog about civility.)

Restaurants change quicker than I can keep track–although I’m sure that there are still some which existed when I was in DC last for business. I’m sure that today, most restaurants provide vegetarian, vegan, gluten-free options (it is, after all, the current trend). That is very different from when I was there for the last AEA in 2002. I did a quick search for vegetarian restaurants using the search options available at the LAWG/Washington Evaluators’ site–there were several…I also went to look at reviews…I wonder about the singular bad (very) review…was it just an off night or a true reflection?

There are so many things to do in DC…please take a day–the newer monuments are amazing–see them.

Getting around DC…use the Metro–it gets you to most places; it is inexpensive; it is SAFE! It has been expanded to reach beyond the DC boundaries. If nothing else, ride the Metro–you will be able to see a lot of DC. You can get from Reagan-Washington NationalAirport to the conference venue (yes, you will have to walk 4 blocks and there may be some problem with a receipt–put the fare plus $0.05 on the Metro card and turn in the card).

The LAWG has done a wonderful job providing information to evaluators…check out their site. See you in DC.

I blogged earlier this week on civility, community, compassion, and comfort. I indicated that these are related to evaluation because it is part of the values of evaluation (remember the root of evaluation is value)–is it mean or is it nice…Harold Jarche talked today about these very issues phrasing it as doing the right thing…if you do the right thing, it is nice. His blog post only reinforces the fact that evaluation is an everyday activity and that you (whether you are an evaluator or not) are the only person who can make a difference. Yes, it usually takes a village. Yes, you usually cannot see the impact of what you do (we can’t get easily to world peace). Yes, you can be the change you want to see. Yes, evaluation is an every day activity. Make nice, folks. Try a little civility; expand your community; remember compassion. Comfort is the outcome. Comfort seems like a good outcome. So does doing the right thing.

I know–how does this relate to evaluation? Although I think it is obvious, perhaps it isn’t.

I’ll start with a little background. In 1994, M. Scott Peck published A World Waiting To Be Born: Civility Rediscovered. In that book he defined a problem (and there are many) facing the then 20th century person ( I think it applies to the 21st century person as well). That problem was incivility or the “…morally destructive patterns of self-absorption, callousness, manipulativeness, and materialism so ingrained in our routine behavior that we do not even recognize them.” He wrote this in 1994–well before the advent of the technology that has enabled humon to disconnect from fellow humon while being connected. Look about you and count the folks with smart phones. Now, I’ll be the first to agree that technology has enabled a myriad of activities that 20 years ago (when Peck was writing this book) were not even conceived by ordinary folks. Then technology took off…and as a result, civility, community, and, yes, even compassion went by the way.

Self-absorption, callousness, manipulativeness, materialism are all characteristics of the lack of, not only civility (as Peck writes), also loss of community and lack of compassion. If those three (civility, community, compassion) are lost–where is there comfort? Seems to me that these three are interrelated.

To expand–How many times have you used your smart phone to text someone across the room? (Was it so important you couldn’t wait until you could talk to him/her in person–face-to-face?) How often have you thought to yourself how awful an event is and didn’t bother to tell the other person? How often did you say the good word? The right thing? That is evaluation–in the everyday sense. Those of us who call ourselves evaluators are only slightly different from those of you who don’t. Although evaluators do evaluation for a living, everyone does it because evaluation is part of what gets us all through the day.

Ask your self as an evaluative task–was I nice or was I mean? This reflects civility, compassion, and even community.–even very young children know that difference. Civility and compassion can be taught to kindergarteners–ask the next five year old you see–was it nice or was it mean? They will tell you. They don’t lie. Lying is a learned behavior–that, too, is evaluative.

You can ask your self guiding questions about community; about compassion; about comfort. They are all evaluative questions because you are trying to determine if you have made a difference. You CAN be the change you want to see in the world; you can be the change you want to be. That, too is evaluative. Civility. Compassion. Community. Comfort.

I read a lot of blogs. Some blogs are on evaluation; some are on education; some are on food; some are random (travel, health)…I get ideas from each of them, although not all at the same time. Once I have an idea I write my blog.

Today, I’m drawing from several blogs to come up with these thoughts. I read AEA 365 by Sheila B, Robinson (I’m a little behind), Harold Jarche blog (Life in perpetual beta), and Eval Central for Friday, September 6, 2013 (Eval Central compiles blogs related to evaluation). These seemingly unrelated posts (all very interesting) talk about continued learning from different perspectives.

AEA365 talks about “Ancora imparo” (from Michelangelo) or still learning. (This saying is attributed to Michelangelo who was supposed to have said this in his 80s.) Evaluators must continue to learn–the field is changing so fast. Michael Scriven, who gave a keynote at the Australasian Evaluation Society (as posted in Genuine Evaluation) talks about the field of evaluation “taking the core” to become the “inner circle” because evaluation is the “alpha discipline”. In order to do that, evaluators must continue to learn (although Michael probably has a more eloquent approach). Reading helps. Talking helps. Working helps. From all of these, learning can occur.

Harold Jarche reviews a book by Charles Jennings, the foremost authority on the 70-20-10 framework, talks about how this “holistic framework…is not a recipe” for learning, even though the numbers stand for 70% Experience, 20% Exposure, and 10% Education which make it sound like a recipe. (Substitute flour, butter, sugar–makes a recipe for a type of short bread; even food makes its way into my blog…) As an evaluator, learning at work (experience) seems to be the same learning that Robinson talks about, not classroom learning. Jennings “…describes workplace learning as based on four key activities:

Exposure to new and rich experiences.
The opportunity to practice.
Engaging in conversation and exchanges with each other.
Making time to reflect on new observations, information, experiences, etc.”

Sounds like the same thing to me–reading, talking, working; continued learning–ancora imparo .

The last blog I read (the Eval Central post by Patricia Rogers in Better Evaluation blogs) reporting on another keynote at the Australasian Evaluation Society (this keynote by Nan Wehipihana). The blog reports Nan Wehipeihana and her framework of increasing control by Indigenous communities in evaluation (Her “to” , “for”, “with”, “by” and “as” framework.) This is a different way of looking at evaluation; one from which all evaluators can learn; one from which all evaluators can add to their own tool box. (Check out the blog post for an informative graphic about this framework.)

Seems like everything I’m reading right now talks about continued learning, and not in the classroom. Hmmm…learning is important if what Michael Scriven talks about is going to happen.

Wow! 25 First Cycle and 6 Second Cycle methods for coding qualitative data.

Who would have thought that there are so many methods of coding qualitative data. I’ve been coding qualitative data for a long time and only now am I aware that what I was doing was, according to Miles and Huberman (1994), my go-to book for coding, miles and huberman qualitative data is called “Descriptive Coding” although Johnny Saldana calls it “Attribute Coding”. (This is discussed at length in his volume The Coding Manual for Qualitative Researchers.) I just thought I was coding; I was, just not as systematically as suggested by Saldana.

Saldana talks about First Cycle coding methods, Second Cycle coding methods and a hybrid method that lies between them. He lists 25 First Cycle coding methods and spends over 120 pages discussing first cycle coding.

I’m quoting now. He says that “First Cycle methods are those processes that happen during the initial coding of data and are divided into seven subcategories: Grammatical, Elemental, Affective, Literary and Language, Exploratory, Procedural and a final profile entitled Themeing the Data. Second Cycle methods are a bit more challenging because they require such analytic skills as classifying, prioritizing, integrating, synthesizing, abstracting, conceptualizing, and theory building.”

He also insists that coding qualitative data is a iterative process; that data are coded and recoded. Not just a one pass through the data.

Somewhere I missed the boat. What occurs to me is that since I learned about coding qualitative data by hand because there were few CAQDAS (Computer Assisted Qualitative Data Analysis Software) available (something Saldana advocates for nascent qualitative researchers) is that the field has developed, refined, expanded, and become detailed. Much work has been done that went unobserved by me.

He also talks about the fact that the study’s qualitative data may need more than one coding method–Yikes! I thought there was only one. Boy was I mistaken. Reading the Coding Manual is enlightening (a good example of life long learning). All this will come in handy when I collect the qualitative data for the evaluation I’m now planning. Another take away point that is stressed in the coding manual and in the third edition of the Miles & Huberman book (with the co-author of Johnny Saldana) is to start coding/reading the data as soon as it is collected. Reading the data when you collect it allows you to remember what you observed/heard, allows/encourages analytic memo writing (more on that in a separate post), and allows you to start building your coding scheme.

If you do a lot of qualitative data collection, you need these two books on your shelf.

Evaluation is an Everyday Activity

Program Evaluation Discussions

Monthly Archives: September 2013

Evaluation ’13

Doing the right thing or doing the thing right?

Civility. Compassion. Community. Comfort.

Some thoughts about evaluation

Qualitative data analysis

Contact Info