Methodology « Evaluation is an Everyday Activity

Did you know that there are at least 11 winter holidays besides Christmas–many of them related to light or the return of light.

One needs evaluation tools to determine the merit or worth, to evaluate the holiday’s value to you. For me, any that return light are important. So for me, there is Hanukkah (and eight candles), Solstice (and bonfires and yule logs), Christmas (and Advent wreaths with five candles), Kwanzaa ( and kinara seven candles). Sometimes Diwali falls late in November to be included (it is the ancient Hindu festival of lights that is a movable feast like Hanukkah).

I have celebrations for Hanukkah (I have several menorahs), for Solstice (I have two special candelabra that holds 12 candles–a mini-bonfire to be sure), for Advent/Christmas (I make a wreath each year), and for Kwanzaa (a handmade Kinara). And foods for each celebration as well. Because I live in a multicultural household, it is important that everyone understand that no holiday is more important than any other–all talk about returning light (literal or figurative). Sometimes the holidays over lap–Hanukkah, Solstice, Christmas all in the same week…phew, I’m exhausted just thinking about it. Sometimes it seems hard to keep them separate–then I realized that returning the light is not separate; it is light returning. It is an evaluative task.

So well come the new born sun/son…the light returns. Evaluation continues.

Happy Holidays…all of them!

I’m taking two weeks holiday–will see you in the new year.

People often ask me what is a good indicator of impact…I usually answer world peace…then I get serious.

I won’t get into language today. Impact–long term outcome. For purposes of today, they are both the same: CHANGE in the person or change in the person’s behavior.

Paul Mazmanian, a medical educator at Virginia Commonwealth University School of Medicine, wanted to determine whether practicing physicians who received only clinical information at a traditional continuing medical education lecture would alter their clinical behavior at the same rate as physicians who received clinical information AND information about barriers to behavioral change. What he found is profound. Information about barriers to change did not change the physician’s clinical behavior. That is important. Sometimes research yields information that is very useful. This is the case here. Mazmanian, etal. (see complete citation below) found (drum roll, please) that both groups of physicians were statistically significantly MORE likely to change their clinical behavior if they indicated their INTENT TO CHANGE their behavior immediately following the lecture they received.

The authors concluded that stated intention to change was important in changing behavior.

We as evaluators can ask the same question: Do you intend to make a behavior change and if so, what specific change.

Albert Bandura talks about self-efficacy. That is often measured by an individual’s confidence to be able to implement a change. By pairing the two questions (How confident are you that…and Do you intend to make a change…) evaluators can often capture an indicator of behavior change; that indicator of behavior change is often the best case for long-term outcome.

I’ll be at AEA this week. Next week, I’m moving offices. I won’t be blogging.

Citation:

Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowitz, M. P. (1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine, 73(8), 882-886.

When I did my dissertation, there were several soon-to-be-colleagues who were irate that I did a quantitative study on qualitative data. (I was looking at cognitive bias, actually.) I needed to reduce my qualitative data so that I could represent it quantitatively. This approach to coding is called magnitude coding. Magnitude coding is just one of the 25 first cycle coding methods that Johnny Saldaña (2013) talks about in his book, The coding manual for qualitative researchers (see pages 72-77). (I know you cannot read the cover title–this is just to give you a visual; if you want to order it, which I recommend, go to Sage Publishers, Inc.) Miles and Huberman (1994) also address this topic.

So what is magnitude coding? It is a form of coding that “consists of and adds a supplemental alphanumeric or symbolic code or sub-code to an existing coded datum…to indicate its intensity, frequency, direction, presence , or evaluative content” (Saldaña, 2013, p. 72-73). It could also indicate the absence of the characteristic of interest. Magnitude codes can be qualitative or quantitative and/or nominal. These codes enhance the description of your data.

Saldaña provides multiple examples that cover many different approaches. Magnitude codes can be words or abbreviations that suggest intensity or frequency or codes can be numbers which do the same thing. These codes can suggest direction (i.e., positive or negative, using arrows). They can also use symbols like a plus (+) or a minus (–), or other symbols indicating presence or absence of a characteristic. One important factor for evaluators to consider is that magnitude coding also suggests evaluative content, that is , did the content demonstrate merit, worth, value? (Saldaña also talks about evaluation coding; see page 119.)

Saldaña gives an example of analysis showing a summary table. Computer assisted qualitative data analysis software (CAQDAS) and Microsoft Excel can also provide summaries. He notes “that is very difficult to sidestep quantitative representation and suggestions of magnitude in any qualitative research” (Saldaña, 2013, p. 77). We use quantitative phrases all the time–most, often, extremely, frequently, seldom, few, etc. These words tend “to enhance the ‘approximate accuracy’ and texture of the prose” (Saldaña, 2013, p. 77).

Making your qualitative data quantitative is only one approach to coding, an approach that is sometimes very necessary.

Wow! 25 First Cycle and 6 Second Cycle methods for coding qualitative data.

Who would have thought that there are so many methods of coding qualitative data. I’ve been coding qualitative data for a long time and only now am I aware that what I was doing was, according to Miles and Huberman (1994), my go-to book for coding, miles and huberman qualitative data is called “Descriptive Coding” although Johnny Saldana calls it “Attribute Coding”. (This is discussed at length in his volume The Coding Manual for Qualitative Researchers.) I just thought I was coding; I was, just not as systematically as suggested by Saldana.

Saldana talks about First Cycle coding methods, Second Cycle coding methods and a hybrid method that lies between them. He lists 25 First Cycle coding methods and spends over 120 pages discussing first cycle coding.

I’m quoting now. He says that “First Cycle methods are those processes that happen during the initial coding of data and are divided into seven subcategories: Grammatical, Elemental, Affective, Literary and Language, Exploratory, Procedural and a final profile entitled Themeing the Data. Second Cycle methods are a bit more challenging because they require such analytic skills as classifying, prioritizing, integrating, synthesizing, abstracting, conceptualizing, and theory building.”

He also insists that coding qualitative data is a iterative process; that data are coded and recoded. Not just a one pass through the data.

Somewhere I missed the boat. What occurs to me is that since I learned about coding qualitative data by hand because there were few CAQDAS (Computer Assisted Qualitative Data Analysis Software) available (something Saldana advocates for nascent qualitative researchers) is that the field has developed, refined, expanded, and become detailed. Much work has been done that went unobserved by me.

He also talks about the fact that the study’s qualitative data may need more than one coding method–Yikes! I thought there was only one. Boy was I mistaken. Reading the Coding Manual is enlightening (a good example of life long learning). All this will come in handy when I collect the qualitative data for the evaluation I’m now planning. Another take away point that is stressed in the coding manual and in the third edition of the Miles & Huberman book (with the co-author of Johnny Saldana) is to start coding/reading the data as soon as it is collected. Reading the data when you collect it allows you to remember what you observed/heard, allows/encourages analytic memo writing (more on that in a separate post), and allows you to start building your coding scheme.

If you do a lot of qualitative data collection, you need these two books on your shelf.

We are approaching Evaluation 2013 (Evaluation ’13). This year October 16-19, with professional development sessions both before and after the conference. One of the criteria that I use to determine a “good” conference is did I get three new ideas (three is an arbitrary number). One way to get a good idea to use outside the conference, in your work, in your everyday activities is to experience a good presentation. Fortunately, in the last 15 years much has been written on how to give a good presentation both verbally and with visual support. This week’s AEA365 blog (by Susan Kistler) talks about presentations as she tells us again about the P2i initiative sponsored by AEA.

I’ve delivered posters the last few years (five or six) and P2i talks about posters in the downloadable handout called, Guidelines for Posters. Under the tab called (appropriately enough) Posters, P2i also offers information on research posters and a review of other posters as well as the above mentioned Guidelines for Posters. Although more and more folks are moving to posters (until AEA runs out of room, all posters are on the program), paper presentations with the accompanying power point are still deriguere, the custom of professional conferences. What P2i has to say about presentations will help you A LOT!! Read it.

Read it especially if presenting in public, whether to a large group of people or not. It will help you. There are some really valuable points that are reiterated in the AEA365 as well as other places. Check out the following TED talks, look especially for Nancy Durate and Hans Rosling. A quick internet search yielded the following: About 241,000,000 results (0.43 seconds). I entered the phrase, “how to make a good presentation“. Some of the sites speak to oral presentations; some address visual presentations. What most people do is try to get too much information on a slide (typically using Power point). Prezi gives you one slide with multiple images imbedded within it. It is cool. There are probably other approaches as well. In today’s world, there is no reason to read your presentation–your audience can do that. Tell them! (You know, tell them what they will hear, tell them, tell them what they heard…or something like that.) If you have to read, make sure what they see is what they hear–see hear compatibility is still important, regardless of the media used.

Make an interesting presentation! Give your audience at least one good idea!

I’m about to start a large scale project, one that will be primarily qualitative (it may end up being a mixed methods study; time will tell); I’m in the planning stages with the PI now. I’ve done qualitative studies before–how could I not with all the time I’ve been an evaluator? My go to book for qualitative data analysis has always been Miles and Huberman (although my volume is black). This is their second edition published in 1994. I loved that book for a variety of reasons: 1) it provided me with a road map to process qualitative data; 2) it offered the reader an appendix for choosing a qualitative software program (now out of date); and 3) it was systematic and detailed in its description of display. I was very saddened to learn that both the authors had died and there would not be a third edition. Imaging my delight when I got the Sage flier of a third edition! Of course I ordered it. I also discovered that Saldana (the new third author on the third edition) has written another book on qualitative data that he sites a lot in this third edition (Coding manual for qualitative researchers ) and I ordered that volume as well.

Saldana, in the third edition, talks a lot about data display, one of the three factors that qualitative researchers must keep in mind. The other two are data condensation and conclusion drawing/verification. In their review, Sage Publications says, “The Third Edition’s presentation of the fundamentals of research design and data management is followed by five distinct methods of analysis: exploring, describing, ordering, explaining, and predicting.” These five chapters are the heart of the book (in my thinking); that is not to say that the rest of the book doesn’t have gems as well–it does. The chapter on “Writing About Qualitative Research” and the appendix are two. The appendix (this time) is an “An Annotated Bibliography of Qualitative Research Resources”, which lists at least 32 different classifications of references that would be helpful to all manner of qualitative researchers. Because it is annotated, the bibliography provides a one sentence summary of the substance of the book. A find, to be sure. Check out the third edition.

I will be attending a professional development session with Mr. Saldana next week. It will be a treat to meet him and hear what he has to say about qualitative data. I’m taking the two books with me…I’ll write more on this topic when I return. (I won’t be posting next week).

You implement a program. You think it is effective; that it makes a difference; that it has merit and worth. You develop a survey to determine the merit and worth of the program. You send the survey out to the target audience which is an intact population–that is, all of the participants are in the target audience for the survey. You get less than 4o% response rate. What does that mean? Can you use the results to say that the participants saw merit in the program? Do the results indicate that the program has value; that it made a difference if only 40% let you know what they thought.

I went looking for some insights on non-responses and non-responders. Of course, I turned to Dillman (my go to book for surveys…). His bottom line: “…sending reminders is an integral part of minimizing non-response error” (pg. 360).

Dillman (of course) has a few words of advice. For example, on page 360, he says, ” Actively seek means of using follow-up reminders in order to reduce non-response error.” How do you not burden the target audience with reminders, which are “…the most powerful way of improving response rate…” (Dillman, pg. 360). When reminders are sent they need to be carefully worded and relate to the survey being sent. Reminders stress the importance of the survey and the need for responding.

Dillman also says (on page 361) to “…provide all selected respondents with similar amounts and types of encouragement to respond.” Since most of the time incentives are not an option for you the program person, you have to encourage the participants in other ways. So we are back to reminders again.

To explore the topic of non-response further, there is a book (Groves, Robert M., Don A. Dillman, John Eltinge, and Roderick J. A. Little (eds.). 2002. Survey Nonresponse. Wiley-Interscience: New York) that deals with the topic. I don’t have it on my shelf, so I can’t speak to it. I found it while I was looking for information on this topic.

I also went on line to EVALTALK and found this comment which is relevant to evaluators attempting to determine if the program made a difference: “Ideally you want your non-response percents to be small and relatively even-handed across items. If the number of nonresponds is large enough, it does raise questions as to what is going for that particular item, for example, ambiguous wording or a controversial topic. Or, sometimes a respondent would rather not answer a question than respond negatively to it. What you do with such data depends on issues specific to your individual study.” This comment was from Kathy Race of Race & Associates, Ltd., September 9, 2003.

A bottom line I would draw from all this is respond…if it was important to you to participate in the program then it is important for you to provide feedback to the program implementation team/person.

Miscellaneous thought 1.

Yesterday, I had a conversation with a long time friend of mine. When we stopped and calculated (which we don’t do very often), we realized that we have know each other since 1981. We met at the first AEA (only it wasn’t AEA then) conference in Austin, TX. I was a graduate student; my friend was a practicing professional/academic. Although we were initially talking about other things evaluation; I asked my friend to look at an evaluation form I was developing. I truly believe that having other eyes (a pilot if you will) view the document helps. It certainly did in this case. I feel really good about the form. In the course of the conversation, my friend advocated strongly for a odd numbered scales. My friend had good reasons, specifically

1) It tends to force more comparisons on the respondents; and

2) if you haven’t given me a neutral point I tend to mess up the scale on purpose because you are limiting my ability to tell you what I am thinking.

I, of course, had an opposing view (rule number 8–question authority). I said, ” My personal preference is an even number scale to avoid a mid-point. This is important because I want to know if the framework (of the program in question) I provided worked well with the group and a mid-point would provide the respondent with a neutral point of view, not a working or not working opinion. An even number (in my case four points) can be divided into working and not working halves. When I’m offered a middle point, I tend to circle that because folks really don’t want to know what I’m thinking. By giving me an opt out/neutral/neither for or against option they are not asking my opinion or view point.”

Recently, I came across an aea365 post on just this topic. Although this specific post was talking about Likert scales, it applies to all scaling that uses a range of numbers (as my friend pointed out). The authors sum up their views with this comment, “There isn’t a simple rule regarding when to use odd or even, ultimately that decision should be informed by (a) your survey topic, (b) what you know about your respondents, (c) how you plan to administer the survey, and (d) your purpose. Take time to consider these four elements coupled with the advantages and disadvantages of odd/even, and you will likely reach a decision that works best for you.” (Certainly knowing my friend like I do, I would be suspicious of responses that my friend submitted.) Although they list advantages and disadvantages for odd and even responses, I think there are other advantages and disadvantages that they did not mentioned yet are summed up in their concluding sentence.

Miscellaneous thought 2.

I’m reading the new edition of Qualitative Data Analysis (QDA). This has always been my go to book for QDA and I was very sad when I learned that both of the original authors had died. The new author, Johnny Saldana (who is also the author of The Coding Manual for Qualitative Researchers), talks (in the third person plural, active voice) about being a pragmatic realist. That is an interesting concept. They (because the new author includes the previous authors in his statement) say “that social phenomena exist not only in the mind but also in the world–and that some reasonably stable relationships can be found among the idiosyncratic messiness of life.” Although I had never used those exact words before, I agree. It is nice to know the label that applies to my world view. Life is full of idiosyncratic messiness; probably why I think systems thinking is so important. I’m reading this volume because I’ve been asked to write the review of one of my favorite books. We will see if I can get through it between now and July 1 when the draft of the review is due. Probably aught to pair it with Saldana’s other book; won’t happen between now and July 1.

I have a few thoughts about causation, which I will get to in a bit…first, though, I want to give my answers to the post last week.

I had listed the following and wondered if you thought they were a design, a method, or an approach. (I had also asked which of the 5Cs was being addressed–clarity or consistency.) Here is what I think about the other question.

Case study is a method used when gathering qualitative data, that is, words as opposed to numbers. Bob Stake, Robert Brinkerhoff, Robert Yin, and others have written extensively on this method.

Pretest-post test Control Group is (according to Campbell and Stanley, 1963) an example of a true experimental design if a control group is used (pg. 8 and 13). NOTE: if only one group is used (according to Campbell and Stanley, 1963), pretest-post test is considered a pre-experimental design (pg. 7 and 8); still it is a design.

Ethnography is a method used when gathering qualitative data often used in evaluation by those with training in anthropology. David Fetterman is one such person who has written on this topic.

Interpretive is an adjective use to describe the approach one uses in an inquiry (whether that inquiry is as an evaluator or a researcher) and can be traced back to the sociologists Max Weber and Wilhem Dilthey in the later part of the 19th century.

Naturalistic is an adjective use to describe an approach with a diversity of constructions and is a function of “…what the investigator does…” (Lincoln and Guba, 1985, pg.8).

Random Control Trials (RCT) is the “gold standard” of clinical trials, now being touted as the be all and end all of experimental design; its proponents advocate the use of RCT in all inquiry as it provides the investigator with evidence that X (not Y) caused Z.

Quasi-Experimental is a term used by Campbell and Stanley(1963) to denote a design where random assignment cannot be made for ethical or practical reasons be accomplished; this is often contrasted with random selection for survey purposes.

Qualitative is an adjective to describe an approach (as in qualitative inquiry), a type of data (as in qualitative data) or
methods (as in qualitative methods). I think of qualitative as an approach which includes many methods.

Focus Group is a method of gathering qualitative data through the use of specific, structured interviews in the form of questions; it is also an adjective for defining the type of interviews or the type of study being conducted (Krueger & Casey, 2009, pg. 2)

Needs Assessment is method for determining priorities for the allocation of resources and actions to reduce the gap between the existing and the desired.

I’m sure there are other answers to the terms listed above; these are mine. I’ve gotten one response (from Simon Hearn at BetterEvaluation). If I get others, I’ll aggregate them and share them with you. (Simon can check his answers against this post.

Now causation, and I pose another question: If evaluation (remember the root word here is value) is determining if a program (intervention, policy, product, etc. ) made a difference, and determined the merit or worth (i.e., value) of that program (intervention, policy, product, etc.), how certain are you that your program (intervention, policy, program, etc.) caused the outcome? Chris Lysy and Jane Davidson have developed several cartoons that address this topic. They are worth the time to read them.

When I teach scientific writing (and all evaluators need to be able to communicate clearly verbally and in writing), I focus on the 5Cs: larity, oherence, onciseness, onsistency, and orrectness, I’ve written about the 5Cs in a previous blog post, so I won’t belabor them here. Suffice it to say that when I read a document that violates one (or more) of these 5Cs, I have to wonder.

Recently, I was reading a document where the author used design (first), then method, then approach. In reading the context, I think (not being able to clarify) that the author was referring to the same thing–a method and used these different words in an effort to make the reading more entertaining where all it did was cause obfuscation, violating larity, one of the 5Cs .

So I’ll ask you, reader. Are these different? What makes them different? Should they have been used interchangeably in the document? I went to my favorite thesaurus of evaluation terms (Scriven) (published by Sage) to see what he had to say, if anything. Only “design” was listed and the definition said, “…process of stipulating the investigatory procedures to be followed in doing a certain evaluation…” OK–investigatory procedure.

So, I’m going to list several terms used commonly in evaluation and research. Think about what each is–design, method, approach. I’ll provide my answers next week. Let me know what you think each of the following is:

Case Study

Pretest-Posttest Control Group

Ethnography

Interpretive

Naturalistic

Random Control Trials (RCT)

Quasi-Experimental

Qualitative

Focus Group

Needs Assessment

Evaluation is an Everyday Activity

Program Evaluation Discussions

Category Archives: Methodology

There are 11 winter holidays at least…

Intention to change

Magnitude coding

Qualitative data analysis

AEA and presentations

Response rates and non-responders

Miscellaneous Thoughts

Causation?

One of the 5Cs–Clarity

Contact Info