activities « Evaluation is an Everyday Activity

Several folks read and commented on my previous AEA post. That is heartening. One comment was about what a new person should do at the conference. That is today’s post. I am fortunate because the Graduate Student and New Evaluator Topical Interest Group post in aea365 last week talked about just that. I recommend you check it out.

Although I couldn’t find a “First Time Attendee” session in this year’s on line program, I could find a lot of other sessions which sounded interesting. In the past, this session was offered just before the reception on Wednesday. That was a long time ago (I found reference to it in the 1999 program), however, and AEA (like so many other things) has moved on, grown, and changed. I remember attending several and contributing because I was a long time attendee. They were informative, much like the aea365 blog post.

The one thing that the aea365 post didn’t mention that I think is important is children. AEA is, and always has been, family friendly. Children are welcome (by most; there are some curmudgeons, to be sure). I am a single parent by choice. I built my family through adoption. From the time my oldest daughter (now 20) came home, I took her and then she and her sister (now 17) to AEA; we went as a family until 2007 when AEA was in Baltimore. By the time they got into high school, it was harder to take them. (I did take Mersedes last year to Minneapolis; she isn’t coming to DC this year; this being her high school senior year…) They developed their own cohort of friends, people who still ask after them today when I go alone.

How did taking them actually work? Most of the hotel venues used by AEA can recommend a sitting service, one that has been used by their guests. It is worth the cost. When I was President, the service was a life saver. (I used White House Nannies in DC) I paid an hourly rate when I needed coverage, which was about 30% of the time. The rest of the time, the girls attended sessions, took their coloring books and sundries to the back of the room and played quietly while I did my professional thing. There were multiple benefits in this arrangement: They got to see Mom “working”; they got to see parts of DC they wouldn’t normally see (arrangements were made at the time of booking), they learned behaviors of a professional meeting, saw what was expected, and learned to talk to grown-ups. They love to tell the story of sitting under the table in the back of the room, though I don’t know how many years they did that…I’m guessing a lot. Some of my best friends are also their friends with whom they maintain relationships. If you have a partner, bringing children is easier. Some of my friends took their son/daughter and traded off responsibilities; their children also created a whole cadre of connections, some of which last to this day. Being in DC is a wonderful opportunity for a young family; DC is an amazing city with lots to see and do. I strongly urge you to take a day (or two) and enjoy the city and the conference with your family.

The American Evaluation Association has opened its registration for the 2013 meeting in Washington DC. This meeting promises to be attended by the most people yet. Eleven years ago we were in D. C. and broke all attendance records to date. I remember because that was my presidential year…the year that the evaluation profession started thinking that evaluation was a system; that everything we do is connected. Several people have commented about AEA–that they didn’t know there was such an association; that they didn’t know about the conference; that they weren’t members. So folks, here is the skinny on AEA (at least part of the skinny…).

The American Evaluation Association was officially founded in 1986 as a combined organization of the Evaluation Research Society and Evaluation Network. ERS was academic and EN was practitioner; merging the two was a challenge as each thought something would be lost. This is a good example of where the whole is greater that the sum of its parts. The differences were pronounced and debatable (now you only see AEA). Robert (Bob) B. Ingle was the force behind the conference; he mounted the first EN/ERS conference in 1981 in Austin, Texas. I was a graduate student. I was in awe. Although I had been to numerous professional conferences before attending this first conference, I had never met any one like Bob Ingle. His first comment to me once we connected after playing phone tag was, “You spell your name wrong!” (Turns out he was the Scotland branch of the German house of Engel; my ancestors changed the spelling when they came out of Germany.) I was a nascent graduate student in love with my studies and here comes this brusque, acerbic, and outrageous giant. He became my good friend–I knew him from 1981 until he died in 1998. He believed passionately in program evaluation. I think he is smiling at the growth in the profession and the organization. He knew a lot of us; he saw the association through the good times and the bad times. I could end here and say, the rest is history…only there is so much to tell. The association went from an all volunteer organization at its founding in 1986 to an organization of over 8,000 members run by an association management firm. Susan Kistler (of Kistler Associates) was our executive director for the last 15 years. (The association has transitioned to a new management firm [SmithBucklin] and a new executive director [Denise Roosendaal]). Seeing the association transition is bittersweet; growth is good, the loss of family feeling is sad. The association is no longer feels intimate, family; it offers so much more to folks who are members.

David Bernstein is the co-chair of the local arrangements working group (LAWG) for this year’s conference. He lead off a week of AEA365 talking about the conference. Read this post. It tells you a lot about the conference. This week AEA365 is being written by the local arrangements working group. The role of the local arrangements group is to make sure the folks who attend the conference have a good time, both at the conference and in DC. DC is a wonderful city. You cannot see it in a week; it is always changing. Take a day if you have never been to see the city’s high points. It is the nation’s capitol, after all, and there are many high points.

The members only AEA August newsletter also talks about registration with hyperlinks to the registration site, the conference program, and hotel accommodations. (The members only newsletter is just one reason to join AEA.) I’ve been going to AEA since 1981. This is the first year I will not have a paper/poster/etc. on the program. (I am doing a professional development session with Jim Altschuld, though; it is number 22).

Each year I attend AEA, I think of the three evaluative criteria that FOR ME makes a good conference: See three long time friends; meet three new people who could become friends; and get three new ideas. If I do all this, I usually come home energized. I hope to see you there.

You implement a program. You think it is effective; that it makes a difference; that it has merit and worth. You develop a survey to determine the merit and worth of the program. You send the survey out to the target audience which is an intact population–that is, all of the participants are in the target audience for the survey. You get less than 4o% response rate. What does that mean? Can you use the results to say that the participants saw merit in the program? Do the results indicate that the program has value; that it made a difference if only 40% let you know what they thought.

I went looking for some insights on non-responses and non-responders. Of course, I turned to Dillman (my go to book for surveys…). His bottom line: “…sending reminders is an integral part of minimizing non-response error” (pg. 360).

Dillman (of course) has a few words of advice. For example, on page 360, he says, ” Actively seek means of using follow-up reminders in order to reduce non-response error.” How do you not burden the target audience with reminders, which are “…the most powerful way of improving response rate…” (Dillman, pg. 360). When reminders are sent they need to be carefully worded and relate to the survey being sent. Reminders stress the importance of the survey and the need for responding.

Dillman also says (on page 361) to “…provide all selected respondents with similar amounts and types of encouragement to respond.” Since most of the time incentives are not an option for you the program person, you have to encourage the participants in other ways. So we are back to reminders again.

To explore the topic of non-response further, there is a book (Groves, Robert M., Don A. Dillman, John Eltinge, and Roderick J. A. Little (eds.). 2002. Survey Nonresponse. Wiley-Interscience: New York) that deals with the topic. I don’t have it on my shelf, so I can’t speak to it. I found it while I was looking for information on this topic.

I also went on line to EVALTALK and found this comment which is relevant to evaluators attempting to determine if the program made a difference: “Ideally you want your non-response percents to be small and relatively even-handed across items. If the number of nonresponds is large enough, it does raise questions as to what is going for that particular item, for example, ambiguous wording or a controversial topic. Or, sometimes a respondent would rather not answer a question than respond negatively to it. What you do with such data depends on issues specific to your individual study.” This comment was from Kathy Race of Race & Associates, Ltd., September 9, 2003.

A bottom line I would draw from all this is respond…if it was important to you to participate in the program then it is important for you to provide feedback to the program implementation team/person.

I have a few thoughts about causation, which I will get to in a bit…first, though, I want to give my answers to the post last week.

I had listed the following and wondered if you thought they were a design, a method, or an approach. (I had also asked which of the 5Cs was being addressed–clarity or consistency.) Here is what I think about the other question.

Case study is a method used when gathering qualitative data, that is, words as opposed to numbers. Bob Stake, Robert Brinkerhoff, Robert Yin, and others have written extensively on this method.

Pretest-post test Control Group is (according to Campbell and Stanley, 1963) an example of a true experimental design if a control group is used (pg. 8 and 13). NOTE: if only one group is used (according to Campbell and Stanley, 1963), pretest-post test is considered a pre-experimental design (pg. 7 and 8); still it is a design.

Ethnography is a method used when gathering qualitative data often used in evaluation by those with training in anthropology. David Fetterman is one such person who has written on this topic.

Interpretive is an adjective use to describe the approach one uses in an inquiry (whether that inquiry is as an evaluator or a researcher) and can be traced back to the sociologists Max Weber and Wilhem Dilthey in the later part of the 19th century.

Naturalistic is an adjective use to describe an approach with a diversity of constructions and is a function of “…what the investigator does…” (Lincoln and Guba, 1985, pg.8).

Random Control Trials (RCT) is the “gold standard” of clinical trials, now being touted as the be all and end all of experimental design; its proponents advocate the use of RCT in all inquiry as it provides the investigator with evidence that X (not Y) caused Z.

Quasi-Experimental is a term used by Campbell and Stanley(1963) to denote a design where random assignment cannot be made for ethical or practical reasons be accomplished; this is often contrasted with random selection for survey purposes.

Qualitative is an adjective to describe an approach (as in qualitative inquiry), a type of data (as in qualitative data) or
methods (as in qualitative methods). I think of qualitative as an approach which includes many methods.

Focus Group is a method of gathering qualitative data through the use of specific, structured interviews in the form of questions; it is also an adjective for defining the type of interviews or the type of study being conducted (Krueger & Casey, 2009, pg. 2)

Needs Assessment is method for determining priorities for the allocation of resources and actions to reduce the gap between the existing and the desired.

I’m sure there are other answers to the terms listed above; these are mine. I’ve gotten one response (from Simon Hearn at BetterEvaluation). If I get others, I’ll aggregate them and share them with you. (Simon can check his answers against this post.

Now causation, and I pose another question: If evaluation (remember the root word here is value) is determining if a program (intervention, policy, product, etc. ) made a difference, and determined the merit or worth (i.e., value) of that program (intervention, policy, product, etc.), how certain are you that your program (intervention, policy, program, etc.) caused the outcome? Chris Lysy and Jane Davidson have developed several cartoons that address this topic. They are worth the time to read them.

Recently, I was privileged to see the recommendations of William (Bill) Tierney on the top education blogs. (Tierney is the Co-director of the Pullias Center for Higher Education at the University of Southern California.) He (among others) writes the blog, 21st scholar. The blogs are actually the recommendation of his research assistant Daniel Almeida. These are the recommendations:

What criteria were used? What criteria would you use? Some criteria that come to mind are interest, readability, length, frequency. But I’m assuming that they would be your criteria (and you know what assuming does…)

If I’ve learned anything in my years as an evaluator, it is to make assumptions explicit. Everyone comes to the table with built in biases (called cognitive biases). I call them personal and situational biases (I did my dissertation on those biases). So by making your assumptions explicit (and thereby avoiding personal and situational biases), you are building a rubric because a rubric is developed from criteria for a particular product, program, policy, etc.

How would you build your rubric? Many rubrics are in chart format, that is columns and rows with the criteria detailed in those cross boxes. That isn’t cast in stone. Given the different ways people view the world–linear, circular, webbed–there may be others, I would set yours up in the format that works best for you. The only thing to keep in mind is be specific.

Now, perhaps you are wondering how this relates to evaluation in the way I’ve been using evaluation. Keep in mind evaluation is an everyday activity. And everyday, all day, you perform evaluations. Rubrics formalizes the evaluations you conduct–by making the criteria explicit. Sometimes you internalize them; sometimes you write them down. If you need to remember what you did the last time you were in a similar situation, I would suggest you write them down. No, you won’t end up with lots of little sticky notes posted all over. Use your computer. Create a file. Develop criteria that are important to you. Typically, the criteria are in a table format; an x by x form. If you are assigning number, you might want to have the rows be the numbers (for example, 1-10) and the columns be words that describe those numbers (for example, 1 boring; 10 stimulating and engaging). Rubrics are used in reviewing manuscripts, student papers, assigning grades to activities as well as programs. Your format might look like this:

Or it might not. What other configuration have you seen rubrics? How would you develop your rubric? Or would you–perhaps you prefer a bunch of sticky notes. Let me know.

Recently, I came across a blog post by Daniel Green, who is the head of strategic media partnerships at the Bill and Melinda Gates Foundation. He coauthored this post with Mayur Patel, vice president of strategy and assessment at the Knight Foundation. I mention this because those two foundations have contributed $3.25 million in seed funding “…to advance a better understanding of audience engagement and media impact…”. They are undertaking an ambitious project to develop a rubric (of sorts) to determine “…how media influences the ways people think and act, and contributes to broader societal changes…”. Although it doesn’t specifically say, I include social media in the broad use of “media”. The blog post talks about broader agenda–that of informed and engaged communities. These foundations believe that an informed and engaged communities will strengthen “… democracy and civil society to helping address some of the world’s most challenging social problems.”

Or in other words, what difference is being made, which is something I wonder about all the time. (I’m an evaluator, after all, and I want to know what difference is made.)

Although there are strong media forces out there (NYTimes, NPR, BBC, the Guardian, among others), I wonder about the strength and effect of social media (FB, Twitter, LinkedIn, blogs, among others). Anecdotally, I can tell you that social media is everywhere and IS changing the way people think and act. I watch my now 17 y/o who uses the IM feature on her social media to communicate with her friends, set up study dates, find out homework assignments, not the phone like I did. I watch my now 20 y/o multitask–talk to me on Skype and read and respond to her FB entry. She uses IM as much as her sister. I know that social media was instrumental in the Arab spring. I know that major institutions have social media connections (FB, Twitter, LinkedIn, etc.). Social media is everywhere. And we have no good way to determine if it is making a difference and what that difference is.

For something so ubiquitous (social media), why is there no way to evaluate social media other than through the use of analytics? I’ve been asking that question since I first posted my query “Is this blog making a difference?” back in March 2012. Since I’ve been posting since December 2009, that gave me over 2 years from which to gather data. That is a luxury when it comes to programming, especially when many programs often are a few hours in duration and an evaluation is expected.

I hope that this project provides useful information for those of us who have come kicking and screaming to social media and have seen the light. Even though they are talking about the world of media, I’m hoping that they can come up with measures that address the social aspect of media. The technology provided IS useful; the question is what difference is it making?

A rubric is a way to make criteria (or standards) explicit and it does that in writing so that there can be no misunderstanding. It is found in many evaluative activities especially assessment of classroom work. (Misunderstanding is still possible because the English language is often not clear–something I won’t get into today; suffice it to say that a wise woman said words are important–keep that in mind when crafting a rubric.)

This week there were many events that required rubrics. Rubrics may have been implicit; they certainly were not explicit. Explicit rubrics were needed.

I’ll start with apologies for the political nature of today’s post.

Yesterday’s activity of the US Senate is an example where a rubric would be valuable. Gabby Giffords said it best:

Certainly, an implicit rubric for this event can be found in this statement:

Only it was not used. When there are clear examples of inappropriate behavior; behavior that my daughters’ kindergarten teacher said was mean and not nice, a rubric exists. Simple rubrics are understood by five year olds (was that behavioir mean OR was that behavior nice). Obviously 46 senators could only hear the NRA; they didn’t hear that the behavior (school shootings) was mean.

Boston provided us with another example of the mean vs. nice rubric. Bernstein got the concept of mean vs. nice.

Music is nice; violence is mean.

Helpers are nice; bullying is mean.

There were lots of rubrics, however implicit, for that event. The NY Times reported that helpers (my word) ran TOWARD those in need not away from the site of the explosion (violence). There were many helpers. A rubric existed, however implicit.

I want to close with another example of a rubric:

I’m no longer worked up–just determined and for that I need a rubric. This image may not give me the answer; it does however give me pause.

For more information on assessment and rubrics see: Walvoord, B. E. (2004). Assessment clear and simple. San Francisco: Jossey-Bass.

Harold Jarche shared in his blog a comment by a participant in one of his presentations. The comment is:

Knowledge is evolving faster than can be codified in formal systems and is depreciating in value over time.

This is really important for those of us who love the printed work (me) and teach (me and you). A statement like this tells us that we are out of date the moment we open our mouths; those institutions on which we depended for information (schools, libraries, even churches) are now passe.

The exponential growth of knowledge is much like that of population. I think this graphic image of population (by Waldir) is pretty telling (click on the image to read the fine print). The evaluative point that this brings home to me is the delay in making information available.

Do you (like me) when you say, “Look it up”, think web, not press, books, library, hard copy? Do you (like me) wonder how and where this information originated when the information is so cutting edge? Do you (like me) wonder how to keep up or even if you can? Books take over a year to come to fruition (I think the 2 year frame is more representative). Journal manuscripts take 6 to 9 months on a quick journal turn around. Blogs are faster and they express opinion; could they be a source of information?

I’ve decided to go to an advanced qualitative data seminar this summer as part of my professional development because I’m using more and more qualitative data (I still use quantitative data, too). It is supposed to be cutting edge. The book on which the seminar is based won’t be published until next month (April). How much information has been developed since that book went to press? How much information will be shared at the seminar? Or will that seminar be old news (and like old news, be ready for fish)? The explosion of information like the explosion of population, may be a good thing; or not. The question is what is being done with that knowledge? How is it being used? Or is it? Is the knowledge explosion an excuse for people to be information illiterate? To become focused (read narrow) in their field? What are you doing with what I would call miscellaneous information that is gathered unsystematically? What are you doing with information now–how are you using it for professional development–or are you?

One of the outcomes of learning about evaluation is informational literacy.

Think about it. How does what is happening in the world affect your program? Your outcomes? Your goals?

When was the last time you applied that peripheral knowledge to what you are doing. Informational literacy is being aware of what is happening in the world. Knowing this information, even peripherally, adds to your evaluation capacity.

Now, this is not advocating that you need to read the NY Times daily (although I’m sure they would really like to increase their readership); rather it is advocating that you recognize that none of your programs (whether little p or big P) occur in isolation. What your participants know affects how the program is implemented. What you know affects how the programs are planned. That knowledge also affects the data collection, data analysis, and reporting. This is especially true for programs developed and delivered in the community, as are Extension programs.

Let me give you a real life example. I returned from Tucson, AZ and the capstone event for an evaluation capacity program I was leading. The event was an outstanding success–not only did it identify what was learned and what needed to be learned, it also demonstrated the value of peer learning. I was psyched. I was energized. I was in an automobile accident 24 hours after returning home. (The car was totaled–I no longer have a car; my youngest daughter and I experienced no serious injuries.) The accident was published in the local paper the following day. Several people saw the announcement; those same several people expressed their concern; some of those several people asked how they could help. Now this is a very small local event that had a serious effect on me and my work. (If I hadn’t had last week’s post already written, I don’t know if I could have written it.) Solving simple problems takes twice as long (at least). This informational literacy influenced those around me. Their knowing changed their behavior to me. Think of what September 11, 2001 did to people’s behavior; think about what the Pope’s RESIGNATION is doing to people’s behavior. Informational literacy. It is all evaluative. Think about it.

Graphic URL: http://www.otterbein.edu/resources/library/information_literacy/index.htm

On January 22, 23, and 24, a group of would be evaluators will gather in Tucson, AZ at the Westin La Paloma Resort.

Even though Oregon State is a co-sponsor for this program, being in Oregon in winter (i.e., now) is not the land of sunshine, and since Vitamin D is critical for everyone’s well being, I chose Tucson for our capstone event. Our able support person, Gretchen, chose the La Paloma, a wonderful site on the north side of Tucson. So even if it is not warm, it will be sunny. Why, we might even get to go swimming; if not swimming, certainly hiking. There are a lot of places to hike around Tucson…in Sabino Canyon ; near/around A Mountain (first year U of A students get to whitewash or paint the A) ; Saguaro National Park ; or maybe in one of the five (yes, five) mountain ranges surrounding Tucson. (If you are interested in other hikes, look here.)

We will be meeting Tuesday afternoon, all day Wednesday, and Thursday morning. Participants have spent the past 17 months participating in and learning about evaluation. They have identified a project/program (either big P or little p), and they participated in a series of modules, webinars, and office hours on topics used everyday in evaluating a project or program.We anticipate over 20 attendees from the cohorts. We have participants from five Extension program areas (Nutrition, Agriculture, Natural Resources, Family and Community Science, and 4-H), from ten western states (Oregon, Washington, California, Utah, Colorado, Idaho, New Mexico, Arizona, Wyoming, and Hawaii.), and all levels of familiarity with evaluation (beginner to expert).

I’m the evaluation specialist in charge of the program content (big P) and Jim Lindstrom (formerly of Washington State, currently University of Idaho) has been the professional development and technical specialist, and Gretchen Cuevas (OSU) has been our wonderful support person. I’m using Patton’s Developmental Evaluation Model to evaluate this program. Although some things were set at the beginning of the program (the topics for the modules and webinars, for example), other things were changed depending on feedback (readings, office hours). Although we expect that participants will grow their knowledge of evaluation, we do not know what specific and measurable outcomes will result (hence, developmental). We hope to run the program (available to Extension faculty in the Western Region) again in September 2013. Our goal is to build evaluation capacity in the Western Extension Region. Did we?

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: activities

More AEA

AEA (American Evaluation Association)

Response rates and non-responders

Causation?

More on rubrics and evaluation

Free Technology for Teachers

MindShift

Joanne Jacobs

Teaching Tolerance

Brian McCall’s Economics of Education Blog

Making a difference

What is a rubric?

Professional development and knowledge

Informational literacy

Capacity building

Contact Info