value « Evaluation is an Everyday Activity

We just celebrated Thanksgiving , a time in the US when citizens pause and reflect on those things for which we are thankful. Often those things for which we are thankful are based in our values–things like education, voting, religion/belief systems, honesty, truth, peace. In thinking about those things, I was reminded that the root word of evaluation is value…I thought this would be a good time to share AEA’s value statement.

Are you familiar with AEA’s values statement? What do these values mean to you?

AEA’s Values Statement

The American Evaluation Association values excellence in evaluation practice, utilization of evaluation findings, and inclusion and diversity in the evaluation community.

i. We value high quality, ethically defensible, culturally responsive evaluation practices that lead to effective and humane organizations and ultimately to the enhancement of the public good.

ii. We value high quality, ethically defensible, culturally responsive evaluation practices that contribute to decision-making processes, program improvement, and policy formulation.

iii. We value a global and international evaluation community and understanding of evaluation practices.

iv. We value the continual development of evaluation professionals and the development of evaluators from under-represented groups.

v. We value inclusiveness and diversity, welcoming members at any point in their career, from any context, and representing a range of thought and approaches.

vi. We value efficient, effective, responsive, transparent, and socially responsible association operations.

See AEA’s Mission, Vision, Values

Values enter into all aspects of evaluation–planning, implementing, analyzing, reporting, and use. Values are all around us. Have you taken a good look at your values lately. Review is always beneficial, informative, and insightful. I encourage it.

I spent much of the last week thinking about what I would write on November 7, 2012.

Would I know anything before I went to bed? Would I like what I knew? Would I breathe a sigh of relief?

Yes, yes, and yes, thankfully. We are one nation and one people and the results of yesterday demonstrate that we are also evaluators.

Yesterday is a good example that everyday we evaluate. (What is the root of the word evaluation?) We review a program (in this case the candidates); we determine the value (what they say they believe); we develop a rubric (criteria); we support those values and that criteria; and we apply those criteria (vote). Yesterday over 117 million people did just that. Being a good evaluator I can’t just talk about the respondents without talking about the total population–the total number of possible respondents. One guess estimates that 169 million people are registered to vote – 86 million Democrat – 55 million Republican – 28 million others registered. The total response rate for this evaluation was 69.2%. Very impressive–especially given the long lines. (Something the President said that needed fixing [I guess he is an evaluator, too.])

I am reminded that Senators and Representatives are elected to represent the voice of the people. Their job is to represent you. If they do not fulfill that responsibility, it is our responsibility to do something about it. If you don’t hold them accountable, you can’t complain about the outcome. Another evaluative activity. (Did I ever tell you that evaluation is a political activity…?) Our job as evaluators doesn’t stop when we cast our ballot; our job continues throughout the life of the program (in this case, the term in office). Our job is to use those evaluation results to make things better. Often, use is ignored. Often, the follow-through is missing. As evaluators, we need to come full circle.

Evaluation is an everyday activity.

As with a lot of folks who are posting to Eval Central, I got back Monday from the TCs and AEA’s annual conference, Evaluation ’12. I

I’ve been going to this conference since 1981 when Bob Ingle decided that the Evaluation Research Society and Evaluation Network needed to pool its resources and have one conference, Evaluation ’81. I was a graduate student. That conference changed my life. This was my professional home. I loved going and being there. I was energized; excited; delighted by what I learned, saw, and did.

Reflecting back over the 30+ years and all that has happened has provided me with insights and new awarenesses. This year was a bittersweet experience for me, for may reasons–not the least of them being Susan Kistler’s resignation from her role as AEA Executive Director. I remember meeting Susan and her daughter Emily in Chicago when Susan was in graduate school and Emily was three. Susan has helped make AEA what it is today. I will miss seeing her at the annual meeting. Because she lives on the east coast, I will rarely see her in person, now. There are fewer and fewer long time colleagues and friends at this meeting. And even though a very wise woman said to me, “Make younger friends”. Making younger friends isn’t easy when you are an old person (aka OWG) like me and see these new folks only once a year.

I will probably continue going until my youngest daughter, now a junior in high school, finishes college. What I bring home is less this year than last; and less last year than the year before. It is the people, certainly. I also find that the content challenges me less and less. Not that the sessions are not interesting or well presented–they are. I’m just not excited; not energized when I get back to the office. To me a conference is a “good” conference (ever the evaluator) if I met three new people with whom I wanted to maintain contact; spent time with three long time friends/colleagues; and brought home three new ideas. This year, not three new people; yes three long time friends; only one new idea. 4/9. I was delighted to hear that the younger folks were closer to the 9/9. Maybe I’m jaded.

The professional development session I attended (From Metaphor to Model) provided me with a visual for conceptualizing a complex program I’ll be evaluating. The plenary I attended with Oren Hesterman from the Fair Food Network in Detroit demonstrated how evaluative tools and good questions support food sustainability. What I found interesting was that during the question/comment session following the plenary, all the questions/comments were about food sustainability, NOT evaluation, even though Ricardo Millett asked really targeted evaluative questions. Food sustainability seems to be a really important topic–talk about a complex messy system. I also attended a couple of other sessions that really stood out and some that didn’t. Is attending this meeting important, even in my jaded view? Yes. It is how evaluators grow and change; even when the change is not the goal. Yes. The only constant is change. AEA provides professional development, in it pre and post sesssions as well as plenary and concurrent sessions. Evaluators need that.

What is the difference between need to know and nice to know? How does this affect evaluation? I got a post this week on a blog I follow (Kirkpatrick) that talks about how much data does a trainer really need? (Remember that Don Kirkpatrick developed and established an evaluation model for professional training back in the 1954 that still holds today.)

Most Extension faculty don’t do training programs per se, although there are training elements in Extension programs. Extension faculty are typically looking for program impacts in their program evaluations. Program improvement evaluations, although necessary, are not sufficient. Yes, they provide important information to the program planner; they don’t necessarily give you information about how effective your program has been (i.e., outcome information). (You will note that I will use the term “impacts” interchangeably with “outcomes” because most Extension faculty parrot the language of reporting impacts.)

OK. So how much data do you really need? How do you determine what is nice to have and what is necessary (need) to have? How do you know?

Look at your logic model. Do you have questions that reflect what you expect to have happen as a result of your program?
Review your goals. Review your stated goals, not the goals you think will happen because you “know you have a good program”.
Ask yourself, How will I USE these data? If the data will not be used to defend your program, you don’t need it.
Does the question describe your target audience? Although not demonstrating impact, knowing what your target audience looks like is important. Journal articles and professional presentations want to know this.
Finally, ask yourself, Do I really need to know the answer to this question or will it burden the participant. If it is a burden, your participants will tend to not answer, then you have a low response rate; not something you want.

Kirkpatrick also advises to avoid redundant questions. That means questions asked in a number of ways and giving you the same answer; questions written in positive and negative forms. The other question that I always include because it will give me a way to determine how my program is making a difference is a question on intention including a time frame. For example, “In the next six months do you intend to try any of the skills you learned to day? If so, which one.” Mazmaniam has identified the best predictor of behavior change (a measure of making a difference) is stated intention to change. Telling someone else makes the participant accountable. That seems to make the difference.

Reference:

Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., & Kantrowits, M. P. (1998). Information about barriers to planned change: A Randomized controlled trail involving continuing medical education lectures and commitment to change. Academic Medicine, 73(8).

P.S. No blog next week; away on business.

Creativity is not an escape from disciplined thinking. It is an escape with disciplined thinking.” – Jerry Hirschberg – via @BarbaraOrmsby

The above quote was in the September 7 post of Harold Jarche’s blog. I think it has relevance to the work we do as evaluators. Certainly, there is a creative part to evaluation; certainly there is a disciplined thinking part to evaluation. Remembering that is sometimes a challenge.

So where in the process do we see creativity and where do we see disciplined thinking?

When evaluators construct a logic model, you see creativity; you also see disciplined thinking

When evaluators develop an implementation plan, you see creativity; you also see disciplined thinking.

When evaluators develop a methodology and a method, you see creativity; you also see disciplined thinking.

When evaluators present the findings for use, you see creativity; you also see disciplined thinking.

So the next time you say “give me a survey for this program”, think–Is a survey the best approach to determining if this program is effective; will it really answer my questions?

Creativity and disciplined thinking are companions in evaluation.

Evaluation costs: A few weeks ago, I posted a summary about evaluation costs. A recent AEA LinkedIn discussion was on the same topic (see this link). If you have not linked to other evaluators, there are other groups besides AEA that have LinkedIn groups. You might want to join one that is relevant.

New topic: The video on surveys posted last week generated a flurry of comments (though not on this blog). I think it is probably appropriate to revisit the topic of surveys. As I decided to revisit this topic, an AEA 365 post from the Wilder Research group talked about data coding related to longitudinal data.

Now, many surveys, especially Extension surveys, focus on cross sectional data not on longitudinal data. They may, however, involve a large number of participants and the hot tips that are provided apply to coding surveys. Whether the surveys Extension professionals develop involve 30, 300, or 3000 participants, these tips are important especially if the participants are divided into groups on some variable. Although the hot tips in the Wilder post talk about coding, not surveys specifically, they are relevant to surveys and I’m repeating them here. (I’ve also adapted the original tip to Extension use).

Anticipate different groups. If you do this ahead of time, and write it down in a data dictionary or coding guide, your coding will be easier. If the raw data are dropped, or for some other reason scrambled (like a flood, hurricane, or a sleepy night), you will be able to make sense out of the data quicker.
Sometimes there are preexisting identifying information (like location of the program) that have a logical code. Use that code.
Precoding by the location sites helps keep the raw data organized and enables coding.

Over the rest of the year, I’ll be revisiting survey on a regular basis. Survey is often used by Extension. Developing a survey that provides you with information you want, can use, and makes sense is a useful goal.

New topic: I’m thinking of varying the format of the blog or offering alternative formats with evaluation information. I’m curious as to what would help you do your work better. Below are a few options. Let me know what you’d like.

Videos in blogs
Short concise (i.e., 10-15 minute) webinars
Guest writers/speakers/etc.
Other ideas

AEA hosts an online, free, and open to anyone list serve called EVALTALK, managed by the University of Alabama. This week, Felix Herzog posted the following question.

How much can/should an evaluation cost?

This is a question I often get asked, especially by Extension faculty. This question is especially relevant because more Extension faculty are responding to request for proposals (RFP) or request for contract (RFC) that call for an evaluation. The questions arise about how does one budget for the evaluation. Felix compiled what he discovered and I’ve listed it below. It is important to note that this is not just the evaluator’s salary, rather all expenses which relate to evaluating the program–data entry, data management, data analysis, report writing, as well as the evaluator’s salary, data collection instrument development, pilot testing, and the salaries of those who do the above mentioned tasks. Felix thoughtfully provided references with sources so that you can read them. He did note that the most useful citation (the Reider, 2011) is in German.

– The benefit of the evaluation should be at least as high as its cost (Rieder, 2011)

– “Rule of thumb”: 1 – 10% of the costs for a policy program (personnal communication from an administrator)

– 5 – 7 % of a program (Kellog Foundation, p. 54)

– 1 – 15 %of the total cost of a program (Rieder, 2011, 5 quotes in Table 5 p. 82)

– 0.5% of a program (EC, 2004, p. 32 ff.)

– Up to 10% of a program (EC 2008, p. 47 ff.)

REFERENCES

EC (2004) Evaluating EU activities and practical guide for the Commission services. Bruxelles. http://ec.europa.eu/dgs/secretariat_general/evaluation/docs/eval_activities_en.pdf.

EC (2008) EVALSED: The resource for the evaluation of socioeconomic development. Bruxelles. http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evalsed/downloads/guide2008_evalsed.pdf .

Kellogg Foundation. (1984). W. K. Kellogg Foundation Evaluation Handbook. <http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CE8QFjAA&url=http%3A%2F%2Fwww.wkkf.org%2F%7E%2Fmedia%2F62EF77BD5792454B807085B1AD044FE7.ashx&ei=gUsAUIOBMIXaqgG6sdiqBw&usg=AFQjCNHorPDftVA4z54-Kx_d8WZT0p5eQg&sig2=0nQwUeUHCR0_4wuaGdwnBw>.

Rieder S. (2011) Kosten von Evaluationen. LEGES 2011(1), 73 – 88.

Recognizing the value of your evaluation work, being able to put a dollar value to that work, and being able to communicate it helps build organizational capacity in evaluation.

Bright ideas are often the result of “Aha” moments. Aha moments are “The sudden understanding or grasp of a concept…an event that is typically rewarding and pleasurable. Usually, the insights remain in our memory as lasting impressions.” —Rick Nauert PhD Senior News Editor for Psych Central.

How often have you had an “A-ha” moment when you are evaluating? A colleague had one, maybe several, that made an impression on her. Talk about building capacity–this did. She has agreed to share that experience, soon (the bright idea).

Not only did it make an impression on her, her telling me made an impression on me. I am once again reminded of how much I take evaluation for granted. Because evaluation is an everyday activity, I often assume that people know what I’m talking about. We all know what happens when we assume something……. I am also reminded how many people don’t know what I consider basic evaluation information, like constructing a survey item (Got Dillman on your shelf, yet?).

What is this symbol called? No, it is not the square root sign–although that is its function. “It’s called a radical…because it gets at the root…the definition of radical is: of or going to the root or origin.”–Guy McPherson

How radical are you? How does that relate to evaluation, you wonder? Telling truth to power is a radical concept (the definition here is departure from the usual or traditional); one to which evaluators who hold integrity sacrosanct adhere. (It is the third AEA guiding principle.) Evaluators often, if they are doing their job right, have to speak truth to power–because the program wasn’t effective, or it resulted in something different than what was planned, or it cost too much to replicate, or it just didn’t work out . Funders, supervisors, program leaders need to know the truth as you found it.

“Those who seek to isolate will become isolated themselves.” —Diederick Stoel This sage piece of advice is the lead for Jim Kirkpatrick’s quick tip for evaluating training activities. He says, “Attempting to isolate the impact of the formal training class at the start of the initiative is basically discounting and disrespecting the contributions of other factors…Instead of seeking to isolate the impact of your training, gather data on all of the factors that contributed to the success of the initiative, and give credit where credit is due. This way, your role is not simply to deliver training, but to create and orchestrate organizational success. This makes you a strategic business partner who contributes to your organization’s competitive advantage and is therefore indispensable.” Extension faculty conduct a lot of trainings and want to take credit for the training effectiveness. It is important to recognize that there may be other factors at work–mitigating factors; intermediate factors; even confounding factors. As much as Extension faculty want to isolate (i.e., take credit), it is important to share the credit.

Yesterday was the 236th anniversary of the US independence from England (and George III, in his infinite wisdom, is said to have said nothing important happened…right…oh, all right, how WOULD he have known anything had happened several thousand miles away?). And yes, I saw fireworks. More importantly, though, I thought a lot about what does independence mean? And then, because I’m posting here, what does independence mean for evaluation and evaluators?

In thinking about independence, I am reminded about intercultural communication and the contrast between individualism and collectivism. To make this distinction clear, think “I- centered” vs. “We-centered”. Think western Europe, US vs. Asia, Japan. To me, individualism is reflective of independence and collectivism is reflective of networks, systems if you will. When we talk about independence, the words “freedom” and “separate” and “unattached” are bandied about and that certainly applies to the anniversary celebrated yesterday. Yet, when I contrast it with collectivism and think of the words that are often used in that context (“interdependence”, “group”, “collaboration”), I become aware of other concepts.

Like, what is missing when we are independent? What have we lost being independent? What are we avoiding by being independent? Think “Little Red Hen”. And conversely, what have we gained by being collective, by collaborating, by connecting? Think “Spock and Good of the Many”.

There is in AEA a topical interest group of “Independent Consulting”. This TIG is home to those evaluators who function outside of an institution and who have made their own organization; who work independently, on contract. In their mission statement, they pro port to “Foster a community of independent evaluators…” So by being separate, are they missing community and need to foster that aspect? They insist that they are “…great at networking”, which doesn’t sound very independent; it sounds almost collective. A small example, and probably not the best.

I think about the way the western world is today; other than your children and/or spouse/significant other are you connected to a community? a network? a group? not just in membership (like at church or club); really connected (like in extended family–whether of the heart or of the blood)? Although the Independent Consulting TIG says they are great at networking and some even work in groups, are they connected? (Social media doesn’t count.) Is the “I” identity a product of being independent? It certainly is a characteristic of individualism. Can you measure the value, merit, worth of the work you do by the level of independence you possess? Do internal evaluators garner all the benefits of being connected. (As an internal evaluator, I’m pretty independent, even though there is a critical mass of evaluators where I work.)

Although being an independent evaluator has its benefits–less bias, different perspective (do I dare say, more objective?), is the distance created, the competition for position, the risk taking worth the lack of relational harmony that can accompany relationships? Is the US better off as its own country? I’d say probably. My musings only…what do you think?

Matt Keene, AEAs thought leader for June 2012 says, “Wisdom, rooted in knowledge of thyself, is a prerequisite of good judgment. Everybody who’s anybody says so – Philo Judaeus,

Socrates, Lao-tse,

Plotinus, Paracelsus,

Swami Ramdas, and Hobbs.

I want to focus on the “wisdom is a prerequisite of good judgement” and talk about how that relates to evaluation. I also liked the list of “everybody who’s anybody.” (Although I don’t know who Matt means by Hobbs–is that Hobbes or the English philosopher for whom the well known previous figure was named, Thomas Hobbes , or someone else that I couldn’t find and don’t know?) But I digress…

“Wisdom is a prerequisite for good judgement.” Judgement is used daily by evaluators. It results in the determination of value, merit, and/or worth of something. Evaluators make a judgement of value, merit, and/or worth. We come to these judgements through experience. Experience with people, activities, programs, contributions, LIFE. Everything we do provides us with experience; it is what we do with that experience that results in wisdom and, therefore, leads to good judgements.

Experience is a hard teacher; demanding, exacting, and often obtuse. My 19 y/o daughter is going to summer school at OSU. She got approval to take two courses and for those courses to transfer to her academic record at her college. She was excited about the subject; got the book; read ahead; and looked forward to class, which started yesterday. After class, I had never seen a more disappointed individual. She found the material uninteresting (it was mostly review because she had read ahead), she found the instructor uninspiring (possibly due to class size of 35). To me, it was obvious that she needed to re-frame this experience into something positive; she needed to find something she could learn from this experience that would lead to wisdom. I suggested that she think of this experience as a cross cultural exchange; challenging because of cultural differences. In truth, a large state college is very different from a small liberal arts college; truly a different culture. She has four weeks to pull some wisdom from this experience; four weeks to learn how to make a judgement that is beneficial. I am curious to see what happens.

Not all evaluations result in beneficial judgements; often, the answer, the judgement, is NOT what the stakeholders want to hear. When that is the case, one needs to re-frame the experience so that learning occurs (both for the individual evaluator as well as the stakeholders) so that the next time the learning, the hard won wisdom, will lead to “good” judgement, even if the answer is not what the stakeholders want to hear. Matt started his discussion with the saying that “wisdom, rooted in knowledge of self, is a prerequisite for good judgement”. Knowing your self is no easy task; you can only control what you say, what you do, and how your react (a form of doing/action). The study of those things is a life long adventure, especially when you consider how hard it is to change yourself. Just having knowledge isn’t enough for a good judgement; the evaluator needs to integrate that knowledge into the self and own it; then the result will be “good judgements”; the result will be wisdom.

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: value

Evaluation and values.

Evaluation everyday

Professional development

Need to know? Nice to know?

Evaluation and creativity

Evaluation Cost Redux and other topics

How much should an evaluation cost.

Lightbulbs and other insights

Independence–a good thing?

Good judgement

Contact Info