Monthly Archives: June 2012

Good judgement

Posted on by

Matt Keene, AEAs thought leader for June 2012 says, “Wisdom, rooted in knowledge of thyself, is a prerequisite of good judgment. Everybody who’s anybody says so – Philo Judaeus,

Socrates,  Lao-tse,

Plotinus, Paracelsus,

Swami Ramdas,  and Hobbs.

I want to focus on the “wisdom is a prerequisite of good judgement” and talk about how that relates to evaluation.  I also liked the list of “everybody who’s anybody.”   (Although I don’t know who Matt means by Hobbs–is that Hobbes  or the English philosopher for whom the well known previous figure was named, Thomas Hobbes , or someone else that I couldn’t find and don’t know?)  But I digress…

 

“Wisdom is a prerequisite for good judgement.”  Judgement is used daily by evaluators.  It results in the determination of value, merit, and/or worth of something.  Evaluators make a judgement of value, merit, and/or worth.  We come to these judgements through experience.  Experience with people, activities, programs, contributions, LIFE.  Everything we do provides us with experience; it is what we do with that experience that results in wisdom and, therefore, leads to good judgements.

Experience is a hard teacher; demanding, exacting, and often obtuse.  My 19 y/o daughter is going to summer school at OSU.  She got approval to take two courses and for those courses to transfer to her academic record at her college.  She was excited about the subject; got the book; read ahead; and looked forward to class, which started yesterday.  After class, I had never seen a more disappointed individual.  She found the material uninteresting (it was mostly review because she had read ahead), she found the instructor uninspiring (possibly due to class size of 35).  To me, it was obvious that she needed to re-frame this experience into something positive; she needed to find something she could learn from this experience that would lead to wisdom.  I suggested that she think of this experience as a cross cultural exchange; challenging because of cultural differences.  In truth, a large state college is very different from a small liberal arts college; truly a different culture.  She has four weeks to pull some wisdom from this experience; four weeks to learn how to make a judgement that is beneficial.  I am curious to see what happens.

Not all evaluations result in beneficial judgements; often, the answer, the judgement, is NOT what the stakeholders want to hear.  When that is the case, one needs to re-frame the experience so that learning occurs (both for the individual evaluator as well as the stakeholders) so that the next time the learning, the hard won wisdom, will lead to “good” judgement, even if the answer is not what the stakeholders want to hear.  Matt started his discussion with the saying that “wisdom, rooted in knowledge of self, is a prerequisite for good judgement”.  Knowing your self is no easy task; you can only control what you say, what you do, and how your react (a form of doing/action).  The study of those things is a life long adventure, especially when you consider how hard it is to change yourself.  Just having knowledge isn’t enough for a good judgement; the evaluator needs to integrate that knowledge into the self and own it; then the result will be “good judgements”; the result will be wisdom.

Unit of analysis–TRUTH

Posted on by

I started this post back in April.  I had an idea that needed to be remembered…it had to do with the unit of analysis; a question which often occurs in evaluation.  To increase sample size and, therefore,  power, evaluators often choose run analyses on the larger number when the aggregate, i.e., smaller number is probably the “true” unit of analysis.  Let me give you an example.

A program is randomly assigned to fifth grade classrooms in three different schools.  School A has three classrooms; school B has two classrooms; and school C has one classroom.  All together, there are approximately 180 students, six classrooms, three schools.  What is the appropriate unit of analysis?  Many people use students, because of the sample size issue.  Some people will use classroom because each got a different treatment.  Occasionally, some evaluators will use schools because that is the unit of randomization.  This issue elicits much discussion.  Some folks say that because students are in the school, they are really the unit of analysis because they are imbedded in the randomization unit.  Some folks say that students is the best unit of analysis because there are more of them.  That certainly is the convention.  What you need to decide is what is the unit and be able to defend that choice.  Even though I would loose power, I think I would go with the the unit of randomization.  Which leads me to my next point–truth.

At the end of the first paragraph, I use the words “true” in quotation marks. The Kirkpatricks in their most recent blog opened with a quote from the US CIA headquarters in Langley Virginia, “”And ye shall know the truth, and the truth shall make you free”.   (We wont’ talk about the fiction in the official discourse, today…)   (Don Kirkpatrick developed the four levels of evaluation specifically in the training and development field.)  Jim Kirkpatrick, Don’s son, posits that, “Applied to training evaluation, this statement means that the focus should be on discovering and uncovering the truth along the four levels path.”  I will argue that the truth is how you (the principle investigator, program director, etc.) see the answer to the question.  Is that truth with an upper case “T” or is that truth with a lower case “t”?  What do you want it to mean?

Like history (history is what is written, usually by the winners, not what happened), truth becomes what do you want the answer to mean.  Jim Kirkpatrick offers an addendum (also from the CIA), that of “actionable intelligence”.  He goes on to say that, “Asking the right questions will provide data that gives (sic) us information we need (intelligent) upon which we can make good decisions (actionable).”  I agree that asking the right question is important–probably the foundation on which an evaluation is based.  Making “good decisions”  is in the eyes of the beholder–what do you want it to mean.

Needs assessment–shifting the paradigm

Posted on by

“Resilience = Not having all of your eggs in one basket.

Abundance = having enough eggs.”

Borrowed from and appearing in the blog by Harold Jarche, Models, flows, and exposure, posted April 28, 2012.

 

In January, John Hagel blogged in  Edge Perspectives:  “If we are not enhancing flow, we will be marginalized, both in our personal and professional life. If we want to remain successful and reap the enormous rewards that can be generated from flows, we must continually seek to refine the designs of the systems that we spend time in to ensure that they are ever more effective in sustaining and amplifying flows.”

That is a powerful message.  Just how do we keep from being marginalized, especially when there is a shifting paradigm?  How does that relate to evaluation?  What exactly do we need to do to keep evaluation skills from being lost in the shift and be marginalized?  Good questions.

The priest at the church I attend is retiring, after 30 years of service.  This is a significant and unprecedented change (at least in my tenure there).  Before he left for summer school in Minnesota, he gave the governing board a pep talk that has relevance to evaluation.  He posited that what we needed to do was not focus on what we needed, rather focus on what strengths and assets we currently have and build on them.  No easy task, to be sure.  And not the  usual approach for an interim.  The usual approach is what do we want; what do we need for this interim.  See the shifting paradigm?  I hope so.

Needs assessment is often the same approach–what do you want; what do you need.  (Notice the use of the word “you” in this sentence; more on that later in another post.)  A well intentioned evaluator recognizes that something is missing or lacking and conducts a needs assessment documenting that need/lack/deficit.  What would happen, do you think, if the evaluator documented what assets existed and developed a program to build that capacity?  Youth leadership development has been building programs to build assets for many years (See citations below).  The approach taken by the youth development professionals is that there are certain skills, or assets, which, if strengthened, build resilience.  Buy building resilience, needs are mitigated; problems solved or avoided; goals met.

So what would happen if, when conducting a “needs” assessment, an evaluator actually conducted an asset assessment and developed programs to benefit the community by building capacity which strengthened assets and built resiliency?  Have you ever tried that approach?

By focusing on strengths and assets instead of weaknesses and liabilities, programs could be built that would benefit more than a vocal minority.  The greater whole could benefit.  Wouldn’t that be novel?  Wouldn’t that be great!

Citations:

1.  Benson, P. L. (1997).  All Kids are Our Kids.  San Francisco:  Jossey-Bass Publishers

2.  Silbereisen, R. K. & Lerner, R. M. (2007).  Approaches to Positive Youth Development. Los Angeles: Sage Publications.