Filed Under (program evaluation) by Molly on 27-08-2013

“In reality, winning begins with accountability. You cannot sustain success without accountability. It is an absolute requirement!” (from

I’m quoting here.  I wish I had thought of this before I read it.  It is important in everyone’s life, and especially when evaluating.


Webster’s defines accountability as, “…“the quality or state of being accountable; an obligation (emphasis added) or willingness to accept responsibility for one’s actions.”  The business dictionary goes a little further and defines accountability as “…The obligation of an individual (or organization) (parentheses added) to account for its activities, accept responsibility for them, and to disclose the results in a transparent manner.”

It’s that last part to which evaluators need to pay special attention; the “disclose results in a transparent manner” part.  There is no one looking over your shoulder to make sure you do “the right thing”; that you read the appropriate document; that you report the findings you found not what you know the client wants to hear.  If you maintain accountability, you are successful; you will win.

AEA has a adopted a set of Guiding Principles Guiding principlesfor the organization and its members.  The principles are 1) Systematic inquiry; 2) Competence; 3) Integrity/Honesty; 4) Respect for people; and 5) Responsibilities for the General and Public Welfare.  I can see where accountability lies within each principle.  Can you?

AEA has also endorsed the Program Evaluation Standards  program evaluation standards of which there are five as well.  They are:  1) Utility, 2) Feasibility, 3) Proprietary, 4) Accuracy, and 5) Evaluation accountability.  Here, the developers were very specific and made accountability a specific category.  The Standard specifically states, “The evaluation accountability standards encourage adequate documentation of evaluations and a metaevaluative perspective focused on improvement and accountability for evaluation processes and products.”

You may be wondering about the impetus for this discussion of accountability (or, not…).  I have been reminded recently that only the individual can be accountable.  No outside person can do it for him or her.  If there is an assignment, it is the individual’s responsibility to complete the assignment in the time required.  If there is a task to be completed, it is the individual’s responsibility (and Webster’s would say obligation) to meet that responsibility.  It is the evaluator’s responsibility to report the results in a transparent manner–even if it is not what was expected or wanted.  As evaluator’s we are adults (yes, some evaluation is completed by youth; they are still accountable) and, therefore, responsible, obligated, accountable.  We are each one responsible–not the leader, the organizer, the boss.  Each of us.  Individually.  When you are in doubt about your responsibility, it is your RESPONSIBILITY to clarify that responsibility however works best for you.  (My rule to live by number 2:  Ask.  If you don’t ask, you won’t get; if you do, you might not get.)

Remember, only you are accountable for your behavior–No. One. Else.  Even in an evaluation.; especially in an evaluation




Filed Under (Data Analysis, Methodology, program evaluation) by Molly on 22-08-2013

We are approaching Evaluation 2013 (Evaluation ’13).  This year October 16-19, with professional development sessions both before and after the conference.  One of the criteria that I use to determine a “good” conference is did I get three new ideasbright idea 3 (three is an arbitrary number).  One way to get a good idea to use outside the conference, in your work, in your everyday activities is to experience a good presentation.  Fortunately, in the last 15 years much has been written on how to give a good presentation both verbally and with visual support.  This week’s AEA365 blog (by Susan Kistler) talks about presentations as she tells us again about the P2i initiative sponsored by AEA.

I’ve delivered posters the last few years (five or six) and P2i talks about posters in the downloadable handout called, Guidelines for Posters.  Under the tab called (appropriately enough) Posters, P2i also offers information on research posters and a review of other posters as well as the above mentioned Guidelines for Posters.  Although more and more folks are moving to posters (until AEA runs out of room, all posters are on the program), paper presentations with the accompanying power point are still deriguere, the custom of professional conferences.   What P2i has to say about presentations will help you A LOT!!  Read it.

Read it especially if presenting in public, whether to a large group of people or not.  It will help you.  There are some really valuable points that are reiterated in the AEA365 as well as other places.  Check out the following TED talks, look especially for Nancy Durate and Hans Rosling.  A quick internet search yielded the following: About 241,000,000 results (0.43 seconds).  I entered the phrase, “how to make a good presentation“.  Some of the sites speak to oral presentations; some address visual presentations.  What most people do is try to get too much information on a slide (typically using Power point).  Prezi gives you one slide with multiple images imbedded within it.  It is cool.  There are probably other approaches as well.  In today’s world, there is no reason to read your presentation–your audience can do that.  Tell them!  (You know, tell them what they will hear, tell them, tell them what they heard…or something like that.)  If you have to read, make sure what they see is what they hear–see hear compatibility is still important, regardless of the media used.

Make an interesting presentation!  Give your audience at least one good idea!bright idea

Filed Under (program evaluation) by Molly on 15-08-2013

Several folks read and commented on my previous AEA post.  That is heartening.  One comment was about what a new person should do at the conference.  That is today’s post.  I am fortunate because the Graduate Student and New Evaluator Topical Interest Group post in aea365 last week talked about just that.  I recommend you check it out.

Although I couldn’t find a “First Time Attendee” session in this year’s on line program, I could find a lot of other sessions which sounded interesting.  In the past, this session was offered just before the reception on Wednesday.  That was a long time ago (I found reference to it in the 1999 program), however, and AEA (like so many other things) has moved on, grown, and changed.  I remember attending several and contributing because I was a long time attendee.  They were informative, much like the aea365 blog post.

The one thing that the aea365 post didn’t mention that I think is important is children.  AEA is, and always has been, family friendly.  Children are welcome (by most; there are some curmudgeons, to be sure).  I am a single parent by choice.  I built my family through adoption.  From the time my oldest daughter (now 20) came home, I took her and then she and her sister (now 17) to AEA; we went as a family until 2007 when AEA was in Baltimore.  By the time they got into high school, it was harder to take them.  (I did take Mersedes last year to Minneapolis;  she isn’t coming to DC this year; this being her high school senior year…unhappy face)  They developed their own cohort of friends, people who still ask after them today when I go alone.

How did taking them actually work?  Most of the hotel venues used by AEA can recommend a sitting service, one that has been used by their guests.  It is worth the cost.  When I was President, the service was a life saver.  (I used White House Nannies in DC)  I paid an hourly rate when I needed coverage, which was about 30% of the time.  The rest of the time, the girls attended sessions, took their coloring books and sundries to the back of the room and played quietly while I did my professional thing.  There were multiple benefits in this arrangement:  They got to see Mom “working”; they got to see parts of DC they wouldn’t normally see (arrangements were made at the time of booking), they learned behaviors of a professional meeting, saw what was expected, and learned to talk to grown-ups.  They love to tell the story of sitting under the table in the back of the room, though I don’t know how many years they did that…I’m guessing a lot.  Some of my best friends are also their friends with whom they maintain relationships.  If you have a partner, bringing children is easier.  Some of my friends took their son/daughter and traded off responsibilities; their children also created a whole cadre of connections, some of which last to this day.  Being in DC is a wonderful opportunity for a young family; DC is an amazing city with lots to see and do.  I strongly urge you to take a day (or two) and enjoy the city and the conference with your family.

Filed Under (program evaluation) by Molly on 07-08-2013

Some days I have no idea what to write.  This is one of those days.  So I thought I’d provide some thoughts that have been circulating around my brain for the past several days.  They are in no particular order and are of no particular importance.  They are just thoughts that relate to evaluation.

  1. Recently, I had the opportunity to attend a professional development session on qualitative data with Johnny Saldana.  Other than there wasn’t enough time to thoroughly cover the topic (it was the structure of the conference), I was once again impressed with the complexity of the topic.  Although it was titled “Advanced Qualitative Inquiry”, I thought a participant would need to know something about coding qualitative data in order to understand the this complexity.  Saldana did a masterful job–which is something given that he comes from a discipline so very different from evaluation (although some people draw a parallel between his discipline–theater–and evaluation; both being an interpretive disciplines).
  2. I’ve mentioned Saldana before–he was commissioned by SAGE to write the third edition of the Miles and Huberman classic, Qualitative Data Analysis.  The third edition was published in April, 2013..Qualitative data analysis ed. 3  Some of what Saldana shared with us in the session was from his book, Coding Manual for Qualitative Researchers. coding manual--johnny saldana  I have that book and have started reading it.  I’ll be submitting a review of it and the Qualitative Data Analysis to American Journal of Evaluation this fall.
  3. I haven’t gotten anything written on the four manuscripts I thought I’d be able to finish this summer.  I’ve about six weeks left (classes at OSU don’t start until the middle of September).  So much else has happened:  I’ve been asked to develop a chapter for a New Directions in Evaluation volume on Needs Assessment in the public sector; to co-lead a professional development session on Needs Assessment and Assets; and to review a manuscript on qualitative evaluation using a logic mode in addition to developing the evaluation I mentioned a while back.  This experience reinforces truly the need to block time for writing and to protect it.  I block time; I don’t tend to protect it.  Lesson learned.
  4. The evaluation model (of that major qualitative evaluation I will conduct) was presented at the county operations meeting Monday.  The group wasn’t opposed; they just weren’t exuberantly enthusiastic, either.  They had some interesting questions and interesting concerns that we will need to address.  It is hard to present a qualitative study to a group who have been trained to think of comparison groups and baselines.  We are using a developmental model (not summative so we don’t need a baseline; we want only to know the now of the experience) and we are using criteria not comparing groups against one another (criteria referenced NOT norm referenced).  More on this later…
  5. Surveys were another question that was posed to me–odd or even scales on the survey.  The answer there is, “It all depends.”  The “all depends” relates to many variables including cost and time and other resources.  The Kirkpatricks talk about aspects of evaluation in their blog this week.  They include a webinar of what to do when resources are scarce.