Sep
25
Filed Under (program evaluation) by Molly on 25-09-2014

Over the last several months, the Local Arrangements Working Group has been blogging at AEA365. One of ways evaluators can get ready for the upcoming annual conference is to read what the LAWG has to say about the conference. This year, the conference is once again in Denver. AEA was in Denver in 2008. Be forewarned–Denver is the mile high city. The air is rarefied and very dry. It may snow!

The LAWG has a lot to say about the conference and there are A LOT of links in these posts that are worth checking.  For those who have not been to AEA before or for those who have recently embraced evaluation, reading their posts are a wealth of information, as is the AEA website.

I will be presenting at two sessions this year–one on blogging (duh…) and one on capacity building. I see them related. I will also (like last year) be assisting with a professional development session (number 25) with my long time friend and colleague, Jim Altschuld. The professional development session occurs on Wednesday, October 15, 2014 from 8:00am MT – 3:00pm MT. It is titled Practical Ways to Link Needs Assessment (NA) with Asset/Capacity Building. (Just a little advertisement :) ) It will draw from his new book, Bridging the Gap Between Asset/Capacity Building and Needs Assessment. Bridging the Gap-altschuld Read the rest of this entry »

Sep
05
Filed Under (Methodology, program evaluation) by Molly on 05-09-2014

I just finished a chapter on needs assessment in the public sector–you know that part of the work environment that provides a service to the community and receives part of its funding from the county/state/federal governments. Most of you know I’ve been an academic for at least 31 years, maybe more (depending on when you start the clock). In that time I’ve worked as an internal evaluator, a program planner, and a classroom teacher. Most of what I’ve done has an evaluative component to it. (I actually earned my doctorate in program evaluation when most people in evaluation came from other disciplines.) During that time I’ve worked on many programs/projects in a variety of situations (individual classroom, community, state, and country). I find it really puzzling that evaluators will take on evaluation without having a firm foundation on which to base those evaluations. (I know I have done this; I can offer all manner of excuses, only not here).

If I had been invited to participate in the evaluation at the beginning of the program, at the conceptualization stage, I would have asked if a needs assessment had been done and what was the outcome of that assessment. Was there really a lack (i.e., a need); or was this “need” contrived to do something else (bring in grant money, further a career, make a stakeholder happy, etc.)? Read the rest of this entry »

Jul
31
Filed Under (program evaluation) by Molly on 31-07-2013

The American Evaluation Association has opened its registration for the 2013 meeting in Washington DC.  This meeting promises to be attended by the most people yet.  Eleven years ago we were in D. C. and broke all attendance records to date.  I remember because that was my presidential year…the year that the evaluation profession started thinking that evaluation was a system; that everything we do is connected.  Several people have commented about AEA–that they didn’t know there was such an association; that they didn’t know about the conference; that they weren’t members.  So folks, here is the skinny on AEA (at least part of the skinny…).

The American Evaluation Association was officially founded in 1986 AEA logoas a combined organization of the Evaluation Research Society and Evaluation Network.  ERS was academic and EN was practitioner; merging the two was a challenge as each thought something would be lost.  This is a good example of where the whole is greater that the sum of its parts.  The differences were pronounced and debatable (now you only see AEA).  Robert (Bob) B. Ingle was the force behind the conference; he mounted the first EN/ERS conference in 1981 in Austin, Texas.  I was a graduate student.  I was in awe.  Although I had been to numerous  professional conferences before attending this first conference, I had never met any one like Bob Ingle.  His first comment to me once we connected after playing phone tag was, “You spell your name wrong!”  (Turns out he was the Scotland branch of the German house of Engel; my ancestors changed the spelling when they came out of Germany.)   I was a nascent graduate student in love with my studies and here comes this brusque, acerbic, and outrageous giant.  He became my good friend–I knew him from 1981 until he died in 1998.  He believed passionately in program evaluation.  I think he is smiling at the growth in the profession and the organization.  He knew a lot of us; he saw the association through the good times and the bad times.  I could end here and say, the rest is history…only there is so much to tell.  The association went from an all volunteer organization at its founding in 1986 to an organization of over 8,000 members run by an association management firm.  Susan Kistler (of Kistler Associates) was our executive director for the last 15 years.  (The association has transitioned to a new management firm [SmithBucklin] and a new executive director [Denise Roosendaal]).  Seeing the association transition is bittersweet;  growth is good, the loss of family feeling is sad.  The association is no longer feels intimate, family; it offers so much more to folks who are members.

David Bernstein is the co-chair of the local arrangements working group (LAWG) for this year’s conference.  He lead off a week of  AEA365 talking about the conference.  Read this post.  It tells you a lot about the conference. This week AEA365 is being written by the local arrangements working group.  The role of the local arrangements group is to make sure the folks who attend the conference have a good time, both at the conference and in DC.  DC is a wonderful city.  You cannot see it in a week; it is always changing.  Take a day if you have never been to see the city’s high points.  It is the nation’s capitol, after all, and there are many high points.

The members only AEA August newsletter also talks about registration with hyperlinks to the registration site, the conference program, and hotel accommodations.  (The members only newsletter is just one reason to join AEA.)  I’ve been going to AEA since 1981.  This is the first year I will not have a paper/poster/etc. on the program.  (I am doing a professional development session with Jim Altschuld, though; it is number 22).

Each year I attend AEA, I think of the three evaluative criteria that FOR ME makes a good conference:  See three long time friends; meet three new people who could become friends; and get three new ideas.  If I do all this, I usually come home energized.  I hope to see you there.

Jun
12
Filed Under (Methodology, program evaluation) by Molly on 12-06-2013

I have a few thoughts about causation, which I will get to in a bit…first, though, I want to give my answers to the post last week.

I had listed the following and wondered if you thought they were a design, a method, or an approach. (I had also asked which of the 5Cs was being addressed–clarity or consistency.)  Here is what I think about the other question.

Case study is a method used when gathering qualitative data, that is, words as opposed to numbers.  Bob Stake, Robert Brinkerhoff, Robert Yin, and others have written extensively on this method.

Pretest-post test Control Group is (according to Campbell and Stanley, 1963) an example of  a true experimental design if a control group is used (pg. 8 and 13).  NOTE: if only one group is used (according to Campbell and Stanley, 1963), pretest-post test is considered a pre-experimental design (pg. 7 and 8); still it is a design.

Ethnography is a method used when gathering qualitative data often used in evaluation by those with training in anthropology.  David Fetterman is one such person who has written on this topic.

Interpretive is an adjective use to describe the approach one uses in an inquiry (whether that inquiry is as an evaluator or a researcher) and can be traced back to the sociologists Max Weber and Wilhem Dilthey in the later part of the 19th century.

Naturalistic is  an adjective use to describe an approach with a diversity of constructions and is a function of “…what the investigator does…” (Lincoln and Guba, 1985, pg.8).

Random Control Trials (RCT) is the “gold standard” of clinical trials, now being touted as the be all and end all of experimental design; its proponents advocate the use of RCT in all inquiry as it provides the investigator with evidence that X (not Y) caused Z.

Quasi-Experimental is a term used by Campbell and Stanley(1963) to denote a design where random assignment cannot be made for ethical or practical reasons be accomplished; this is often contrasted with random selection for survey purposes.

Qualitative is an adjective to describe an approach (as in qualitative inquiry), a type of data (as in qualitative data) or
methods (as in qualitative methods).  I think of qualitative as an approach which includes many methods.

Focus Group is a method of gathering qualitative data through the use of specific, structured interviews in the form of questions; it is also an adjective for defining the type of interviews or the type of study being conducted (Krueger & Casey, 2009, pg. 2)

Needs Assessment is method for determining priorities for the allocation of resources and actions to reduce the gap between the existing and the desired.

I’m sure there are other answers to the terms listed above; these are mine.  I’ve gotten one response (from Simon Hearn at BetterEvaluation).  If I get others, I’ll aggregate them and share them with you.  (Simon can check his answers against this post.

Now causation, and I pose another question:  If evaluation (remember the root word here is value) is determining if a program (intervention, policy, product, etc. ) made a difference, and determined the merit or worth (i.e., value) of that program (intervention, policy, product, etc.), how certain are you that your program (intervention, policy, program, etc.) caused the outcome?  Chris Lysy and Jane Davidson have developed several cartoons that address this topic.  They are worth the time to read them.

Dec
05
Filed Under (Methodology, program evaluation) by Molly on 05-12-2012

Needs Assessment is an evaluative activity; the first assessment that a program developer must do to understand the gap between what is and what needs to be (what is  desired).  Needs assessments are the evaluative activity in the Situation box of a linear logic model. 

Sometimes, however, the target audience doesn’t know what they need to know and that presents challenges for the program planner.  How do you capture a need when the target audience doesn’t know they need the (fill in the blank).  That challenge is the stuff of other posts, however.

I had the good fortune to talk with Sam Angima, an Oregon  Regional Administrator who has been tasked with the charge of developing expertise in needs assessment.  Each Regional Administrator (there are 12) has been tasked with different charges to whom faculty can be referred.  We captured Sam’s insights in a conversational Aha! moment.  Let me know what you think.

 

 

May
06
Filed Under (Methodology, program evaluation) by Molly on 06-05-2011

I’ve talked about how each phase of a logic model has evaluative activities.  I’ve probably even alluded to the fact that needs assessment is the evaluative activity for that phase called situation (see the turquoise area on the left end of the image below.)

What I haven’t done is talk about is the why, what,  and how of needs assessment (NA).  I also haven’t talked about the utilization of the findings of a needs assessment–what makes meaning of the needs assessment.

OK.  So why is a NA conducted?  And what is a NA?

Jim Altschuld is my go-to person when it comes to questions about needs assessment.  He recently edited a series of books on the topic.

Although Jim is my go-to person, Belle Ruth Witkin (a colleague, friend, and collaborator of Jim Altschuld) says in the preface to the co-authored volume (Witkin and Altschuld, 1995–see below),  that the most effective way to decide the best way to divide the (often scarce) resources among the demands (read programs) is to conduct a needs assessment when the planning for the use of those resources begins.

Book 1 of the kit discusses an overview.  In that volume, Jim defines what a needs assessment is: “Needs assessment is the process of identifying needs, prioritizing them, making needs-based decisions, allocating resources, and implementing actions in organizations to resolve problems underlying important needs (pg.20).”  Altschuld states that there are many models for assessing needs and provides citations for those models.  I think the most important aspect of this first volume is the presentation of the phased model developed by Belle Ruth Witkin in 1984 and revised by Altschuld and Witkin in their 1995 and 2000 volumes.Those phases are preassessment, assessment, and postassessment.  They divide those three phases into three levels, primary, secondary, and tertiary, each level targeting a different group of stakeholders.  This volume also discusses the why and the how.  Subsequent volumes go into more detail–volume 2 discusses phase 1 (getting started); volume 3 discusses phase II (collecting data); volume 4 discusses analysis and priorities; and volume 5 discusses phase III (taking action).

Laurie Stevahn and Jean A. King are the authors of this volume. In chapter 3, they discuss strategies for the action plan using facilitation procedures that promote positive relationships, develop shared understanding, prioritize decisions, and assess progress.  They warn of interpersonal conflict and caution against roadblocks that impede change efforts.  They also promote the development of evaluation activities at the onset of the NA because that helps ensure the use of the findings.

Needs assessment is a political experience.  Some one (or ones) will feel disenfranchised, loose resources, have programs ended.  These activities create hard feelings and resentments.  These considerations need to be identified and discussed at the beginning of the process.  It is like the elephant and the blind people–everyone has an image of what the creature is, there may or may not be consensus, yet for the NA to be successful, consensus is important.  Without it, the data will sit on someone’s shelf or in someone’s computer.  Not useful.