Needs assessment–shifting the paradigm

Posted on by

“Resilience = Not having all of your eggs in one basket.

Abundance = having enough eggs.”

Borrowed from and appearing in the blog by Harold Jarche, Models, flows, and exposure, posted April 28, 2012.

 

In January, John Hagel blogged in  Edge Perspectives:  “If we are not enhancing flow, we will be marginalized, both in our personal and professional life. If we want to remain successful and reap the enormous rewards that can be generated from flows, we must continually seek to refine the designs of the systems that we spend time in to ensure that they are ever more effective in sustaining and amplifying flows.”

That is a powerful message.  Just how do we keep from being marginalized, especially when there is a shifting paradigm?  How does that relate to evaluation?  What exactly do we need to do to keep evaluation skills from being lost in the shift and be marginalized?  Good questions.

The priest at the church I attend is retiring, after 30 years of service.  This is a significant and unprecedented change (at least in my tenure there).  Before he left for summer school in Minnesota, he gave the governing board a pep talk that has relevance to evaluation.  He posited that what we needed to do was not focus on what we needed, rather focus on what strengths and assets we currently have and build on them.  No easy task, to be sure.  And not the  usual approach for an interim.  The usual approach is what do we want; what do we need for this interim.  See the shifting paradigm?  I hope so.

Needs assessment is often the same approach–what do you want; what do you need.  (Notice the use of the word “you” in this sentence; more on that later in another post.)  A well intentioned evaluator recognizes that something is missing or lacking and conducts a needs assessment documenting that need/lack/deficit.  What would happen, do you think, if the evaluator documented what assets existed and developed a program to build that capacity?  Youth leadership development has been building programs to build assets for many years (See citations below).  The approach taken by the youth development professionals is that there are certain skills, or assets, which, if strengthened, build resilience.  Buy building resilience, needs are mitigated; problems solved or avoided; goals met.

So what would happen if, when conducting a “needs” assessment, an evaluator actually conducted an asset assessment and developed programs to benefit the community by building capacity which strengthened assets and built resiliency?  Have you ever tried that approach?

By focusing on strengths and assets instead of weaknesses and liabilities, programs could be built that would benefit more than a vocal minority.  The greater whole could benefit.  Wouldn’t that be novel?  Wouldn’t that be great!

Citations:

1.  Benson, P. L. (1997).  All Kids are Our Kids.  San Francisco:  Jossey-Bass Publishers

2.  Silbereisen, R. K. & Lerner, R. M. (2007).  Approaches to Positive Youth Development. Los Angeles: Sage Publications.

 

Perpetual Beta

Posted on by

They [CEO’s] simply expect unpredictability. For them, there is no “new normal.” This is why perpetual Beta is a constant theme here. It is a necessary perspective in dealing with increasing complexity.

This is a piece of interesting information I picked up on a blog about letting go.  I found it interesting reading, especially when viewed wearing my evaluators hat.  Increasingly, the programs we devise address problems which are complex.  No longer are we willing to look at a small slice of a problem find one solution and address it (control, according to Jarche).  Rather, all problems are interrelated; consequently, all solutions must also be interrelated (Jarche talks about letting go [of control]).

Every year, Corvallis Sister Cities-Gondar holds a walk for water.  The idea being to raise funds to support the activities of the Sister Cities programs (in this case clean, potable, available water or planting trees in treeless areas).  I asked the president of the organization if in their teaching activities they were talking about population control.  He answered, sadly, No.  Even though he could see the connection between clean water and population, that step hadn’t been taken in the educational efforts of the program.  But providing funding for individual wells to provide clean water was.  I see this as simplification of a complex problem.  (Your point, Engle?)  Right.  Point.

All the above relates to program planning and developing a logic model for that program.  This tries to remind folks to expect the unexpected in planning the program to address a problem and think about building the unexpected into the model.  Yes, you might get clean water if you teach well digging skills; yet that clean water will reduce infant mortality.  With the reduction of infant mortality, population reaching adulthood will increase.  An increase in population will tax already limited resources (food, water, shelter–not to mention consumer resources); that leads back to the basic problem–no/little access to clean water.  Teaching well drilling skills is only part of the problem; the rest of the problem is unexpected.  By building that into the model, one can see the relation to the bigger picture; the relation to the system.  Even if designing the program will only address that one small part (of the problem), identifying the unexpected, the unpredictable, will help the program planner see clearly.

 


Ordinary Life

Posted on by

Once again, it is the whole ‘balance’ thing…(we) live in ordinary life and that ordinary life is really the only life we have…I’ll take it. It has some great moments…

 

These wise words come from the insights of Buddy Stallings, Episcopal priest in charge of a large parish in a large city in the US.  True, I took them out of context; the important thing is that they resonated with me from an evaluation perspective.

Too often, faculty and colleagues come to me and wonder what the impact is of this or that program.  I wonder, What do they mean?  What do they want to know? Are they only using words they have heard–the buzz words?  I ponder how this fits into their ordinary life. Or are they outside their ordinary life, pretending in a foreign country?

A faculty member at Oregon State University equated history to a foreign country.  I was put in a mind that evaluation is a foreign country to many (most) people, even though everyone evaluates every day, whether they know it or not.  Individuals visit that contry because they are required to visit; to gather information; to report what they discovered.  They do this with out any special preparation.  Visiting a foreign country entails preparation (at least it does for me).  A study of customs, mores, foods, language, behavior, tools (I’m sure I’m missing something important in this list) is needed; not just necessary, mandatory.  Because although the foreign country may be exotic and unique and novel to you, it is ordinary life for everyone who lives there.  The same is true for evaluation.  There are customs; students are socialized to think and act in a certain way.  Mores are constantly being called into question; language, behaviors, tools, which not known to you in your ordinary life, present themselves. You are constantly presented with opportunities to be outside your ordinary life.  Yet, I wonder what are you missing by not seeing the ordinary; by pretending that it is extraordinary?  By not doing the preparation to make evaluation part of your ordinary life, something you do without thinking.

So I ask you, What preparation have you done to visit this foreign country called EVALUATION?  What are you currently doing to increase your understanding of this country?  How does this visit change your ordinary life or can you get those great moments by recognizing that this is truly the only life you have?   So I ask you, What are you really asking when you ask, What are the impacts?

 

All of this has significant implications for capacity building.

Some random thoughts

Posted on by

Today, I’m writing about several random thoughts which have occurred to me in the last two weeks–some prompted by comments; some not.  And before I forget–I was sick last week and didn’t blog (did you miss me?); I’ll be gone next week and won’t blog (will you miss me?).  I’ll be back the week of May 21.

Random thought 1.

I’m doing this evaluation capacity building program for folks who are interested in evaluation.  Several (more than 2) participants have commented to me, in person or electronically, that they are being asked to use their evaluation expertise garnered from this program in projects that are in their content area and are not their programs–serving as evaluation consultants, if you will.  This evaluation capacity building program draws Extension professionals from across the western states and includes folks from natural resources, agriculture, family and community science, nutrition, and 4-H.  It is another example in my life where I put information together (like this blog) and send it out and rarely know what happens at the other end of the send.  Getting comments like this is heart warming. Capturing these stories will be part of our summative evaluation.

Random thought 2.

Although I am a program evaluator, specializing in community-based educational programs, there are other kinds of evaluation. (Note: This is not a discussion of evaluation models, like naturalistic evaluation or developmental evaluation–that is another discussion.)   Evaluations can be conducted on products, processes, policy (including proposals, plans, and possibilities), performance, personnel as well as program.  Michael Scriven, in his Evaluation Thesaurus discusses all these.  The goal of all forms of evaluation is to determine the merit, worth, or effectiveness of the thing being evaluated (evaluand).

Random thought 3.

I love getting comments about what I write.  If you comment, I may not respond.  I always read them.  One comment was about subscribing.  You can subscribe to the blog through an RSS feed button in the upper left side of the blog (under the word, Subscribe).  You can also subscribe by email.  The blog postings are archived (thank you OSU), so you can go back and see what I’ve said.  I try not to repeat myself.

Random thought 4.

Because I blog as part of my work, and because I work at Oregon State University, I use the resources available to me there (like wordpress for this blog).  It is written using Firefox NOT Internet Explorer.  The appearance may be different depending on the browser used.  Try viewing it in a different browser if you are having trouble seeing the blog.

Random thought 5.

Most folks who are program people know a lot about their content area, whether it is invasive species, viticulture, woodlot management, nutrition, or youth development.  What I’m finding is that knowing what to do with the data they have about their programs isn’t as well known.  The data from most of what Extension does can be analyzed using relatively simple statistics, such as chi-square, t-test, ANOVA.  Sometimes using an ANCOVA is needed.  There are many useful resources available for broadening understanding about statistics.  One I really like is a TED Talk by Hans Rosling.

 

Cogent thoughts from other blogs

Posted on by

CAVEAT:  This may be too political for some readers.

Sometimes, there are ideas that appear in other blogs that may or may not be directly related to my work in evaluation.  Because I read them, I see evaluative relations and think they are important enough to pass along.  Today is one of those days.  I’ll try to connect the dots  between what I read and share here and evaluation.  (For those of you who are interested in the Connect the Dots, a major event day on climate change and weather on May 5, 2012, go here.)

First, Valerie Williams, AEA365 blog, April18, 2012 says, “…Many environmental education programs struggle with the question of whether environmental education is a means to an end (e.g. increased stewardship) or an end itself. This question has profound implications for how programs are evaluated, and specifically the measures used to determine program success.”

I think that many educational programs (whether environmentally focused or not) struggle with this question.  Is the program a means to an end or the end itself?  I am reminded of programs which are instituted for cost savings and then the program designers want that program evaluated.  Means or end?

Williams also offers comments about evaluability assessment–that evaluation task that helps evaluators decide whether to evaluate a new programs, especially if that new program’s readiness for evaluation is in question. (She provides resources if you are interested.)  She offers reasons for conducting an evaluability assessment.  Specifically:

  • Surfacing disagreements among stakeholders about the program theory, design and/or structure;
  • Highlighting the need for changes in program design; and
  • Clarifying the type of evaluation most helpful to the program.

Evauability assessment is a topic for future discussion.

Second, a colleague offered the following CDC reference and says, “The purpose of this workbook is to help public health program managers, administrators, and evaluators develop an effective evaluation plan in the context of the planning process. It is intended to assist in developing an evaluation plan but is not intended to serve as a complete resource on how to implement program evaluation.”  I offer it here because I know that evaluation plans are often added after the program has been implemented.  Although it has as a focus pubic health programs, one source familiar with this work commented that there is enough in the workbook that can be applied to a variety of settings.  Check it out; the link is below

 

Next, Nigerian novelist Chimamanda Ngozi Adichie is quoted as saying, “The single story creates stereotypes, and the problem with stereotypes is not that they are untrue, but that they are incomplete. They make one story become the only story.”

Given that

  • Extension uses story to evaluate a lot of programs; and
  • Story is used to convince legislators of Extension’s value; and
  • Story, if done right, is a powerful tool;

Then it behooves us all to remember this–are we using the story because it captures the effect or because it is the only story?  If only story, is it promoting a stereotype?  Adichie, though a novelist, may be an evaluator at heart.

Finally, there is this quote, also from an AEA365 blog (Steve Mayer) “There are elements of Justice and Injustice everywhere – in society, in reform efforts, and in the evaluation of reform efforts. The choice of outcomes to be assessed is a political act. “Noticing progress” probably takes us further than “measuring impact,” always being mindful of who benefits.”

We often are stuck on “measuring impact”; after all, isn’t that what everyone wants to know.  If world peace is the ultimate impact, then what is the likelihood of measuring that?  I think that “noticing progress” (i.e., change) will take us much further because of the justice it can capture (or not–and that is telling).  And by capturing “noticing progress”, we can make explicit who benefits.

This runs long today.

 

Other comments about my blog

Posted on by

I wonder (as y’all know) if anyone reads this; if the blog makes a difference; and should I keep writing (because blogging is hard work).

Over the last two weeks, I’ve received over 50 comments about my posts, from folks who are not subscribed and who read the post.  I don’t know if their search engine has optimized my blog so it pops up or if they are really interested in evaluation.  Some of the comments appear genuine; some seem specious at best.  Please know I read them all.  And I appreciate the feedback.  There were some questions posted in the comments.  Here are some answers, not in any particular order.

  1. AEA365 is a blog sponsored by the American Evaluation Association.  It invited known evaluators who blog (like me) to contribute a post to their AEA365.  Susan Kistler is AEA’s executive director; she has really good ideas.  I wouldn’t be surprised if this was one of them.
  2. To be an evaluator who blogs, you first need to be an evaluator.  You get to be an evaluator by studying evaluation.  There are numerous places to do that–universities, Evaluator’s Institute, AEA’s summer institute, on the job training.  I went to university; I got a Ph.D in program evaluation.  Most people who come to evaluation come through some social science–sociology, psychology, social work, anthropology, other disciplines.  If you want to know more, I’ll be happy to elaborate in a future blog.
  3. When I preview my post, the graphics look fine.  I don’t have to click on them more than once; they just are there.  My IT person says it might be the browser being used.  I use Firefox; I am a PC user.  I don’t know how this looks on a Mac.
  4. Although I try to stay off my political soap box when I post, there are times where the topic (Viktor Frankl, for example) is both political and evaluative.  For those of you new to evaluation, evaluation is a political discipline.  I have a few passions in my life that I return to again and again as they have been with me for a long time (some as long as 50 years).  Evaluatiion is one of those passions even though I’ve been a professional evaluator for only 30 years. (I’ve probably been a lay evaluator for as long as I’ve known my passions.)
  5. 500 words seems to be a good length.

I’m working with my IT person to make this blog better.  Since I’m a technopeasant, learning something new is hard work for me.  Next week I’ll talk about evaluation again.  I promise.  Hopefully, y’all will see the sun where you are. Here in Oregon, we are eager for the sun.   Even if it is the sunset in Florida.

Making a difference?

Posted on by

Today I’m reporting the results of the survey I ran for two weeks.

I asked five questions:

  1. Is this blog making a difference in what you do?  (answer yes or no)
  2. In which of the following ways is the blog making a difference.  (choose all that apply)
  3. What is your opinion about the length?  (approximately 500 words–too long, too short, just right)
  4. How often do you read the blog?  (weekly; regularly, depending on topic; rarely; archive only; never)
  5. What topic would you like to see discussed (various with an option for other)

I don’t know how many subscribe (I am a technopeasant, after all) and that I blog at all is close to miraculous, so the results I report may or may not be reflective of what is actually happening.

So what are the results?

1.  Of those 22 people responding, 21 people (100% of those responding; one person skipped this question) said that the blog is making a difference in what they do.

2.  Of those 22 people responding, 15 (68.2%) said that they get new ideas; 15 (68.2%) said that they get new perspectives; 8 (36.4%) said that they get old information clarified;  13 (59.1%) said that they learn new information; and 11 (50.0%) said they review previously learned information.  No one responded that the blog has not made a difference. [Phew…:) ] [Keep in mind that percentages will not add to 100% because multiple responses could be selected.)

3.  Everyone who responded (N=22; 100%) said that 500 words is just right in length.

4.  When respondents were asked how often the blog was read by them, 4 (18.2%) said weekly, as it is posted; 17 (77.3%) said regularly, depending on topic; 1 (4.5%) said rarely [although they obviously read it to respond…:) ].

5.  When asked which topic they would like to see addressed in future blogs, 16 (72.7%) said methodology; 9 (40.9%) said quantitative data analysis; 12 (54.5%) said qualitative data analysis; 10 (45.5%) said data collection methods; 15 (68.2%) said survey development; 8 (36.4%) said program evaluation theory; and 8 (36.4%) said program evaluation models.  Three people (13.6%) offered comments.  [Keep in mind that percentages will NOT add to 100% because multiple responses could be selected.]

Comments were about the graphics (hard to read); could be eliminated.  I’ll talk to my tech people about that.  On my reading the graphics are clear.  May be the browser; may be something else, says the technopeasant.  The other comment was about getting new ideas even though the ideas have not been implemented yet.

 

So what does this tell me–given the small sample size, I am cautiously optimistic.   (If I find out  how many are subscribed, I’ll let y’all know.)   I will continue to blog.  I will figure out other ways to determine if I’m making a difference.  And thanks for all of you who took the time to answer the survey and all of you who take the time to read my musings.  Of the several things about which I am passionate, evaluation is close to the top.  (oh, and no graphics this time…)

 

 

Do you know what test to run?

Posted on by

I had a conversation (ok–an electronic conversation) with colleagues a few weeks ago.  The conversation was about the use of inappropriate analyses for manuscripts being submitted.  The specific question raised was about the t-test and went something like this:

Should a t-test be used on a sample that has not been RANDOMLY drawn from a population.  If the sample is not randomly drawn, or if the entire population is used (as is often the case with Extension evaluations), then a t-test is NOT appropriate.

What exactly is the appropriate test?  First, one has to identify if the underlying assumptions have been met; if not, a parametric, i.e., t–test, is not appropriate.

So what exactly are the assumptions underlying the use of a t-test?

Glass and Stanley (1970, pg. 297) [a classic statistics text in the field of education and psychology and there may be a more recent edition than the one I have on my shelf] say that for dependent samples (the kind most often used by Extension professionals ), the sample:

  1. is normally distributed;
  2. has homogeneous variances (i.e., same spread); and
  3. is RANDOMLY drawn from a population.

Some would add that the scale of measurement needs to result in interval or ratio data (not nominal–ordinal is a questionable case) and have a sample size of 30 or over.

This presents a quandary, to say the least.  Extension professionals know that journals want a probability value (if the study is quantitative) and how do you get a probability value with out a t-test?

Answer:  Use a nonparametric equivalent test.

The nonparametric equivalent for a test of dependent means is a Wilcoxon matched pair test.

Marascuilo and McSweeney (1977, pg 5) say that that the researcher needs to select “…a test for which the power of rejection is maximized when the hypothesis is tested false.”  They go on to say, “If the data adhere to the assumptions required for a classical, normally based t or F test, a researcher would be foolish…not (to) use them, since they are optimum, when justified.”  Justification is the key here.  And the justification is: Do the data meet the assumptions for the test?

Their bottom line is that a researcher should never think that a nonparametric test is exclusively a substitute for a parametric test.  It is not.  Use the right test for the hypothesis being tested.  It may be (probably is) a nonparametric test.

 

Citations

Glass, G. V., & Stanley, J. C. (1970). Statistical Methods in Education and Psychology.  Englewood Cliffs, NJ: Prentice-Hall, Inc.

Marascuilo, L. A. & McSweeney, M. (1977).  Nonparametric and Distribution-free Methods for the Social Sciences. Monterey, CA: Brooks/Cole Publishing.

 

 

Please complete the survey.

Posted on by

Before Spring break, I blogged about making a difference.  I realize that many who subscribe to this blog were on break last week when the post came.  So I’m sending an extra post this week:  PLEASE COMPLETE THE SURVEY that was posted through an imbedded hyperlink in the post two weeks ago.  I plan to close the survey on Friday, COB.  The URL if the link above doesn’t work is http://www.surveymonkey.com/s/ZD33HFS.  You can copy and paste the URL into your browser.  PLEASE…:)

Is this blog making a difference?

Posted on by

An important question that evaluators ask is, “What difference is this program making?”  Followed quickly with, “How do you know?”

Recently, I happened on a blog called {grow} and the author, Mark Schaefer,  had a post called, “Did this blog make a difference?”  Since this is a question as an evaluator I am always asking, I jumped on the page.  Mr. Schaefer is in marketing and as a marketing expert he says the following, “You’re in marketing for one reason: Grow. Grow your company, reputation, customers, impact, profits. Grow yourself. This is a community that will help. It will stretch your mind, connect you to fascinating people, and provide some fun along the way.”  So I wondered how relevant this blog would be to me and other evaluators whether they blogged or not.

Mr. Schaefer is taking stock of his blog–a good thing to do for a blog that has been posted for a while.  So although he lists four innovations, he asks the reader to “…be the judge if it made a difference in your life, your outlook, and your business.”  The four innovations are

  1. Paid contributing columnists.  He actually paid the folks who contributed to his blog; not something those of us in Extension can do.
  2. {growtoons}. Cartoons designed specifically for the blog that “…adds an element of fun and unique social media commentary.”  Hmmm…
  3. New perspectives. He showcased fresh deserving voices; some that he agreed with and some that he did not.  A possibility.
  4. Video. He did many video blogs and that gave him the opportunity to “…shine the light on some incredible people…”  He interviews folks and posts the short video.  Yet another possibility.

His approach seems really different to what I do.  Maybe it is the content; maybe it is the cohort; maybe it is something else.  Maybe there is something to be learned from what he does.  Maybe this blog is making a difference.  Only I don’t know.  So, I take a clue from Mr. Schaefer and ask you to judge if it has made a difference in what you do–then let me know.  I’ve imbedded a link  to a quick survey that will NOT link to you nor in anyway identify you.  I will only be  using the findings for program improvement.  Please let me know.  Click here to link to the survey.

 

Oh, and I won’t be posting next week–spring break and I’ll be gone.