Jun
23
Filed Under (program evaluation) by Molly on 23-06-2016 and tagged , , , , ,

We close every rehearsal and concert with the song, “Be the change”. Using the words from Gandhi, I try to remember to make a difference;difference 2 to be the change I want to see in the world.

That is not easy. I ride my bike all the time. (Yep. Really.) I compost. I grow my own vegetables in the summer and support my farmers’ market and CSA (both of which, thankfully, run through Thanksgiving). But I ask my self, “Am I making a difference?” make a difference Read the rest of this entry »

Jun
15
Filed Under (Methodology) by Molly on 15-06-2016

It has been almost a month since I last blogged. When I last blogged, I talked about evaluation history. That blog was a bunny path from what I had been talking about: methodology. I was talking about the implementationimplementation, monitoringmonitoring-2, and delivery deliveryof interventions which are to be evaluated. Another methodology I want to talk about is case study. I did go through the archives to locate the blogs relating to case study. They are below.

http://blogs.oregonstate.edu/programevaluation/2015/01/15/blogging-case-study/

http://blogs.oregonstate.edu/programevaluation/2010/04/13/other-ways-to-gather-information-the-case-study/

http://blogs.oregonstate.edu/programevaluation/2013/06/12/causation/

http://blogs.oregonstate.edu/programevaluation/2013/06/07/one-of-the-5cs-clarity/ Read the rest of this entry »

May
26
Filed Under (program evaluation) by Molly on 26-05-2016 and tagged , , , ,

Having written about evaluation history previously, I identified  those who contributed, not those who could be called evaluation pioneers; rather those who had influenced my thinking.  I think it is noteworthy to mention those evaluation pioneers who set the field on the path we see today, those whom I didn’t mention and need to be. As a memorial (it is Memorial Day weekend Memorial-Day-weekend, after all), Michael Patton (whom I’ve mentioned previously) is coordinating an AEA365 to identify and honor those evaluation pioneers who are no longer with us. (Thank you, Michael). The AEA365 link above will give you more details.  I’ve also linked the mentioned evaluation pioneers that have been remembered. Some of these pioneers I’ve mentioned before; all are giants in the field; some are dearly loved as well. All those listed below have died. Patton talks about the recent-dead, the sasha, and the long-dead, the zamani. He cites the Historian James W. Loewen when he makes this distinction. Some of the listed are definitely the sasha (for me); some are zamani (for me). Perhaps photos will help (for whom photos could be found) and dates. There are Read the rest of this entry »

May
13
Filed Under (program evaluation) by Molly on 13-05-2016

Mistakes are a great educator when one is honest enough to admit them and willing to learn from them.

– Alexander Solzhenitsyn

Even after 30+ years of evaluation, I make mistakes mistakes3 . It may be a mistake that occurs in the planning and modeling; it may be a mistake that occurs in the implementation, monitoring, and delivery; or a mistake in data management (qualitative or quantitative); or more than likely, a mistake in the use of the findings. mistakes1

Probably the biggest mistake mistakes4I have ever made was making an assumption assumption at the planning stage. Read the rest of this entry »

May
06
Filed Under (Methodology) by Molly on 06-05-2016

Focus groupsfocusgroups  are a wonderful data gathering collection methodology. Not only are there different skills to learn for interviewing, analysis gives you the opportunity to explore qualitative data analysis. (It is all related after all.)

Now, I will confess that I’ve only ordered the 5th edition of the Krueger and Casey book (I don’t have it). I’m eager to see what is new. So I’ll settle for the 4th edition and try and regale you with information you may not know. (I will talk in a future post about the ways virtual focus groups data base 2 are envisioned.)

Focus group describes (although sometimes incorrectly) a variety of group processes. Krueger and Casey give the reader a sense of to what to pay attention and to what is based on faulty data. So starting at the beginning, let’s look at an overview of what exactly is a focus group.

Groups are experiences that affect the individual throughout life and are used for planning, decision making, advising, learning, sharing, self-help, problem solving, among others. Yet group membership often leaves the individual Read the rest of this entry »

May
03
Filed Under (Methodology) by Molly on 03-05-2016

Previously, I talked about Survey’s (even though I posted it April 27, 2016). Today, I’ll collect all the posts about focus groups and add a bit more.

2010/01/05 Talks about the type of questions to use in a Focus Group

2010/01/27 One of three topics mentioned

2010/09/09 Talks about focus groups in terms of analyzing a conversation

2011/05/31 Talks about focus groups in the context of sampling

2011/06/23 Mentions Krueger, my go to

2013/11/15 Mentions focus groups

2014/10/23  Mentions focus groups and an individual with information

2015/02/11 Mentions focus groups…

2015/05/08 Virtual focus groups

Discovery

Although focus groups are a mentioned throughout many of my posts, there are few that are exclusively devoted to focus groups. That surprises me. I need to talk more about focus groups. I especially need to talk about what I found when I did the virtual focus groups, more than with the specific post. From the interest at AEA last year, there needs to be much discussion.

So OK. More about focus groups.

Although Dick  Krueger dick-1997 is my go to reference for focus groups krueger 4th ed(I studied with him, after all), there are other books on focus groups. (I just discovered that Krueger and Casey have also revised and published a 5th edition.) krueger 5th edition

The others for example (in no particular order),

  1. Stewart, D. W. & Shamadasani, P. N. David Stewart Focus groups 1 ed(1990). Focus groups: Theory and practice. Newbury Park, CA: Sage Publications. There is a 3rd edition of this book available David Stewart Focus groups
  2. Morgan, D. L. (ed.)david morgan focus groups (1993). Successful focus groups: Advancing the state of the art. Newbury Park, CA: Sage Publications.
  3. Greenbaum, T. L. greenbaum-focus groups (2000). Moderating focus groups. Thousand Oaks, CA: Sage Publications.
  4. Greenbaum, T. L. (2nd edition). Greenbaum-focus group research(1998). The handbook for focus group research. Thousand Oaks, CA: Sage Publications.
  5. Carey, M. A. & Asbury, J-E. Carey book cover(2012). Focus group research. Walnut Creek, CA: Left Coast Press, Inc.

Plus many others that are published by Sage, available from Amazon, and others. I think you can find one that works for you.

Mary Marczak and Meg Sewell have an introduction to focus groups here (it is shorter that reading the book by Krueger and Casey).

I think it is important to remember that focus groups:

  1. Yield qualitative data;
  2. Are used in evaluation (just not in a pre-post sense);
  3. Are a GROUP activity of people who are typically unfamiliar with each other.

Next time: More on focus groups.

my two cents.

molly.

Apr
27

NOTE: This was written last week. I didn’t have time to post. Enjoy.

 

Methodologymethodology 2, aka implementationimplementation, monitoringmonitoring-2, and deliverydeliveryis important. What good is it if you just gather the first findings that come to mind. Being rigorous here is just as important as when you are planning and modeling the program. So I’ve searched the last six years of blogs posts and gathered some of them for you. They are all about Survey, a form of methodology.survey image 3 Survey is a methodology that is often used by Extension, as it is easy to use. However, organizing the surveysurvey organization, getting the survey’s backsurvey return, and dealing with non-response are problematicnonresponse (another post, another time).

The previous posts are organized by date from the oldest to the most recent:

 

2010/02/10

2010/02/23

2010/04/09

2010/08/25

2012/08/09

2012/10/12

2013/03/13

2014/03/25

2014/04/15

2014/05/19

2015/06/29

2015/07/24

2015/12/07

2016/04/15

2016/04/21 (today’s post isn’t hyperlinked)

Just a few words on surveys today: A colleague asked about an evaluation survey for a recent conference. It will be an online survey probably using the University system, Qualtrics. My colleague jotted down a few ideas. The thought occurred to me that this book (by Ellen Taylor-Powell and Marcus Renner) would be useful. On page ten of this book, it asks for the type of information that is needed and wanted. It lists five types of possible information:

  1. Participant reaction (some measure of satisfaction);
  2. Teaching and facilitation (strengths and weaknesses of the presenter, who may (or may not) change the next time);
  3. Outcomes (what difference/benefits/intentions did the participant experience);
  4. Future programming (other educational needs/desires); and
  5. Participant background (who is attending and who isn’t can be answered here).

Thinking through these five categories made all the difference for my colleague. (Evaluation was a new area.) I had forgotten about how useful this booklet is for people being exposed to evaluation for the first time and to surveys, as well. I recommend it.

The WECT program arbitrarily divided the WECT program into four parts. Those “modules” are:

  • Program Planning and Logic Modeling;
  • Program Implementation, Monitoring, and Delivery;
  • Data Management and Analysis (divided into Qualitative data and Quantitative data); and
  • Program Evaluation Utilization

Read the rest of this entry »

Apr
01
Filed Under (program evaluation) by Molly on 01-04-2016

“Speak your mind, even if your voice shakes.”

~~Maggie Kuhnmaggie_kuhn, founder of the Gray Panthers, 1970 .

The Gray Panthersgray panthers is a group of people advocating for the rights of oldsters (among other things). Aging is the brunt of many jokes. At least in the US. Unfortunately.

A long time friend of mine says  aging. If you get there, it takes as long as it takes. (Some people say the years wiz by as the days drag on and on and on…I know this first hand.)

Another long time friend relayed the NPR story about aging, which says anchovies, rosemary, vino, and leisure are the answers. Now I’m not saying that anchovies, rosemary, vino, and leisure are the reason evaluation as a discipline has come as far as it has in the last 50+ years; I’m just saying that perhaps we need to look a little deeper than just the surface. I think Maggie Kuhn says it clearly:  “Speak your mind, even if your voice shakes.”

Stand up for what you believe! (even if your voice shakes).

I believe that evaluation makes a difference.

I believe that there is a need for evaluation.

I believe that y’all do evaluation every day in the choices you make.choice

No one said making choices would be easy…even getting out of bed in the morning! If make a choice, you take a chance; if you take a chance your life will change. choice-3

So how will you stand up today? What choice will you make? Speak your mind unambiguously!

New Topic: I learned today that Will Shadish died on March 27, 2016sadish-2

Will was very active as a quantitative psychologist and an evaluator. We served AEA together.  I will miss him.

my two cents

molly.

 

 

Mar
24
Filed Under (program evaluation) by Molly on 24-03-2016 and tagged , , , ,

I am a social scientist. I look for the social in the science of what I do.

I am an evaluator as a social scientist. I want to determine the merit, worth, value of what I do. I want to know that the program I’m evaluating (or offering) made a difference. (After all, the root of evaluation is value.)

Keeping that in mind has resulted (over the years) in the comment, “no wonder she is the evaluator” when I ask an evaluative question. So I was surprised when I read a comment by a reader that implied that it didn’t matter. The reader said, “The ugly truth is, it does not matter if it makes a difference. Somewhere down the road someone will see your post and may be it will be useful for him.” (Now you must know that I’ve edited the comment, although the entire comment doesn’t support my argument:  Evaluators need to know if the program made a difference.)

So the thought occurred to me, what if it didn’t make a difference? What if the program has no value? No worth? No merit? What if by evaluating the program you find that it won’t be useful for the participant? What does that say about you as an evaluator? You as a program designer? You as an end user? Is it okay for the post to be useful “somewhere down the road”? Is blogging truly “a one way channel to transfer any information you have over the web.” How long can a social-scientist-always-looking-at-the-social continue to work when the information goes out and rarely comes back? I do not know. I do know that blogging is hard work. After six and one-half years of writing this blog almost weekly,  writer’s block is my constant companion.writers-block 2 (although being on a computer, I do not have a pile of paper, just blank screens). So I’m turning to you, readers:

Does it make a difference whether I write this blog or not?

Am I abdicating my role as an evaluator when I write the blog?

I don’t know. Over the years I have gotten some interesting comments (other than the “nice job” “keep up the work” types of comments). I will pause (not in my writing; I’ll continue to do that) and think about this. After all, I am an evaluator wanting to know what difference this program makes.

my two cents.

molly.