Evaluation is an Everyday Activity « Program Evaluation Discussions

In a recent post, I said that 30 was the rule of thumb, i.e., 30 cases was the minimum needed in a group to be able to run inferential statistics and get meaningful results. How do I know, a colleague asked? (Specifically, “Would you say more about how it takes approximately 30 cases to get meaningful results, or a good place to find out more about that?”) When I was in graduate school, a classmate (who was into theoretical mathematics) showed me the mathematical formula for this rule of thumb. Of course I don’t remember the formula, only the result. So I went looking for the explanation. I found this site. Although my classmate did go into the details of the chi-square distribution and the formula computations, this article doesn’t do that. It even provides an Excel Demo for calculating sample size and verifying this rule of thumb. I am so relieved that there is another source besides my memory.

New Topic:

Continue reading →

Had a comment a while back on analyzing survey data…hmm…that is a quandary as most surveys are done on line (see Survey monkey, among others).

If you want to reach a large audience (because your population from which you sampled is large), you will probably use an on-line survey. The on-line survey companies will tabulate the data for you. Can’t guarantee that the tabulations you get will be what you want, or will tell you want you want to know. Typically (in my experience), you can get an Excel file which can be imported into a soft ware program and you can run your own analyses, separate from the on line analyses. Continue reading →

I had a comment about last week’s post on Sustainability and Evaluation. I will share it here. I wonder what you readers think of this comment:

In preventive health/health promotion, ‘sustainability’ has generally been used to indicate that the intervention program, or elements of it, or benefits of it, are continued beyond the life of the funded program. It is about extending the value of the investment in a program, beyond the life of the funded program. So it’s about a legacy, about the continuation of things achieved, about leaving things, circumstances or communities better off than when you first arrived (more empowered, more resourceful, more able to continue improvements on their own).

I wonder how that fits with the definitions I provided? Is this a different sustainability? Does it speak to the future generations? Does that include equity and justice? Continue reading →

To quote Annie Leonard, the word sustainability “gets thrown around all the time now and it’s not always clear what is intended.” She goes on to talk about the UN World Commission on Environment and Development definition of sustainable development as “…meeting the needs of the current generation without compromising the ability of future generations to meet their own needs.” That is a good definition, I think. Yet it is missing something which (according to Leonard) are equity and justice. Robert Gilman defines sustainability as “…equity over time”. She says (and I agree), quoting the Center for Sustainable Communities, that sustainability “consider(s) the whole instead of the specific. Sustainability emphasizes relationships rather than pieces in isolation.”

Now, given that evaluation to be effective must look at the whole (here is a good example of when the whole is greater than the sum of its parts); and

given that evaluation works to find out information that will benefit both the current and future generations; and

given that evaluation works to determine what difference was made in people’s lives, it seems to me that there is a relationship here that needs to be acknowledged.

A colleague of mine works in youth development and loves the job. My colleague has to determine the value of the program; the program needs to be evaluated. Yet, if the work is only for the program (i.e., the pieces in isolation) not the whole, what good is it that my colleague loves the job? The relationship between the youth involved and the bigger picture is truly more than can probably be captured in any evaluation. Still, the evaluation needs to be planned to consider that, even if the resources are limited (that is the “probably” above).

So yes, evaluation has something to learn from sustainability. Certainly sustainability can learn from evaluation (and economics, and equity, and ecology…).

NEW TOPIC

I’ve been, once again, getting comments about making a difference. I thought I’d post some of those comments (I’ve copied and pasted comments so the spelling is as it appears in the original text):

…every blog post makes a difference in a way or in another. You can answer at your questions just seeing how many comments are here, how many people are interested in answering you. I think you are a good person, and everything said by a good person is always a life’s lesson to keep in mind. Thank you for every helpful information, good job!
It may be a temporary difference – i.e. limited on the time, but of course that at least for some seconds your writing are touching the life’s of all your readers.
Every blog or article makes a difference to those who read it! They might strongly agree or disagree with what the blogger has wrote, making a difference by reafirming there opinion or being outraged that somebody else looks at ideas different to them! Keep writing Molly, you are making people think, which is always good
I think the best measure of the effectiveness of a blog are the number of shares it gets, as people that found something useful in it tend to want to share with others.
…I have written quite a bit about this topic and challenge that bloggers face and the bottom line is that you really can’t measure the value. Sure I think asking for responses like you did might help you see a bit of it, but the reality is 99.9% of people will never comment. As such, we as bloggers have to remember that each pageview is a real person who was on our site and who was impacted by what we wrote!
Blogs are probably the best tool for engaging a customer in todays times.

My question: are blogs engaging readers or are they only outreach, even if the blog is read?

P.S. I also got a lot of comments about my analytics post…for next time.

References:

Leonard, A. (2011). Story of Stuff. NY: Free Press. (good book–worth the read)

Report of the World Commission on Environment and Development. (1987). NY: UN World Commission on Environment and Development. http://www.un-documents.net/our-common-future.pdf

Gilman, R., Director, Context Institute.

He says: Sustainability is equity over time. As a value, it refers to giving equal weight in your decisions to the future as well as the present. You might think of it as extending the Golden Rule through time, so that you do unto future generations (as well as to your present fellow beings) as you would have them do unto you.

Center for Sustainable Communities is quoted in a variety of places: http://sustainablesonoma.org/keyconcepts/sustainability.html; http://isocs-sustainability.wikispaces.com, among others.

The entire definition is: Sustainability is part of a trend to…consider the whole instead of the specific. Sustainability emphasizes relationships rather than pieces in isolation…Sustainability is not about regressing to primitive living conditions. It is about understanding our situation, and developing as communities in ways that are equitable, and make sense ecologically and economically.

Having just read Harold Jarche’s April 27, 2014 blog, making sense of the network era, about personal knowledge mastery (PKM), I am once again reminded about the challenge of evaluation. I am often asked, “Do you have a form I could use about…?” My nutrition and exercise questions notwithstanding (I do have notebooks of those), this makes evaluation sound like it is routine, standardized, or prepackaged rather than individualized, customized, or specific. For me, evaluation is about the exceptions to the rule; how the evaluation this week may have similarities to something I’ve done before (after all this time, I would hope so…), yet is so different; unique, specific.

You can’t expect to find a pre-made form for your individual program (unless, of course you are replicating a previously established program). Evaluations are unique and the evaluation approach needs to match that unique program specialness. Whether the evaluation uses a survey, a focus group, or an observation (or any other data gathering approach), that approach to gathering data needs to focus on the evaluation question you want answered. You can start with “What difference did the program make?” Only you, the evaluator, can determine if you have enough resources to conduct the evaluation to answer the specific questions that result from what difference did the program make. You probably do not have enough resources to determine if the program led your target audience to world peace; you might have enough resources to determine if the intention to do something different is there. You probably have enough resources to decide how to use your findings. It is so important that the findings be used; use may be how world peace may be accomplished.

There are a few commonalities in data collection; those are the demographics, the data that tell you what your target audience looks like. Things like gender, age, marital status, education level, SES, probably a few other things depending on the program. Make sure when you ask demographic information that a “choose not to answer” option is provided in the survey. Sometimes you have to ask; observations don’t always provide the answer. You need to make sure you include demographics in your survey as most journals want to know what the target audience looked like.

Readers, what makes your evaluations different, unique, special? I’d like to hear about that. Oh and while you are at it…like and share this post, if you do.

My friend, Susan, who promised instructions on how to use Excel to select a sample, wrote a post on that very topic.

She tells me that using screen shots made providing instructions easier so she posted it in a tutorial. Thank you, Susan, for adding this information. Now there is no reason for not selecting a random sample from your large population. Whether the sample will respond or not is out of your control.

Response rates are another thing, to be covered later.

A reader commented that I need to be attending to my analytics not just reading my comments. Hmmm…

My question is: what do analytics tell me about making a difference–by providing an educational forum that changes people am I making a difference? Keep in mind that I am an evaluator and that the root for the word evaluation is “value”. So I wonder, do the analytics tell me about the merit, worth, value of this educational intervention?

What will the analytics really tell me about the readers? What will the comments tell me that the analytics don’t? Will the analytics tell me what difference this blog has made in the readers. Will analytics tell me about intention to change? How will analytics help me write posts to which more people will respond; make me more of an authority in my posting?

I DO NOT KNOW.

If someone, any one out in cyber space knows the answers (readers?), I’d love to hear from you. I blog weekly; sometimes more than weekly (like this week because, although I had the post written, I didn’t get it posted before I left the office so I posted it when I came back). I check my blog regularly for comments. I approve those which provide thoughtful meaningful responses for other readers as well as for me.

Another reader suggests that I look at the number of readers who have established an RSS feed or established a subscription. Hmmm…Not sure what that will tell me. I’ll talk to the IT folks for an answer to that question.

I would certainly appreciate any thoughts from readers.

(used with permission.)

After last week’s post on random sampling, I received a comment from a friend. She recommended some tools that might help calculate sample size especially when the population is different from the list Dillman offers. It is called Macorr. It has fields for the same variables that Dillman lists AND the population size can vary. Very important since Extension programs that repeat may not have a nice round number in the population.

She also says that you can calculate a random sample in Excel. She has to send me the directions on how to do that. I offer that to those of you who are much more adapt at Excel than I am.

When she sends it to me, I will post it. In the meantime, we wait or use the resources already available.

New Topic.

A comment was made: How important is it to have geographical representation in your random sample?

Theoretically, the random sample allows all individuals in a population to have the same chance of being in the sample. Because of that chance, there is also an excellent likelihood that the geographic representation will be distributed to represent the population. Of course, you have to decide before you sample, to what questions you want answers. If geographic areas may affect the outcome, then I would suggest the following. If you want to make sure that a particular area is represented (i.e., mixed metropolitan and rural areas), you can stratify on the type of representation you want. I’m doing this in an evaluation I will be undertaking this summer and fall. We hypothesize that the metropolitan areas are different from the mixed metropolitan/rural areas and both are different from the rural areas. The evaluation team stratified on the density question and are randomly selecting in the three areas. I’ll let you know how the stratification worked.

A reader asked how to choose a sample for a survey. Good question.

My daughters are both taking statistics (one in college, one in high school) and this question has been mentioned more than once. So I’ll give you my take on sampling. There are a lot of resources out there (you know, references and other sources). My favorite is in Dillman 3rd edition, page 57.

Sampling is easier than most folks make it out to be. Most of the time you are dealing with an entire population. What, you ask, how can that be?

You are dealing with an entire population when you survey the audience of a workshop (population 20, or 30, or 50). You are dealing with a population when you deal with a series of workshops (anything under 100). Typically, workshops are a small number; only happen once or twice; rarely include participants who are there because they have to be there. If you have under 100, you have an entire population. They can all be surveyed.

Now if your workshop is a repeating event with different folks over the offerings, then you will have the opportunity to sample your population because it is over 100 (see Dillman, 3rd edition, page 57). If you have over 100 people to survey AND you have contact information for them, then you want to randomly sample from that population. Random selection (another name for random sampling) is very different from random assignment; I’m talking about random sampling.

Random sampling is a process where everyone gets an identification number (and an equal chance to be selected), sequentially; so 1- 100. Then find a random number table; usually found in statistic books in the back. Close your eyes and let your hand drop onto a number. Let’s say that number is 56997. You know you need numbers between 1 and 100 and you will need (according to Dillman) for a 95% confidence level with a plus or minus 3% margin of error and a 50/50 split at least 92 cases (participants) OR if you want an 80/20 split, you will need 87 cases (participants). So you look at the number and decide which two digit number you will select (56, 69, 99, 0r 97). That is your first number. Let us say you chose 99 that is the third two digit number found in the above random number (56 and 69 being the first two). So participant 99 will be on the randomly selected (random sampling) list. Now you can go down the list, up the list, to the left or the right of the list and identify the next two digit number in the same position. For this example, using the random numbers table from my old Minium (for which I couldn’t find a picture since it is OLD) stat book (the table was copied from the Rand Corporation, A million random digits with 100,000 normal deviates, Glencoe, IL: The Free Press, 1955), the number going right is 41534, I would choose participant number 53. Continuing right, with the number 01953, I would choose participant number 95, etc. If you come across a number that you have already chosen, go to the next number. Do this process until you get the required number of cases (either 92 or 87). You can select fewer if you want a 10% plus or minus margin of error (49, 38) or a 5% plus or minus margin of error (80, 71). (I always go for the least margin of error, though.) Once you have identified the required number, drafted the survey, and secured IRB approval, you can send out the survey. We will talk about response rates next week.

The question of surveys came up the other day. Again.

I got a query from a fellow faculty member and a query from the readership. (No not a comment; just a query–although I now may be able to figure out why the comments don’t work.)

So surveys; a major part of evaluation work. (My go-to book on surveys is Dillman’s 3rd edition ; I understand there is a 4th edition coming later this year. )

After getting a copy of Dillman for your desk, This is what I suggest: Start with what you want to know.

This may be in the form of statements or questions. If the result is complicated, see if you can simplify it by breaking it into more than one statement or question. Recently, I got a “what we want to know” in the form of complicated research questions. I’m not sure that the resulting survey questions answered the research questions because of the complexity. (I’ll have to look at the research questions and the survey questions side by side to see.) Multiple simple statements/questions are easier to match to your survey questions, easier to see if you have survey questions that answer what you want to know. Remember: if you will not use the answer (data), don’t ask the question. Less can actually be more, in this case, and just because it would be interesting to know doesn’t mean the data will answer your “what you want to know” question.

Evaluators strive for evaluation use . (See: Patton, M. Q. (2008). Utilization Focused Evaluation, 4ed. Thousand Oaks, CA: Sage Publications, Inc.; AND/OR Patton, M. Q. (2011). Essentials of Utilization-Focused Evaluation. Thousand Oaks, CA: Sage Publications, Inc.). See also the The Program Evaluation Standards , which lists utility (use) as the first attribute and standard for evaluators. (Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The Program Evaluation Standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.)

Evaluation use is related to stated intention to change about which I’ve previously written. If your statements/questions of what you want to know will lead you to using the evaluation findings, then stating the question in such a way as to promote use will foster use, i.e., intention to change. Don’t do the evaluation for the sake of doing an evaluation. If you want to improve the program, evaluate. If you want to know about the program’s value, merit, and worth, evaluate. Then use. One way to make sure that you will follow-through is to frame your initial statements/questions in a way that will facilitate use. Ask simply.

Evaluation is an Everyday Activity

Program Evaluation Discussions

30 is the rule of thumb…

Survey data

Wednesday’s musings

Sustainability relates to evaluation

Routine vs. Exceptions

Excel and sampling

Making a difference–analytics?

More on Survey Sampling

Sampling

Surveys and other miscellanous information

Contact Info