My friend, Susan, who promised instructions on how to use Excel to select a sample, wrote a post on that very topic. 
She tells me that using screen shots made providing instructions easier so she posted it in a tutorial. Thank you, Susan, for adding this information. Now there is no reason for not selecting a random sample from your large population. Whether the sample will respond or not is out of your control.
Response rates are another thing, to be covered later.
A reader commented that I need to be attending to my analytics not just reading my comments. Hmmm…
My question is: what do analytics tell me about making a difference–by providing an educational forum that changes people am I making a difference? Keep in mind that I am an evaluator and that the root for the word evaluation is “value”. So I wonder, do the analytics tell me about the merit, worth, value of this educational intervention?
What will the analytics really tell me about the readers? What will the comments tell me that the analytics don’t? Will the analytics tell me what difference this blog has made in the readers. Will analytics tell me about intention to change? How will analytics help me write posts to which more people will respond; make me more of an authority in my posting?
I DO NOT KNOW.
If someone, any one out in cyber space knows the answers (readers?), I’d love to hear from you. I blog weekly; sometimes more than weekly (like this week because, although I had the post written, I didn’t get it posted before I left the office so I posted it when I came back). I check my blog regularly for comments. I approve those which provide thoughtful meaningful responses for other readers as well as for me.
Another reader suggests that I look at the number of readers who have established an RSS feed or established a subscription. Hmmm…Not sure what that will tell me. I’ll talk to the IT folks for an answer to that question.
I would certainly appreciate any thoughts from readers.
A reader asked how to choose a sample for a survey. Good question.
My daughters are both taking statistics (one in college, one in high school) and this question has been mentioned more than once. So I’ll give you my take on sampling. There are a lot of resources out there (you know, references and other sources). My favorite is in Dillman 3rd edition, page 57. 
Sampling is easier than most folks make it out to be. Most of the time you are dealing with an entire population. What, you ask, how can that be?
You are dealing with an entire population when you survey the audience of a workshop (population 20, or 30, or 50). You are dealing with a population when you deal with a series of workshops (anything under 100). Typically, workshops are a small number; only happen once or twice; rarely include participants who are there because they have to be there. If you have under 100, you have an entire population. They can all be surveyed.
Now if your workshop is a repeating event with different folks over the offerings, then you will have the opportunity to sample your population because it is over 100 (see Dillman, 3rd edition, page 57). If you have over 100 people to survey AND you have contact information for them, then you want to randomly sample from that population. Random selection (another name for random sampling) is very different from random assignment; I’m talking about random sampling.
Random sampling is a process where everyone gets an identification number (and an equal chance to be selected), sequentially; so 1- 100. Then find a random number table; usually found in statistic books in the back. Close your eyes and let your hand drop onto a number. Let’s say that number is 56997. You know you need numbers between 1 and 100 and you will need (according to Dillman) for a 95% confidence level with a plus or minus 3% margin of error and a 50/50 split at least 92 cases (participants) OR if you want an 80/20 split, you will need 87 cases (participants). So you look at the number and decide which two digit number you will select (56, 69, 99, 0r 97). That is your first number. Let us say you chose 99 that is the third two digit number found in the above random number (56 and 69 being the first two). So participant 99 will be on the randomly selected (random sampling) list. Now you can go down the list, up the list, to the left or the right of the list and identify the next two digit number in the same position. For this example, using the random numbers table from my old Minium (for which I couldn’t find a picture since it is OLD) stat book (the table was copied from the Rand Corporation, A million random digits with 100,000 normal deviates, Glencoe, IL: The Free Press, 1955), the number going right is 41534, I would choose participant number 53. Continuing right, with the number 01953, I would choose participant number 95, etc. If you come across a number that you have already chosen, go to the next number. Do this process until you get the required number of cases (either 92 or 87). You can select fewer if you want a 10% plus or minus margin of error (49, 38) or a 5% plus or minus margin of error (80, 71). (I always go for the least margin of error, though.) Once you have identified the required number, drafted the survey, and secured IRB approval, you can send out the survey. We will talk about response rates next week.
The question of surveys came up the other day. Again.
I got a query from a fellow faculty member and a query from the readership. (No not a comment; just a query–although I now may be able to figure out why the comments don’t work.)
So surveys; a major part of evaluation work. (My go-to book on surveys is Dillman’s 3rd edition ; I understand there is a 4th edition coming later this year.
)
After getting a copy of Dillman for your desk, This is what I suggest: Start with what you want to know.
This may be in the form of statements or questions. If the result is complicated, see if you can simplify it by breaking it into more than one statement or question. Recently, I got a “what we want to know” in the form of complicated research questions. I’m not sure that the resulting survey questions answered the research questions because of the complexity. (I’ll have to look at the research questions and the survey questions side by side to see.) Multiple simple statements/questions are easier to match to your survey questions, easier to see if you have survey questions that answer what you want to know. Remember: if you will not use the answer (data), don’t ask the question. Less can actually be more, in this case, and just because it would be interesting to know doesn’t mean the data will answer your “what you want to know” question.
Evaluators strive for evaluation use . (See: Patton, M. Q. (2008). Utilization Focused Evaluation, 4ed. Thousand Oaks, CA: Sage Publications, Inc.
; AND/OR Patton, M. Q. (2011). Essentials of Utilization-Focused Evaluation. Thousand Oaks, CA: Sage Publications, Inc.
). See also the The Program Evaluation Standards , which lists utility (use) as the first attribute and standard for evaluators. (Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The Program Evaluation Standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.
)
Evaluation use is related to stated intention to change about which I’ve previously written. If your statements/questions of what you want to know will lead you to using the evaluation findings, then stating the question in such a way as to promote use will foster use, i.e., intention to change. Don’t do the evaluation for the sake of doing an evaluation. If you want to improve the program, evaluate. If you want to know about the program’s value, merit, and worth, evaluate. Then use. One way to make sure that you will follow-through is to frame your initial statements/questions in a way that will facilitate use. Ask simply.
I’ve just read Ernie House’s book, Regression to the Mean. 
It is a NOVEL about evaluation politics. A publishers review says, “Evaluation politics is one of the most critical, yet least understood aspects of evaluation. To succeed, evaluators must grasp the politics of their situation, lest their work be derailed. This engrossing novel illuminates the politics and ethics of evaluation, even as it entertains. Paul Reeder, an experienced (and all too human) evaluator, must unravel political, ethical, and technical puzzles in a mysterious world he does not fully comprehend. The book captures the complexities of evaluation politics in ways other works do not. Written expressly for learning and teaching, the evaluation novel is an unconventional foray into vital topics rarely explored.”
Many luminaries (Patton, Lincoln, Scriven, Weiss) made pre-publication comments. Although I found the book fascinating, I found the quote that is included attributed to Freud compelling. That quote is, “The voice of the intellect is a soft one, but it does not rest until it has gained a hearing. Ultimately, after endless rebuffs, it succeeds. This is one of the few points in which we can be optimistic about the future of mankind (sic).” Although Freud wasn’t speaking about evaluation, House contends that this statement applies, and goes on to say, “Sometimes you have to persist against your emotions as well as the emotions of others. None of us are rational.”
So how does rationality fit into evaluation. I would contend that it doesn’t. Although the intent of evaluation is to be objective, none of us can be because of what I called personal and situational bias; what is known in the literature as cognitive bias. I contend that if one has cognitive bias (and everyone does) then that prevents us from being rational, try as we might. Our emotions get in the way. House’s comment (above) seems fitting to evaluation–evaluators must persist against personal emotions as well as emotions of others. I would add persists against personal and situational bias. I believe it is important to make explicit the personal and situational bias prior to commencing an evaluation. By clarifying assumptions that occur with the stakeholders and the evaluator, surprises are minimized, and the evaluation may be more useful to program people.
Warning: This post may contain information that is controversial .
Schools (local public schools) were closed (still are).
The University (which never closes) was closed for four days (now open).
The snow kept falling and falling and falling. 
(Thank you Sandra Thiesen for the photo.)
Eighteen inches. Then freezing rain. It is a mess (although as I write this, the sun is shining, and it is 39F and supposed to get to 45F by this afternoon).
This is a complex messy system (thank you Dave Bella). It isn’t getting better. This is the second snow Corvallis has experienced in the same number of months, with increasing amounts.
It rains in the valley in Oregon; IT DOES NOT SNOW.
Another example of a complex messy system is what is happening in the UK. 
These are examples extreme events; examples of climate chaos.
Evaluating complex messy systems is not easy. There are many parts. If you hold constant one part, what happens to the others? If you don’t hold constant one part, what happens to the rest of the system?. Systems thinking and systems evaluation has come of age with the 21st century; there were always people who viewed the world as a system; one part linked to another, indivisible. Soft systems theory dates back to at least von Bertalanffy who developed general systems theory and published the book by the same name in 1968
(ISBN 0-8076-0453-4).
One way to view systems is in this photo (compliments of Wikipedia) 
.
Evaluating systems is complicated and complex.
Bob Williams, along with Iraj Imam, edited the volume Systems Concepts in Evaluation
(2007), and along with Richard Hummelbrunner, wrote the volume Systems Concepts in Action: A Practitioner’s Toolkit 
(2010). He is a leader in systems and evaluation.
These two books relate to my political statement at the beginning and complex messy systems. According to Amazon, the second book “explores the application of systems ideas to investigate, evaluate, and intervene in complex and messy situations”.
If you think your program works in isolation, think again. If you think your program doesn’t influence other programs, individuals, stakeholders, think again. You work in a complex messy system. Because you work in a complex messy system, you might want to simplify the situation (I know I do); only you can’t. You have to work within the system.
Might be worth while to get von Bertalanffy’s book; might be worth while to get Williams books; might be worth while to get a copy of Gunderson and Holling book Panarchy: Understanding Transformations in Systems of Humans and Nature.
After all, nature is a complex messy system.
On February 1 at 12:00 pm PT, I will be holding my annual virtual tea party. This is something I’ve been doing since February of 1993. I was in Minnesota and the winter was very cold, and although not as bleak as winter in Oregon, I was missing my friends who did not live near me. I had a tea party for the folks who were local and wanted to think that those who were not local were enjoying the tea party as well. So I created a virtual tea party. At that time, the internet was not available; all this was done in hard copy (to this day, I have one or two friends who do not have internet…sigh…). Today, the internet makes the tea party truly virtual–well the invitation is; you have to have a real cup of tea where ever you are.
How is this evaluative? Gandhi says that only you can be the change you want to see…this is one way you can make a difference. How will you know?
I know because my list of invitees has grown exponentially. And some of them share the invitation. They pass it on. I started with a dozen or so friends. Now my address list is over three pages long. Including my daughters and daughters of my friends (maybe sons, too for that matter…)
Other ways: Design an evaluation plan; develop a logic model; create a metric/rubric. Report the difference. This might be a good place for using an approach other than a survey or Likert scale. Think about it.
Did you know that there are at least 11 winter holidays besides Christmas–many of them related to light or the return of light.
One needs evaluation tools to determine the merit or worth, to evaluate the holiday’s value to you. For me, any that return light
are important. So for me, there is Hanukkah 
(and eight candles), Solstice 
(and bonfires and yule logs), Christmas 
(and Advent wreaths with five candles), Kwanzaa 
( and kinara seven candles). Sometimes Diwali 
falls late in November to be included (it is the ancient Hindu festival of lights that is a movable feast like Hanukkah).
I have celebrations for Hanukkah (I have several menorahs), for Solstice (I have two special candelabra 
that holds 12 candles–a mini-bonfire to be sure), for Advent/Christmas (I make a wreath each year), and for Kwanzaa (a handmade Kinara). And foods for each celebration as well. Because I live in a multicultural household, it is important that everyone understand that no holiday is more important than any other–all talk about returning light (literal or figurative). Sometimes the holidays over lap–Hanukkah, Solstice, Christmas all in the same week…phew, I’m exhausted just thinking about it. Sometimes it seems hard to keep them separate–then I realized that returning the light is not separate; it is light returning. It is an evaluative task.
So well come the new born sun/son…the light returns. Evaluation continues.
Happy Holidays…all of them!
I’m taking two weeks holiday–will see you in the new year.
Variables.
We all know about independent variables, and dependent variables. Probably even learned about moderator variables, control variables and intervening variables. Have you heard of confounding variables? Variables over which you have no (or very little) control. They present as a positive or negative correlation with the dependent and independent variable. This spurious relationship plays havoc with analyses, program outcomes, and logic models. You see them often in social programs.
Ever encounter one? (Let me know). Need an example? Here is one a colleague provided. There was a program developed to assist children removed from their biologic mothers (even though the courts typically favor mothers) to improve the children’s choices and chances of success. The program had included training of key stakeholders (including judges, social service, potential foster parents). The confounding variable that wasn’t taken into account was the sudden appearance of the biological father. Judges assumed that he was no longer present (and most of the time he wasn’t); social service established fostering without taking into consideration the presence of the biological father; potential foster parents were not allerted in their training of the possibility. Needless to say, the program failed. When biologic fathers appeared (as often happened), the program had no control over the effect they had. Fathers had not been included in the program’s equation.
Reviews.
Recently, I was asked to review a grant proposal, the award would result in several hundred thousand dollars (and in today’s economy, no small change). The PI’s passion came through in the proposal’s text. However, the PI and the PI’s colleagues did some major lumping in the text that confounded the proposed outcomes. I didn’t see how what was being proposed would result in what was said to happen. This is an evaluative task. I was charged to with evaluating the proposal on technical merit, possibility of impact (certainly not world peace), and achievability. The proposal was lofty and meant well. The likelihood that it would accomplish what it proposed was unclear, despite the PI’s passion. When reviewing a proposal, it is important to think big picture as well as small picture. Most proposals will not be sustainable after the end of funding. Will the proposed project be able to really make an impact (and I’m not talking here about world peace).
Conversations.
I attended a meeting recently that focused on various aspects of diversity. (Now among the confounding here is what does one mean by diversity; is it only the intersection of gender and race/ethnicity? Or something bigger, more?) One of the presenters talked about how just by entering into the conversation, the participants would be changed. I wondered, how can that change be measured? How would you know that a change took place? Any ideas? Let me know.
Focus groups.
A colleague asked whether a focus group could be conducted via email. I had never heard of such a thing (virtual, yes; email, no). Dick Krueger and Mary Ann Casey only talk about electronic reporting in their 4th edition of their Focus Group book. 
If I go to Wikipedia (keep in mind it is a wiki…), there is a discussion of online focus groups. Nothing offered about email focus groups. So I ask you, readers, is it a focus group if it is conducted by email?
I had a topic all ready to write about then I got sick. I’m sitting here typing this trying to remember what that topic was, to no avail. That topic went the way of much of my recent memory; another day, perhaps.
I do remember the conversation with my daughter about correlation. She had a correlation of .3 something with a probability of 0.011 and didn’t understand what that meant. We had a long discussion of causation and attribution and correlation.
We had another long conversation about practical v. statistical significance, something her statistics professor isn’t teaching. She isn’t learning about data management in her statistics class either. Having dealt with both qualitative and quantitative data for a long time, I have come to realize that data management needs to be understood long before you memorize the formulas for the various statistical tests you wish to perform. What if the flood happens????
So today I’m telling you about data management as I understand it, because the 
did actually happen and, fortunately, I didn’t loose my data. I had a data dictionary.
Data dictionary. The first step in data management is a data dictionary. There are other names for this, which escape me right now…know that a hard copy of how and what you have coded is critical. Yes, make a back up copy on your hard drive…have a hard copy because the flood might happen. (It is raining right now and it is Oregon in November.)
Take a hard copy of your survey, evaluation form, qualitative data coding sheet and mark on it what every code notation you used means. I’d show you an example of what I do, only they are at the office and I am home sick without my files. So, I’ll show you a clip art instead…
. No, I don’t use cards any more for my data (I did once…most of you won’t remember that time…), I do make a hard copy with clear notations. I find my self doing that with other things to make sure I code the response the same way. That is what a data dictionary allows you to do–check yourself.
Then I run a frequencies and percentages analysis. I use SPSS (because that is what I learned first). I look for outliers, variables that are miscoded, and system generated missing data that isn’t missing. I look for any anomaly in the data, any humon error (i. e. my error). Then I fix it. Then I run my analyses.
There are probably more steps than I’ve covered today. These are the first steps that absolutely must be done BEFORE you do any analyses. Then you have a good chance of keeping your data safe.