survey development « Evaluation is an Everyday Activity

NOTE: This was written last week. I didn’t have time to post. Enjoy.

Methodology, aka implementation, monitoring, and deliveryis important. What good is it if you just gather the first findings that come to mind. Being rigorous here is just as important as when you are planning and modeling the program. So I’ve searched the last six years of blogs posts and gathered some of them for you. They are all about Survey, a form of methodology. Survey is a methodology that is often used by Extension, as it is easy to use. However, organizing the survey, getting the survey’s back, and dealing with non-response are problematic (another post, another time).

The previous posts are organized by date from the oldest to the most recent:

2016/04/21 (today’s post isn’t hyperlinked)

Just a few words on surveys today: A colleague asked about an evaluation survey for a recent conference. It will be an online survey probably using the University system, Qualtrics. My colleague jotted down a few ideas. The thought occurred to me that this book (by Ellen Taylor-Powell and Marcus Renner) would be useful. On page ten of this book, it asks for the type of information that is needed and wanted. It lists five types of possible information:

Participant reaction (some measure of satisfaction);
Teaching and facilitation (strengths and weaknesses of the presenter, who may (or may not) change the next time);
Outcomes (what difference/benefits/intentions did the participant experience);
Future programming (other educational needs/desires); and
Participant background (who is attending and who isn’t can be answered here).

Thinking through these five categories made all the difference for my colleague. (Evaluation was a new area.) I had forgotten about how useful this booklet is for people being exposed to evaluation for the first time and to surveys, as well. I recommend it.

The use of a survey is a valuable evaluation tool, especially in the world of electronic media. The survey allows individuals to gather data (both qualitative and quantitative) easily and relatively inexpensively. When I want information about surveys, I turn to the 4th edition of the Dillman book (Dillman, Smyth, & Christian, 2014*). Dillman has advocated the “Tailored Design Method” for a long time. (I first became aware of his method, which he called “Total Design Method,” in his 1978 first edition, a thin, 320 page volume [as opposed to the 509 page fourth edition].)

Today I want to talk about the “Tailored Design” method (originally known as total design method).

In the 4th edition, Dillman et al. say that “…in order to minimize total survey error, surveyors have to customize or tailor their survey designs to their particular situations.” They are quick to point out (through various examples) that the same procedures won’t work for all surveys. The “Tailored Design Method” refers to the customizing survey procedures for each separate survey. It is based upon the topic of the survey and the audience being surveyed as well as the resources available and the time-line in use. In his first edition, Dillman indicated that the TDM (Tailored Design Method) would produce a response rate of 75% for mail surveys and an 80%-90% response rate is possible for telephone surveys. Although I cannot easily find the same numbers in the 4th edition, I can provide an example (from the 4th edition on page 21-22) where the response rate is 77% after a combined contact of mail and email over one month time. They used five contacts of both hard and electronic copy.

This is impressive. (Most surveys I and others I work with conduct have a response rate less than 50%.) Dillman et al. indicate that there are three fundamental considerations in using the TDM. They are:

Reducing four sources of survey error–coverage, sampling, nonresponse, and measurement;
Developing a set of survey procedures that interact and work together to encourage all sample members to respond; and
Taking into consideration elements such as survey sponsorship, nature of survey population, and the content of the survey questions.

The use of a social exchange perspective suggests that respondent behavior is motivated by the return that behavior is expected, and usually does, bring. This perspective affects the decisions made regarding coverage and sampling, the way questions are written and questionnaires are constructed, and determines how contacts will produce the intended sample.

If you don’t have a copy of this book (yes, there are other survey books out there) on your desk, get one! It is well worth the cost ($95.00, Wiley; $79.42, Amazon).

* Dillman, D. A., Smyth, J. D. & Christian, L. M. (2014) Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, N. J.: John Wiley & Sons, Inc.

my .

molly.

I don’t know what to write today for this week’s post. I turn to my book shelf and randomly choose a book. Alas, I get distracted and don’t remember what I’m about. Mama said there would be days like this…I’ve got writer’s block (fortunately, it is not contagious). (Thank you, Calvin). There is also an interesting (to me at least because I learned a new word–thrisis: a crisis of the thirties) blog on this very topic (here).

So this is what I decided rather than trying to refocus. In the past 48 hours I’ve had the following discussions that relate to evaluation and evaluative thinking.

In a faculty meeting yesterday, there was the discussion of student needs which occur during the students’ matriculation in a program of study. Perhaps it should include assets in addition to needs as students often don’t know what they don’t know and cannot identify needs.
A faculty member wanted to validate and establish the reliability for a survey being constructed. Do I review the survey, provide the reference for survey development, OR give a reference for validity and reliability (a measurement text)? Or all of the above.
There appears to be two virtual focus group transcripts for a qualitative evaluation that have gone missing. How much affect will those missing focus groups have on the evaluation? Will notes taken during the sessions be sufficient?
A candidate came to campus for an assistant professor position who presented a research presentation on the right hand (as opposed to the left hand) [Euphemisms for the talk content to protect confidentiality.] Why even study the right hand when the left hand is what is the assessment?
Reading over a professional development proposal dealing with what is, what could be, and what should be. Are the questions being asked really addressing the question of gaps?

Continue reading →

I am an evaluator, charter member of the American Evaluation Association, and former member of the forerunner organization, Evaluation Network. When you push my “on ” button, I can talk evaluation until (a lot of metaphors could be used here); and often do. (I can also talk about other things with equal passion, though not professionally.) When my evaluation button is pushed or, for that matter, most of the time, I wonder what difference am I making. In this case, I wonder what difference I am making with this blog.

One of my readers (I have more than I ever imagined) suggested that I develop an “online” survey that I can include regularly in my posts. I thought that was a good idea. I thought I’d go one better and have it be a part of the blog. Then I would tabulate the findings (if there are any 🙂 ). Just so you know, I DO read all the comments; I get at least six daily. I often do not comment on those, however.

So, reader, here is the making a difference survey . This link will (should) take you to Surveymonkey and the survey. Below, I’ve listed the questions that are in the survey.

Check all that apply.

Reading this blog makes a difference to me by:

_____ Giving me a voice to follow
_____ Providing interesting content
_____ Providing content I can use in my work
_____ Providing dependable post
_____ Providing me with information to share
_____ Building my skills in evaluation
_____ Showing me that there are others in the world concerned with similar things
_____ Offering me good reading about an interesting weekly topic
_____ Offering me content of value to me
_____ Other. Please specify in comment

Please complete the survey.

my .

molly.

I’ve talked about bias bias 2 before (cognitive bias; personal and situational bias); I’ve probably talked about bias in surveys and sampling. Today I want to talk specifically about self report bias…you know, the bias that exists when people answer questions themselves (as opposed to having their behavior be observed).

First, what is self-report bias (often called self-response bias)? It is the bias that exists when people answer survey questions by themselves. Everyone has this bias; it is unavoidable. It can be seen as social desirability bias (what the the respondent thinks the survey writer wants to hear); self-selection bias (a person decides to respond when invited as opposed to not responding); and what I’m going to call a “clarity bias” (whether the respondent understands the survey content).

I’m finding more and more that the S of good writing are applicable to all writing–fiction, non-fiction, scholarly, SURVEY. If the survey isn’t clear, the respondent isn’t going to be able to answer in a way that is meaningful. If the respondent cannot answer the survey in a way that is meaningful, there will be no meaningful data. If data are not meaningful, then the evaluation will not be able to tell you the value or merit or worth of the project being evaluated.

It is important to

Pilot test the survey before sending it out to the target audience.
Have naive readers read over the survey (different from pilot testing).
Only ask one thing at a time in the questions.

I’m sure there are other things that would help minimize bias–let me know other options used.

Bottom line: Self report bias is always part of evaluation that involves people; it can be minimized.

New topic.

This is the time of year that one thinks about changes and how one will do that in the new year. Yet, those changes often fall by the way side, getting left in the dust (so to speak) of every day life. One way I’ve kept those changes fresh is to follow how the new year presents itself. There is the calendar new year new_year_2015 (on January 1); there is the lunar new year 2015-Chinese-New-Year-Free-Design (this year on February 19, the year of the goat); there is the spring equinox (norooz, the Persian new year); Rosh Hashana (Jewish new year beginning on the evening of September 13); there is the Islamic new year, the Thai new year, the Ethiopian new year, and the list goes on. (What is your favorite new year and new year’s celebration?) By refreshing the year regularly, I can keep my “resolutions” alive all year. My wish for you is a prosperous and healthy new year. Welcome 2015.

my.

molly.

Within the last 24 hours I’ve had two experiences that remind me of how tenuous our connection is to others.

Yesterday, I was at the library to return several books and pick up a hold. As I went to check out using the digitally connected self-check out station, I got an “out of service” message. Not thinking much of it, as I had received that message before, I moved to another machine. And got the same message! So I went to the main desk. There was a person in front of me; she was taking a lot of time. Turns out it wasn’t her; it was the internet (or intranet, don’t know which). There was no connection! After several minutes, a paper system was implemented and I was assured that the book would be listed by this evening. That the library had a back up system impressed me; I’ve often wondered what would happen if the electricity went out for a long periods of time since the card catalogs are no longer available.
Also, yesterday, I received a phone call on my office land line (!), which is a rare occurrence these days. On the other end was a long time friend and colleague. We are working feverishly on finishing a NDE volume. We have an August 22 deadline and I will be out of town taking my youngest daughter to college. Family trumps everything. He was calling because the gardeners at his condo had cut the cable to his internet, television, and most importantly, his wi-fi. He couldn’t Skype me (our usual form of communication)! He didn’t expect resumption of service until the next day (August 20 at 9:47am PT he went back on line–he lives in the Eastern Time Zone). Continue reading →

Many of you have numerous lists for summer reading (NY Times, NPR, Goodreads, Amazon, others…). My question is what are you reading to further your knowledge about evaluation? Perhaps you are; perhaps you’re not. So I’m going to give you one more list 🙂 …yes, it is evaluative.

If you want something light: Regression to the Mean by Ernest R. House. It is a novel. It is about evaluation. It explains what evaluators do from a political perspective.

If you want something qualitative: Qualitative Data Analysis by Matthew B. Miles, A. Michael Huberman, and Johnny Saldana. It is the new 3rd edition which Sage (the publisher) commissioned. A good thing, too, as both Miles and Huberman are no longer able to do a revision. My new go-to book.

If you want something on needs assessment: Bridging the Gap Between Asset/Capacity Building and Needs Assessment by James W. Altschuld. Bridging the Gap-altschuld Most needs assessments start with what is lacking (i.e., needed); this proposes that an assessment start with what is present (assets) and build from there, and in the process, meeting needs.

If you want something on higher education: College (Un)bound by Jeff Selingo. college unbound by jeffry selingo The state of higher education and some viable alternatives by a contributing editor at the Chronicle of Higher Education. Yes, it is evaluative.

Most of these I’ve mentioned before. I’ve read the above. I recommend them.

Continue reading →

A reader asked how to choose a sample for a survey. Good question.

My daughters are both taking statistics (one in college, one in high school) and this question has been mentioned more than once. So I’ll give you my take on sampling. There are a lot of resources out there (you know, references and other sources). My favorite is in Dillman 3rd edition, page 57.

Sampling is easier than most folks make it out to be. Most of the time you are dealing with an entire population. What, you ask, how can that be?

You are dealing with an entire population when you survey the audience of a workshop (population 20, or 30, or 50). You are dealing with a population when you deal with a series of workshops (anything under 100). Typically, workshops are a small number; only happen once or twice; rarely include participants who are there because they have to be there. If you have under 100, you have an entire population. They can all be surveyed.

Now if your workshop is a repeating event with different folks over the offerings, then you will have the opportunity to sample your population because it is over 100 (see Dillman, 3rd edition, page 57). If you have over 100 people to survey AND you have contact information for them, then you want to randomly sample from that population. Random selection (another name for random sampling) is very different from random assignment; I’m talking about random sampling.

Random sampling is a process where everyone gets an identification number (and an equal chance to be selected), sequentially; so 1- 100. Then find a random number table; usually found in statistic books in the back. Close your eyes and let your hand drop onto a number. Let’s say that number is 56997. You know you need numbers between 1 and 100 and you will need (according to Dillman) for a 95% confidence level with a plus or minus 3% margin of error and a 50/50 split at least 92 cases (participants) OR if you want an 80/20 split, you will need 87 cases (participants). So you look at the number and decide which two digit number you will select (56, 69, 99, 0r 97). That is your first number. Let us say you chose 99 that is the third two digit number found in the above random number (56 and 69 being the first two). So participant 99 will be on the randomly selected (random sampling) list. Now you can go down the list, up the list, to the left or the right of the list and identify the next two digit number in the same position. For this example, using the random numbers table from my old Minium (for which I couldn’t find a picture since it is OLD) stat book (the table was copied from the Rand Corporation, A million random digits with 100,000 normal deviates, Glencoe, IL: The Free Press, 1955), the number going right is 41534, I would choose participant number 53. Continuing right, with the number 01953, I would choose participant number 95, etc. If you come across a number that you have already chosen, go to the next number. Do this process until you get the required number of cases (either 92 or 87). You can select fewer if you want a 10% plus or minus margin of error (49, 38) or a 5% plus or minus margin of error (80, 71). (I always go for the least margin of error, though.) Once you have identified the required number, drafted the survey, and secured IRB approval, you can send out the survey. We will talk about response rates next week.

I had a topic all ready to write about then I got sick. I’m sitting here typing this trying to remember what that topic was, to no avail. That topic went the way of much of my recent memory; another day, perhaps.

I do remember the conversation with my daughter about correlation. She had a correlation of .3 something with a probability of 0.011 and didn’t understand what that meant. We had a long discussion of causation and attribution and correlation.

We had another long conversation about practical v. statistical significance, something her statistics professor isn’t teaching. She isn’t learning about data management in her statistics class either. Having dealt with both qualitative and quantitative data for a long time, I have come to realize that data management needs to be understood long before you memorize the formulas for the various statistical tests you wish to perform. What if the flood happens????

So today I’m telling you about data management as I understand it, because the did actually happen and, fortunately, I didn’t loose my data. I had a data dictionary.

Data dictionary. The first step in data management is a data dictionary. There are other names for this, which escape me right now…know that a hard copy of how and what you have coded is critical. Yes, make a back up copy on your hard drive…have a hard copy because the flood might happen. (It is raining right now and it is Oregon in November.)

Take a hard copy of your survey, evaluation form, qualitative data coding sheet and mark on it what every code notation you used means. I’d show you an example of what I do, only they are at the office and I am home sick without my files. So, I’ll show you a clip art instead… data management . No, I don’t use cards any more for my data (I did once…most of you won’t remember that time…), I do make a hard copy with clear notations. I find my self doing that with other things to make sure I code the response the same way. That is what a data dictionary allows you to do–check yourself.

Then I run a frequencies and percentages analysis. I use SPSS (because that is what I learned first). I look for outliers, variables that are miscoded, and system generated missing data that isn’t missing. I look for any anomaly in the data, any humon error (i. e. my error). Then I fix it. Then I run my analyses.

There are probably more steps than I’ve covered today. These are the first steps that absolutely must be done BEFORE you do any analyses. Then you have a good chance of keeping your data safe.

Miscellaneous thought 1.

Yesterday, I had a conversation with a long time friend of mine. When we stopped and calculated (which we don’t do very often), we realized that we have know each other since 1981. We met at the first AEA (only it wasn’t AEA then) conference in Austin, TX. I was a graduate student; my friend was a practicing professional/academic. Although we were initially talking about other things evaluation; I asked my friend to look at an evaluation form I was developing. I truly believe that having other eyes (a pilot if you will) view the document helps. It certainly did in this case. I feel really good about the form. In the course of the conversation, my friend advocated strongly for a odd numbered scales. My friend had good reasons, specifically

1) It tends to force more comparisons on the respondents; and

2) if you haven’t given me a neutral point I tend to mess up the scale on purpose because you are limiting my ability to tell you what I am thinking.

I, of course, had an opposing view (rule number 8–question authority). I said, ” My personal preference is an even number scale to avoid a mid-point. This is important because I want to know if the framework (of the program in question) I provided worked well with the group and a mid-point would provide the respondent with a neutral point of view, not a working or not working opinion. An even number (in my case four points) can be divided into working and not working halves. When I’m offered a middle point, I tend to circle that because folks really don’t want to know what I’m thinking. By giving me an opt out/neutral/neither for or against option they are not asking my opinion or view point.”

Recently, I came across an aea365 post on just this topic. Although this specific post was talking about Likert scales, it applies to all scaling that uses a range of numbers (as my friend pointed out). The authors sum up their views with this comment, “There isn’t a simple rule regarding when to use odd or even, ultimately that decision should be informed by (a) your survey topic, (b) what you know about your respondents, (c) how you plan to administer the survey, and (d) your purpose. Take time to consider these four elements coupled with the advantages and disadvantages of odd/even, and you will likely reach a decision that works best for you.” (Certainly knowing my friend like I do, I would be suspicious of responses that my friend submitted.) Although they list advantages and disadvantages for odd and even responses, I think there are other advantages and disadvantages that they did not mentioned yet are summed up in their concluding sentence.

Miscellaneous thought 2.

I’m reading the new edition of Qualitative Data Analysis (QDA). This has always been my go to book for QDA and I was very sad when I learned that both of the original authors had died. The new author, Johnny Saldana (who is also the author of The Coding Manual for Qualitative Researchers), talks (in the third person plural, active voice) about being a pragmatic realist. That is an interesting concept. They (because the new author includes the previous authors in his statement) say “that social phenomena exist not only in the mind but also in the world–and that some reasonably stable relationships can be found among the idiosyncratic messiness of life.” Although I had never used those exact words before, I agree. It is nice to know the label that applies to my world view. Life is full of idiosyncratic messiness; probably why I think systems thinking is so important. I’m reading this volume because I’ve been asked to write the review of one of my favorite books. We will see if I can get through it between now and July 1 when the draft of the review is due. Probably aught to pair it with Saldana’s other book; won’t happen between now and July 1.

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: survey development

Methodology

Surveys. Again. Still.

Blogging and Writer’s Block

Making a difference

Bias (again)

Evaluation and the electronic age

Summer Reading

Sampling

Quandry

Miscellaneous Thoughts

Contact Info