Chris Lysy 
draws cartoons.
Evaluation and research cartoons.
Logic Model cartoons. 
Presentation cartoons.
Data cartoons. 
He has offered an alternative to presenting survey data. He has a wonderful cartoon for this.
He is a wonderful resource. Use him. You can contact him through his blog, fresh spectrum.
my 
.
molly.
AEA365 is honoring living evaluators for Labor Day (Monday, September 5, 2016).
Some of the living evaluators I know (Jim Altschuld, Tom Chapel, Michael Patton, Karen Kirkhart, Mel Mark, Lois-Ellin Datta, Bob Stake); Some of them I don’t know (Norma Martinez-Rubin, Nora F. Murphy, Ruth P. Saunders, Art Hernandez, Debra Joy Perez). One I’m not sure of at all (Mariana Enriquez). Over the next two weeks, AEA365 is hosting a recognition of living evaluator luminaries.
The wonderful thing is that this give me an opportunity to check out those I don’t know; to read about how others see them, what makes them special. I know that the relationships that develop over the years are dear, very dear.
I also know that the contributions that these folks have made to evaluation cannot be captured in 450 words (although we try). They are living giants, legends if you will.
These living evaluators have helped move the field to where it is today. Documenting their contributions to evaluation enriches the field. We remember them fondly.
If you don’t know them, look for them at AEA ’16 in Atlanta 
. Check out their professional development sessions or their other contributions (paper, poster, round-table, books, etc). Many of them have been significant contributors to AEA; some have only been with AEA since the early part of this century. All have made a meaningful contribution to AEA.
Many evaluators could be mentioned and are not. Sheila B. Robinson suggests that “…we recognize that many, many evaluators could and should be honored as well as the 13 we feature this time, and we hope to offer another invitation next year for those who would like to contribute a post, so look for that around this time next year, and sign up!
James W. Altschuld Thomas J. Chapel
Norma Martinez-Rubin Michael Quinn Patton
Nora F. Murphy Ruth P. Saunders
Art Hernandez Karen Kirkhart
Mel Mark Lois-Ellin Datta
Debra Joy Perez Bob Stake
Mariana Enriquez (Photo not known/found)
my .
molly.
The use of a survey is a valuable evaluation tool, especially in the world of electronic media. The survey allows individuals to gather data (both qualitative and quantitative) easily and relatively inexpensively. When I want information about surveys, I turn to the 4th edition of the Dillman book 
(Dillman, Smyth, & Christian, 2014*). Dillman has advocated the “Tailored Design Method” for a long time. (I first became aware of his method, which he called “Total Design Method,” in his 1978 first edition,
a thin, 320 page volume [as opposed to the 509 page fourth edition].)
Today I want to talk about the “Tailored Design” method (originally known as total design method).
In the 4th edition, Dillman et al. say that “…in order to minimize total survey error, surveyors have to customize or tailor their survey designs to their particular situations.” They are quick to point out (through various examples) that the same procedures won’t work for all surveys. The “Tailored Design Method” refers to the customizing survey procedures for each separate survey. It is based upon the topic of the survey and the audience being surveyed as well as the resources available and the time-line in use. In his first edition, Dillman indicated that the TDM (Tailored Design Method) would produce a response rate of 75% for mail surveys and an 80%-90% response rate is possible for telephone surveys. Although I cannot easily find the same numbers in the 4th edition, I can provide an example (from the 4th edition on page 21-22) where the response rate is 77% after a combined contact of mail and email over one month time. They used five contacts of both hard and electronic copy.
This is impressive. (Most surveys I and others I work with conduct have a response rate less than 50%.) Dillman et al. indicate that there are three fundamental considerations in using the TDM. They are:
The use of a social exchange perspective suggests that respondent behavior is motivated by the return that behavior is expected, and usually does, bring. This perspective affects the decisions made regarding coverage and sampling, the way questions are written and questionnaires are constructed, and determines how contacts will produce the intended sample.
If you don’t have a copy of this book (yes, there are other survey books out there) on your desk, get one! It is well worth the cost ($95.00, Wiley; $79.42, Amazon).
* Dillman, D. A., Smyth, J. D. & Christian, L. M. (2014) Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, N. J.: John Wiley & Sons, Inc.
my 
.
molly.
This is a link to an editorial in Basic and Applied Social Psychology
. It says that inferential statistics are no longer allowed by authors in the journal.
“What?”, you ask. Does that have anything to do with evaluation? Yes and no. Most of my readers will not publish here. They will publish in evaluation journals (of which there are many) or if they are Extension professionals, they will publish in the Journal of Extension.
And as far as I know, BASP is the only journal which has established an outright ban on inferential statistics. So evaluation journals and JoE still accept inferential statistics.
Still–if one journal can ban the use, can others?
What exactly does that mean–no inferential statistics? The journal editors define this ban as as “…the null hypothesis significance testing procedure is invalid and thus authors would be not required to perform it.” That means that authors will remove all references to p-values, t-values, F-values, or any reference to statements about significant difference (or lack thereof) prior to publication. The editors go on to discuss the use of confidence intervals (No) and Bayesian methods (case-by case) and what inferential statistical procedures are required by the journal. Continue reading
I am an evaluator, charter member of the American Evaluation Association, and former member of the forerunner organization, Evaluation Network. When you push my “on ” button, I can talk evaluation until (a lot of metaphors could be used here); and often do. (I can also talk about other things with equal passion, though not professionally.) When my evaluation button is pushed or, for that matter, most of the time, I wonder what difference
am I making. In this case, I wonder what difference I am making with this blog.
One of my readers (I have more than I ever imagined) suggested that I develop an “online” survey that I can include regularly in my posts. I thought that was a good idea. I thought I’d go one better and have it be a part of the blog. Then I would tabulate the findings (if there are any 🙂 ). Just so you know, I DO read all the comments; I get at least six daily. I often do not comment on those, however.
So, reader, here is the making a difference survey . This link will (should) take you to Surveymonkey and the survey. Below, I’ve listed the questions that are in the survey.
Check all that apply.
Reading this blog makes a difference to me by:
Please complete the survey.
my .
molly.
Recently, I got a copy of Marvin Alkin’s book, Evaluation Roots (his first edition; eventually, I will get the second edition).
In Chapter Two, he and Tina Christie talk about an evaluation theory tree and presents this idea graphically (all be it in draft form).
Think of your typical tree with three strong branches (no leaves) and two roots. Using this metaphor, the authors explain the development of evaluation theory as it appears in western (read global north) societies.
As you can see, the roots are “accountability and control” (positivist paradigm?) and social inquiry (post-positivist paradigm?). Continue reading
Personal and situational bias are forms of cognitive bias and we all have cognitive bias.
When I did my dissertation on personal and situational biases, I was talking about cognitive bias (only I didn’t know it, then).
According to Wikipedia, the term cognitive bias was introduced in 1972 (I defended my dissertation in 1983) by two psychologists Daniel Kahneman and Amos Tversky 
.
Then, I hypothesized that previous research experience (naive or sophisticated) and the effects of exposure to expected project outcomes (positive, mixed, negative) would affect the participant and make a difference in how the participant would code data. (It did.) The Sadler article which talked about intuitive data processing was the basis for this inquiry. Now many years later, I am encountering cognitive bias again. Sadler says that “…some biases can be traced to a particular background knowledge…”(or possibly–I think–lack of knowledge), “…prior experience, emotional makeup or world view”. 
(This, I think, falls under the category of, according to Tversky and Kahneman, human judgements and it will differ from rational choice theory (often given that label). Continue reading
Recently, I drafted a paper about a capacity building; I’ll be presenting it at the 2014 AEA conference. The example on which I was reporting was regional and voluntary; it took a dedication, a commitment from participants. During the drafting of that paper, I had think about the parts of the program; what would be necessary for individuals who were interested in evaluation and had did not have a degree. I went back to the competencies listed in the AJE article (March 2005) that I cited in a previous post. I found it interesting to see that the choices I made (after consulting with evaluation colleagues) were listed in the competencies identified by Stevahn et al., yet they list so much more. So the question occurs to me is: To be competent, to build institutional evaluation capacity are all those needed? Continue reading
Within the last 24 hours I’ve had two experiences that remind me of how tenuous our connection is to others.
Had a comment a while back on analyzing survey data…hmm…that is a quandary as most surveys are done on line (see Survey monkey, among others).
If you want to reach a large audience (because your population from which you sampled is large), you will probably use an on-line survey. The on-line survey companies will tabulate the data for you. Can’t guarantee that the tabulations you get will be what you want, or will tell you want you want to know. Typically (in my experience), you can get an Excel file which can be imported into a soft ware program and you can run your own analyses, separate from the on line analyses. Continue reading