A colleague asks for advice on handling evaluation stories, so that they don’t get brushed aside as mere anecdotes. She goes on to say of the AEA365 blog she read, ” I read the steps to take (hot tips), but don’t know enough about evaluation, perhaps, to understand how to apply them.” Her question raises an interesting topic. Much of what Extension does can be captured in stories (i.e., qualitative data) rather than in numbers (i.e., quantitative data). Dick Krueger, former Professor and Evaluation Leader (read specialist) at the University of Minnesota has done a lot of work in the area of using stories as evaluation. Today’s post summarizes his work.
There are all types of stories. The type we are interested in for evaluation purposes are organizational stories. Organizational stories can do the following things for an organization:
He suggests six common types of organizational stories:
To use stories as evaluation, the evaluator needs to consider how stories might be used, that is, do they depict how people experience the program? Do they understand program outcomes? Do they get insights into program processes?
You (as evaluator) need to think about how the story fits into the evaluation design (think logic model; program planning). Ask yourself these questions: Should you use stories alone? Should you use stories that lead into other forma of inquiry? Should you use stories that augment/illustrate results from other forms of inquiry?
You need to establish criteria for stories. Rigor can be applied to story even though the data are narrative. Criteria include the following: Is the story authentic–is it truthful? Is the story verifiable–is there a trail of evidence back to the source of the story? Is there a need to consider confidentiality? What was the original intent–purpose behind the original telling? And finally, what does the story represent–other people or locations?
You will need a plan for capturing the stories. Ask yourself these questions: Do you need help capturing the stories? What strategy will you use for collecting the stories? How will you ensure documentation and record keeping? (Sequence the questions; write them down the type–set up; conversational; etc.) You will also need a plan for analyzing and reporting the stories as you, the evaluator, are responsible for finding meaning.