Last week, the National Outreach Scholarship Conference was held at Michigan State University campus.  There was an impressive array of speakers and presentations.  I had the luxury of attending Michael Quinn Patton’s session on Utilization-focused Evaluation. And although the new edition of the book is 600+ pages, Michael distilled the essentials down.  He also announced a new book (only 400+ pages) called The Essentials of Utilization Focused Evaluation. .  This volume is geared to practitioners as opposed to the classroom or the academic.

One take away message for me was this:  “Context changes the focus of ‘use’ “.  So if you have a context whereby the reports are only for accounting purposes, the report will look very different from a context whereby the reports are for detailing the difference being made.  Now, this sounds very intuitive.  Like, DUH, Molly, tell me something I don’t know.  Yet this is so important because you, as the evaluator, have the responsibility and the obligation to prepare stakeholders to use data in OTHER ways than as a reporting activity. That responsibility and obligation is tied to the Program Evaluation Standards.  The Joint Committee revised the standards after soliciting feedback from multiple sources.  This 3rd Ed. addresses with numerous examples and discussion the now five standards.  These standards are:

1. Utility
2. Feasibility
3. Propriety
4. Accuracy
5. Accountability.

Apparently, there was considerable discussion as the volume was being compiled that Accountability needed to be first.  Think about it, folks.  If Accountability was first, then evaluations would build on “the responsible use of resources to produce value.”  Implementation, improvement, worth, and costs would drive evaluation.  By placing utilization first, evaluators have the responsibility and obligation to base judgements “…on the extent to which program stakeholders find evaluation processes and products valuable in meeting their needs…to examine the variety of possible uses for evaluation processes, findings, and products.”

Certainly validates use as defined in Utilization-Focused Evaluation.  Take Michael’s workshop.  The American Evaluation Association is offering this workshop at its annual meeting, in Anaheim, CA and the workshop is on Wednesday, November 2.  Go to eval.org and click on Evaluation Conference.  If you can’t join the workshop–Read the book!  (either one).  It is well worth it.

For the last three weeks, since I posted the history matching game, I’ve not been able to post with images.  Every time I go to save the draft, the post vanishes.  I’m working with the IT folks.  They haven’t given me any alternatives.  I posting this today without images to let you know that I am still here, that I still have thoughts, and that I will post something of substance again soon.  Please be patient.  Thank you.

Independence is an evaluative question.

Think about it while you enjoy the holiday.

Were the folks who fought the Revolutionary War, truly revolutionaries? OR were they terrorists?

Was King George a despot or just a micromanager?

My favorite is this:  Was the War Between the States, the last battle of the War of/for Independence?

I’m sure there are other  evaluative questions.  Got a question that is evaluative?  Let me know.

My creative effort this past year (other than my blog) has been to create new and (hopefully) wonderful pie.  This pie is vegetarian, not vegan, and obviously not dairy free…contains milk products and coconut.

Today, a bonus post–my gift to you:

WHITE CHRISTMAS PIE

You will need a 9 inch pie crust, fully baked.  (Although I make mine, getting one premade and following the directions for prebaking the crust will also work.)

Enough crushed peppermint candy to cover the pie crust that has cooled to room temperature.  Save about 1 tsp for garnish.

Melt over a double boiler, 12 ounces of white chocolate chips. A double boiler helps keep the chocolate warm and pourable.

Whip to soft peaks, 1 1/2 Cups of whipping cream. Stir in 1/8 tsp of mint or peppermint extract.

Continue whipping until firm peaks form.  SLOWLY fold into the cream, the cool white chocolate.  There will be layers of cooled chocolate throughout the cream.  That is the way it is supposed to be.

Spoon the chocolate mixture into the prepared pie crust.

Freeze for at least one hour or over night.

Prior to serving, remove pie from the freezer.

Whip 1 Cup of whipping cream until soft peaks form.

Add 1/4 tsp of mint or peppermint extract.

Sift into the cream, 2 Tbs of powdered sugar.

Continue whipping until stiff peaks form.

Spoon over the frozen pie, peaking the whipped cream to look like snow drifts.

Sprinkle with 1 Tbs unsweetened coconut and 1 tsp. crushed peppermint candy (that which you had left over above).

Cut into small slices.  Serves 12.  Happy Holidays!

After experiencing summer in St. Petersburg,  FL, then peak color in Bar Harbor and Arcadia National Park, ME,  I am once again reminded of how awesome these United States truly are.  Oregon holds its own special brand of beauty and it is nice to be back home.  Evaluation was everywhere on this trip.

A recent  AEA365 post talks about systems thinking and evaluating educational programs.  Bells went off with me because Extension DOES educational programs and does them in existing systems.  Often, Extension professionals neglect the systems aspect of their programming and attempt to implement the program in isolation.  In today’s complex world, isolation isn’t possible.  David Bella, an Emeritus professor at OSU uses the term “complex messy systems”.  I think that clearly characterizes what Extension faces in developing programs. The AEA365 post has some valuable points for Extension professionals to remember (see the link for more details):

1.  Build relationships with experts from across disciplines.

2.  Ensure participation from stakeholders across the entire evaluated entity.

3.  Create rules of order to guide the actions of the evaluation team.

These are points for Extension professionals to keep in mind as they develop their programs.  By keeping them in mind and using them,  Extension professionals can strengthen their programs.  More and more, extension programs are multi-site as well as multi-discipline.   Ask yourself:   What part of the program is missing because of failure to consult across disciplines? or What part of the program won’t be recognized because of failure to include as many stakeholders as possible in helping to design the evaluation?  Who will know better what makes an effective program than those individuals in the target audience?  Helping everyone know what the expectations are helps systems work, change, and grow.

It is also important consider the many contextual factors.  When working in community-based programs, Extension professionals need to develop partnerships and those partnerships need to work in agreement.  This is another example Extension work and evaluation of that work occurs withing an existing system.

Last week, I talked briefly about what test to use to analyze your data. Most of the evaluation work conducted by Extension professionals results in one group, often an intact group or population.  Today I want to talk about what you can do with those data.

One of the first things you can do is to run frequencies and percentages on these data.  In fact, I recommend you compute them as the first analyzes you run.  Most softwear (SPSS, SAS, Excel, etc.) programs will do this for you.  When you run frequencies in SPSS, the computer returns an output that looks something like the first image:

When compute frequencies in SAS, the resulting output looks like the second image:

Both images report frequencies, percentages of those frequencies, and cumulative percentage (that is, it adds the percents of frequency A to the percent of frequency B, etc. until 100% is reached).

To compute frequencies in Excel, read here.  Excel has a number of COUNT functions depending on what you want to know.

Once you have computed frequencies and percentages, most people want to know if change occurred.  Although there are other analyses  which can be performed (reliability, validity, correlation, prediction), all of these require that you know what type of data do you have

• nominal–whether people’s answers named something (e.g., gender; marital status);
• ordinal–whether people ordered their responses on how strongly they agreed (e.g., agree or disagree);
• interval–the scores on a standardized scale (e.g., temperature or nutrition test).

If you have nominal data and you want to compute change, you need to know how many times participants are answering the questionnaire and how many categories you have in your questions (e.g.pre/post; yes/no).  If you are giving the questionnaire twice and there are two categories for some of your questions, you can compute the McNemar change test.  The McNemar change test is a non-parametric (meaning that the parameters are not known) that is applied to a 2×2 contingency table.   It tests for changes in responses using the chi-square distribution and is useful for detecting changes in responses due to “before-and-after” designs.  A 2×2 contingency table has two columns and two rows.  The frequencies from the nominal data are in the cells where the rows and columns cross; the totals for rows and columns are in the margins (or the last row and the far right column).  SPSS computes the following statistics when a cross tabs test is run–Pearson’s Chi Square, Continuity Correction, Likelihood Ratio, Fisher’s Exact Test, and Linear by Linear Association. A McNemar test can be specified.

This will be very brief.

The answer to the question,  “What test do I use?”  is, “It all depends.”

If you have one group you can do the following:

• check the reliability.
• check the validity.
• look at relationships between variables.
• predict something from other variables.
• look at change across time.
• look at scores on one variable measured under different conditions (within group difference)

If you have two groups you can do the following:

• compare the two groups on one variable (between group difference).
• look at change across time between the two groups
• compare two groups on one variable under different conditions (within group difference).

If you have more than two groups, it gets more complicated and I’ll talk about that another day.  Most Extension work doesn’t have more than two groups.

So you can see, it all depends.  More later.

Last Friday, I had the opportunity to talk with a group of graduate students about evaluation as I have seen it (for the past now almost 30 years) and currently see it.

The previous day, I  finished an in depth, three-day professional development session on differences.  Now, I would guess you are wondering what do these two activities have in common and how they relate to evaluation.  All three are tied together through an individual’s perspective.  I was looking for a teachable moment and I found one.

A response often given by evaluators when asked a question about the merit and worth of something (program, process, product, policy, personnel, etc.) is, “It all depends.”

And you wonder, “Depends on what?”

The answer is:  PERSPECTIVE.

Your experiences place you in a unique and original place.  Your view point is influenced by those experiences; as are your attitudes, your behaviors, your biases; your understanding of differences; your approach to problem solving; your view of inquiry.  All this is perspective.  And when you make decisions about something, those experiences (i.e., your perspective) affect your decisions.  Various dimensions of experience and birth (the diversity wheel to the left lists the dimensions of difference) affect what choices you make; affect how you approach a problem; affect what questions you ask; affect your interpretation of a situation.

The graduate students came from different employment backgrounds; were of different ages, genders, marital status, ethnicity, appearance, educational background, health status, income, geographic location and probably other differences I couldn’t tell from looking or listening.  Their view of evaluation was different.  They asked different questions.  The answer to which was “It all depends.”  And even that (It all depends)  is an evaluative activity–not unlike talking to graduate students, understanding perspective, or doing evaluation.