Hi everybody–it is time for another TIMELY TOPIC.  This week’s topic is about using pretest/posttest evaluation or a post-then-pre evaluation.

There are many considerations for using these designs.  You have to look at the end result and decide what is most appropriate for your program.  Some of the key considerations include:

• the length of your program;
• the information you want to measure;
• the factors influencing participants response; and
• available resources.

Before explaining the above four factors, let me urge you to read on this topic.  There are a couple of resources (yes, print…) I want to pass your way.

1. Campbell, D. T. & Stanley, J. C. (1963).  Experimental and quasi-experimental designs for research.  Houghton Mifflin Company:  Boston, MA.  (The classic book on research and evaluation designs.)
2. Rockwell, S. K., & Kohn, H. (1989). Post-then-pre evaluation. Journal of Extension [On-line]. 27(2). Available at: http://www.joe.org/joe/1989summer/a5.htm (A seminal JoE paper explaining post-then-pre test.)
3. Nimon, K. Zigarami, D. & Allen, J. (2011).  Measures of program effectiveness based on retrospective pretest data:  Are all created equal? American Journal of Evaluation, 32, 8 – 28.  (A 2011 publication with an extensive bibliography.)

Let’s talk about considerations.

Length of program.

For pre/post test, you want a program that is long.  More than a day.  Otherwise you risk introducing a desired response bias and the threats to internal validity that  Campbell and Stanley identify.  Specifically the threats called history, maturation, testing, and instrumentation,  also a possible regression to the mean threat, though that is on a possible source of concern.  These threats to internal validity assume no randomization and a one group design, typical for Extension programs and other educational programs.  Post-then-pre works well for short programs, a day or less, and  tend to control for response shift and desired response bias.  There may still be threats to internal validity.

Information you want to measure.

If you want to know a participants specific knowledge, a post-then-pre cannot provide you with that information because you can not test something you cannot unknow.  The traditional pre/post can focus on specific knowledge, e.g., what food is the highest in Vitamin C in a list that includes apricot, tomato, strawberry cantaloupe. (Answer:  strawberry)  If you are wanting agreement/disagreement with general knowledge (e.g., I know what the key components of strategic planning are), the post-pre works well.  Confidence, behaviors, skills, and attitudes can all be easily measured with a post-then-pre.

Factors influencing participants response.

I mentioned threats to internal validity above.  These factors all influence participants responses.  If there is a long time between the pretest and the post test, participants can be affected by history (a tornado prevents attendance to the program); maturation (especially true with programs with children–they grow up); testing (having taken the pretest, the post test scores will be better);  and instrumentation (the person administering the posttest administers it differently than the pretest was administered).  Participants desire to please the program leader/evaluator, called desired response bias, also affects participants response.

Available resources.

Extension programs (as well as many other educational programs) are affected by the availability of resources (time, money, personnel, venue, etc.).  If you only have a certain amount of time, or a certain number of people who can administer the evaluation, or a set amount of money, you will need to consider which approach to evaluation you will use.

The idea is to get usable, meaningful data that accurately reflects the work that went into the program.