Tag Archives: formative

Process, progress, product?

Posted on by

Last week, I talked about formative and summative evaluations.  Formative and summative  evaluation roles  can help you prioritize what evaluations you do when.  I was then reminded of another approach to viewing evaluation that relates to prioritizing evaluations that might also be useful.

When I first started in this work, I realized that I could view evaluation in three parts–process, progress, product.  Each part could be conducted or the total approach could be used.  This approach provides insights to different aspects of a program.  It can also provide a picture of the whole program.  Deciding on which part to focus is another way to prioritize an evaluation.

Process evaluation captures the HOW of a program.  Process evaluation has been defined as the evaluation that assesses the delivery of the program (Scheirer, 1994).  Process evaluation identifies what the program is and if it is delivered as intended both to the “right audience” and in the “right amount”.  The following questions (according to Scheirer) can guide a process evaluation:

  1. Why is the program expected to produce its results?
  2. For what types of people may it be effective?
  3. In what circumstances may it be effective?
  4. What are the day-to-day aspects of program delivery?

Progress evaluation captures the FIDELITY of a program–that is, did the program do what the planners said would be done in the time allotted? Progress evaluation has been very useful when I have grant activities and need to be accountable for the time-line.

Product evaluation captures a measure of the program’s products or OUTCOMES.  Sometimes outputs are also captured and this is fine.  Just keep in mind that outputs may be (and often are) necessary; they are not sufficient for demonstrating the impact of the program.  A product evaluation is often summative.  However, it can also be formative, especially if the program planners want to gather information to improve the program rather than to determine the ultimate effectiveness of the program.

This framework may be useful in helping Extension professionals decide what to evaluate and when.  It may help determine what program needs a process, progress, or product evaluation.  Trying to evaluate all your program all at once often defeats being purposeful in your evaluation efforts and often leads to results that are confusing, invalid, and/or useless.  It makes sense to choose carefully what evaluation to do when–that is, prioritize.

Formative–Summative

Posted on by

A question was raised in a meeting this week about evaluation priorities and how to determine them.  This reminded me that perhaps a discussion of formative and summative was needed as knowing about these roles of evaluation will help you answer your questions about priorities.

Michael Scriven coined the terms formative and summative evaluation in the late 1960s.  Applying these terms to the role evaluation plays in a program has been and continues to be a useful distinction for investigators.  Simply put, formative evaluation provides information for program improvement.  Summative evaluation provides information to assist decision makers in making judgments about a program, typically for adoption, continuation, or expansion.  Both are important.

When Extension professionals evaluate a program at the end of an training or other program, typically, they are gathering information for program improvement.  The data gathered after a program are for use by the program designers to help improve it.  Sometimes, Extension professionals gather outcome data at the end of a training or other program.  Here, information is gathered to help determine the effectiveness of the program.  These data are typically short term outcome data, and although they are impact data of a sort, they do not reflect the long term effectiveness of a program.  These data gathered to determine outcomes are summative.  In many cases, formative and summative are gathered at the same time.

Summative data are also gathered to reflect the intermediate and long term outcomes.  As Ellen Taylor-Powell points out when she talks about logic models, impacts are the social, economic, civic, and/or environmental consequences of a program and tend to be longer term.  I find calling these outcomes condition changes helps me keep in mind that they are the consequences or impacts of a program and are gathered using a summative form of evaluation.

So how do you know which to use when?  Ask yourself the following questions:

  1. What is the purpose of the evaluation? Do you want to know if the program works or if the participants were satisfied?
  2. What are the circumstances surrounding the program?Is the program in its early development or late development? Are the politics surrounding the program challenging?
  3. What resources are available for the evaluation? So you have a lot of time or only a few weeks?  Do you have access to people to help you or are you on your own?
  4. What accountability is required? Do you have to report about the effectiveness of a program or do you just have to offer it?
  5. What knowledge generation is expected or desired? Do you need to generate scholarship or support for promotion and tenure?

Think of the answers to these questions as a decision tree as the answers to these questions will help you prioritize your evaluation.  Those answers will help you decide if you are going to conduct a formative evaluation, a summative evaluation, or include components of both in your evaluation.