Filed Under (program evaluation) by Molly on 24-09-2010

Ryan asks a good question: “Are youth serving programs required to have an IRB for applications, beginning and end-of-year surveys, and program evaluations?”  His question leads me to today’s topic.

The IRB is concerned with “research on human subjects”.  So you ask, When is evaluation a form research?

It all depends.

Although evaluation methods have evolved from  social science research, there are important distinctions between the two.

Fitzpatrick, Sanders, and Worthen list five differences between the two and it is in those differences that one must consider IRB assurances.

These five differences are:

  1. purpose,
  2. who sets the agenda,
  3. generalizability of results,
  4. criteria, and
  5. preparation.

Although these criteria differ for evaluation and research, there are times when evaluation and research overlap.    If the evaluation study adds to knowledge in a discipline or research informs our judgments about a program, then the distinctions are blurred and a broader view of the inquiry is needed and possibly an IRB approval.

IRB considers children a vulnerable population.  Vulnerable populations require IRB protection.  Evaluations with vulnerable populations may need IRB assurances.  IF you have a program that involves children AND you plan to use the program activities as the basis of an effectiveness evaluation (ass opposed to program improvement) AND use that evaluation as scholarship you will need IRB.

Ryan asks “what does publish mean”.  That question takes us to what is scholarship.  One definition of scholarship is that scholarship is creative work, that is validated by peers and communicated.  Published means communicating to peers in a peer reviewed journal or professional meeting not, for example, in a press release.

How do you decide if your evaluation needs IRB?  How do you decide if your evaluation is research or not?   Start with the purpose of your inquiry.  Do you want to add knowledge in the field?   Do you want to see if what you are doing is applicable in other settings?  Do you want others to know what you’ve done and why?  They you want to communicate this.  In academics, that means publishing it in a peer reviewed journal or presenting it at a professional meeting.  And to do that and use the information provided you by your participants who are human subjects, you will need IRB assurance that they are protected.

Every IRB is different.  Check with your institution.  Most work done by Extension professionals falls under the category of “exempt from full board review”.  It is the shortest review and the least restrictive.  Vulnerable populations, audio and/or video taping, or asking sensitive questions typically is categorized as expedited, a more stringent review than the “exempt” category, which takes a little longer.  IF you are working with vulnerable populations and asking for sensitive information,  doing an invasive procedure, or involving participants in something that could be viewed as coercive, then the inquiry will probably need full board review (which takes the longest turn around time.

Filed Under (program evaluation) by Molly on 01-03-2010

I was asked about the need for an evaluation plan to be reviewed by the institutional review board (IRB) office.  In pausing to answer, the atrocities that have occurred and are occurring throughout the world428px-Durer_Revelation_Four_Riders registered once again with me…the Inquisition, the Crusades, Cortes, Auschwitz, Nuremberg trials, Sudan, to name only a few.  Now, although I know there is little or no evaluation in most of these situations, humans were abused in the guise of finding the truth.  (I won’t capitalize truth, although some would argue that Truth was the impetus for these acts.)

So what responsibility DO evaluators have for protecting individuals who participate in the inquiry we call evaluation?  The American Evaluation Association has developed and endorsed for all evaluators a set of Guiding Principals.  There are five principals–Systematic Inquiry, Competence, Integrity/Honesty, Respect for People, and Responsibilities for the General and Public Welfare.  An evaluator must perform the systematic inquiry competently with integrity, respecting the individuals participating and recognizing diversity of police imagespublic interests and values.  This isn’t a mandated code; there are no evaluation police; an evaluator will not be sanctioned if these principals are not followed (the evaluator may not get repeat work, though).  These guiding principals were established to “guide” the evaluator to do the best work possible within the limitations of the job.

IRB imagesThe IRB is there to protect the participant first and foremost; then the investigator and the institution.  So although there is not a direct congruence with the IRB principals of voluntary participation, confidentiality, and minimal risk, to me, evaluators following the guiding principals will be able to assure participants that they will be respected and that the inquiry will be conducted with integrity.  No easy task…and a lot of work.

I think evaluators have a responsibility embedded in the guiding principals to assure individuals participating in evaluations that participants engage voluntarily, that they provide information that will remain confidential, and that what is expected of them involves minimal risk.  Securing IRB approval will assure participants that this is so.

Although two different monitoring systems (one federal; one professional), I think it is important meet both sets of expectations.