Monthly Archives: November 2013

Update.

Posted on by

As a bonus, there are two posts this week (I won’t be posting next week…Thanksgivukkah thanksgivukkah image).

I keep getting comments about my post “Is this post making a difference” and the subsequent posts related to that.  (The survey is closed–has been for a long time.)  Although I was looking for some tangible measure of difference (i.e., what is the merit, worth, value of this blog), I find that this post elicits more positive comments than not.  Although I still get the random advertising post on that blog, I mostly get thought provoking comments about how it is or can be making a difference.  I also get the comment at least weekly that says my blog isn’t making any difference to the reader (perhaps the reader needs to follow the blog across time rather than just reading this post) …sigh…

One comment I received this week said, “I can confirm that the information that you share has: 1) made sense; 2) made a difference; 3)  been worthwhile.  Keep it up; don’t stop.”  Nice.  Not specific; still nice.  That the blog is being read is important.  That people take the time to comment is important.  That this venue is valuable to many is important.

I get positive comments on my writing; I get positive comments on my content.  I’m an evaluator.  I want to know what difference this information is making in your life.  This is a program and it needs to be evaluated.  And number of page views isn’t the answer.

So I ask you to keep commenting, that way I know the blog is being read.  That way I know I am making a difference.

Thanksgivukkah

Posted on by

For the first time in my lifetime the first day of Hanukkah is also Thanksgiving.  The pundits are are sagely calling the event Thanksgivukkah.thanksgivukkah image  According to this referenced source, the first day of Hanukkah will not happen again for over 70,000 years.  However, according to another source, this overlap could happen again in 2070 and 2165.  Although I do not think I’ll be around in 2070, my children could be (they are 17 and 20 of this writing).  I find this phenomenon really interesting–Thanksgiving usually starts the US holiday season and Hanukkah falls later, during Advent.  Not so this year.  I wonder how people combine latkes and Thanksgiving (even without the turkey).  Loaded latkes? Thanksgivukkah latkes (My appreciation to Kia.)

So I’m sure you are wondering, HOW EXACTLY DOES THIS RELATE TO EVALUATION?

I decided that it was time to revisit my blog title, Evaluation is an Everyday Activity. Every day you evaluate something.  Although you do not necessarily articulate out loud the criteria against which you are determining merit, worth, and value, you have those criteria.  I have them for latkes AND Thanksgiving.  Our latkes must be crispy; of winter vegetables including potatoes.  This allows me to use a variety of winter vegetables I may have gotten in my CSA.  (Beet latkes? Sweet potato latkes?  Celeriac latkes?  You bet!)   Our Thanksgiving is to have foods for which we are truly thankful.  That allows us to think about gratitude.  Each year our menu is different because each year we are thankful for different things.  (I must confess, however, we always have pie–pumpkin, which I make from home grown pumpkin/squash, and chocolate pecan, which is an original old family recipe.)  One year when we put all the food on the table, all the food was green.  We didn’t plan it that way; it just happened because they were foods for which we were thankful.  This year, we will have mashed potatoes (by the Queen of mashed potatoes), Celebration Filo, both the gluten-free (made with rice wrappers and no onion, garlic, or dairy) and glutened versions (the version which we renamed and is in the link above), and something else that will probably be green.  This year I’m thankful for my gluten-free; dairy-free friend who will join us for Thanksgiving and I’m working up alternatives to accommodate her and still satisfy the rest of us.

So you see, even when I’m thinking about Thanksgiving, latkes, and gratitude, I’m thinking about evaluation.  What merit does the “program” have?  What is its worth?  What is its value?  Those are all evaluative questions that apply to Thanksgiving (and latkes and gratitude). Thanksgiving 2

So you see, Evaluation is an Everyday Activity.

I won’t be blogging next week.  Enjoy.  Be grateful.Thanksgiving

 

 

Variables and other things

Posted on by

Variables.

We all know about independent variables, and dependent variables.  Probably even learned about moderator variables, control variables and intervening variables.  Have you heard of confounding variables?  Variables over which you have no (or very little) control.  They present as a positive or negative correlation with the dependent and independent variable.  This spurious relationship plays havoc with analyses, program outcomes, and logic models.  You see them often in social programs.

Ever encounter one? (Let me know).  Need an example?  Here is one a colleague provided.  There was a program developed to assist children removed from their biologic  mothers (even though the courts typically favor mothers) to improve the children’s choices and chances of success.  The program had included training of key stakeholders (including judges, social service, potential foster parents).  The confounding variable that wasn’t taken into account was the sudden appearance of the biological father.  Judges assumed that he was no longer present (and most of the time he wasn’t); social service established fostering without taking into consideration the presence of the biological father; potential foster parents were not allerted in their training of the possibility.  Needless to say, the program failed.  When biologic fathers appeared (as often happened), the program had no control over the effect they had.  Fathers had not been included in the program’s equation.

Reviews.

Recently, I was asked to review a grant proposal, the award would result in several hundred thousand dollars (and in today’s economy, no small change).  The PI’s passion came through in the proposal’s text.  However, the PI and the PI’s colleagues did some major lumping in the text that confounded the proposed outcomes.  I didn’t see how what was being proposed would result in what was said to happen.  This is an evaluative task.  I was charged to with evaluating the proposal on technical merit, possibility of impact (certainly not world peace), and achievability.  The proposal was lofty and meant well.  The likelihood that it would accomplish what it proposed was unclear, despite the PI’s passion.  When reviewing a proposal, it is important to think big picture as well as small picture.  Most proposals will not be sustainable after the end of funding.  Will the proposed project be able to really make an impact (and I’m not talking here about world peace).

Conversations.

I attended a meeting recently that focused on various aspects of diversity.  (Now among the confounding here is what does one mean by diversity; is it only the intersection of gender and race/ethnicity?  Or something bigger, more?)  One of the presenters talked about how just by entering into the conversation, the participants would be changed.  I wondered, how can that change be measured?  How would you know that a change took place?  Any ideas?  Let me know.

Focus groups.

A colleague asked whether a focus group could be conducted via email.  I had never heard of such a thing (virtual, yes; email, no).  Dick Krueger and Mary Ann Casey only talk about electronic reporting in their 4th edition of their Focus Group book. krueger 4th ed  If I go to Wikipedia (keep in mind it is a wiki…), there is a discussion of online focus groups.  Nothing offered about email focus groups.  So I ask you, readers, is it a focus group if it is conducted by email?

 

 

 

Quandry

Posted on by

I had a topic all ready to write about then I got sick.  I’m sitting here typing this trying to remember what that topic was, to no avail. That topic went the way of much of my recent memory; another day, perhaps.

I do remember the conversation with my daughter about correlation.  She had a correlation of .3 something with a probability of 0.011 and didn’t understand what that meant.  We had a long discussion of causation and attribution and correlation.

We had another long conversation about practical v. statistical significance, something her statistics professor isn’t teaching.  She isn’t learning about data management in her statistics class either.  Having dealt with both qualitative and quantitative data for a long time, I have come to realize that data management needs to be understood long before you memorize the formulas for the various statistical tests you wish to perform.  What if the flood happens????lost data

So today I’m telling you about data management as I understand it, because the flood  did actually happen and, fortunately, I didn’t loose my data.  I had a data dictionary.

Data dictionary.  The first step in data management is a data dictionary.   There are other names for this, which escape me right now…know that a hard copy of how and what you have coded is critical.  Yes, make a back up copy on your hard drive…have a hard copy because the flood might happen. (It is raining right now and it is Oregon in November.)

Take a hard copy of your survey, evaluation form, qualitative data coding sheet and mark on it what every code notation you used means.  I’d show you an example of what I do, only they are at the office and I am home sick without my files.  So, I’ll show you a clip art instead…data management    smiley.  No, I don’t use cards any more for my data (I did once…most of you won’t remember that time…), I do make a hard copy with clear notations.  I find my self doing that with other things to make sure I code the response the same way.  That is what a data dictionary allows you to do–check yourself.

Then I run a frequencies and percentages analysis.  I use SPSS (because that is what I learned first).  I look for outliers, variables that are miscoded, and system generated missing data that isn’t missing.  I look for any anomaly in the data, any humon error (i. e. my error).  Then I fix it.  Then I run my analyses.

There are probably more steps than I’ve covered today.  These are the first steps that absolutely must be done BEFORE you do any analyses.  Then you have a good chance of keeping your data safe.