change « Evaluation is an Everyday Activity

From Social Networks: What Maslow Misses | Psychology Today – via @mslogophiliac

“Humans are social animals for good reason. Without collaboration, there is no survival. It was not possible to defeat a Woolley Mammoth, build a secure structure, or care for children while hunting without a team effort. It’s more true now than then. Our reliance on each other grows as societies became more complex, interconnected, and specialized. Connection is a prerequisite for survival, physically and emotionally.”

This statement, which I found on Harold Jarche ‘s blog, applies to evaluation as much as it applies to the example provided by Psychology Today.

Evaluation is a collaborative effort; a team effort, a social effort. Without the collaboration, evaluation lacks much. I’m not sure that survival is dependent on evaluation and collaborative effort; perhaps. The evaluator may know all about evaluation and yet not be able to solve the problem presented by the client because the evaluator doesn’t know about the topic needing to be evaluated. The evaluator may know about something similar to and different from what the client needs and yet, not know about the specific problem. Let me give you an example.

I’ve done a lot of evaluation in the natural resources area and as a result, I’ve learned much about various natural resource topics, including horticulture, plant science, crop science, marine science. I do not know much about potatoes. A while back, a colleague called me and asked if I could/would serve as the evaluator on a ZEBRA CHIP project. Before I said, Sure, I asked about ZEBRA CHIP. Apparently, it is a potato disease transmitted by bacteria carrying psyllid that is causing much economic devastation among growers. It shows up best when the potatoes are made into chips (hence the name). It looks like this: . To me, it isn’t particularly stripped like the animal which offers its name, yet it doesn’t look like potato chips I’m used to seeing. I”m told that there is an unpleasant flavor to the chips as well. I knew a lot about growing things, not about potatoes, even though I’ve worked with potato growers before, just not about this disease.

So, I said sure, only to discover that I have 11 lines in which to write a cogent evaluation section for the work that Extension will be doing (if the grant is funded). If the grant is funded, it will be a five year effort. A continuation actually, which brings me full circle–a collaboration of multiple universities, multiple disciplines, multiple investigators. So what could I say cogently in 11 lines? I suggested that perhaps looking at intention and confidence would be appropriate because we (remember, I said, “Sure”) would not be able to measure actual behavior change. And to overcome the psyllids and eradicate this problem (not unlike the spotted wing drosophila which is affecting the soft fruits of the NW, specifically blueberries), we would need to get as close to behavior change as possible once the teaching has occurred. How can social media be used here? Good question–something to explore. At what level of Maslow’s hierarchy is this collaboration? Survival, sure. Somehow I don’t think Maslow was focused on economic survival.

my .

molly.

Wikipedia says the phrase of “dog days” refers to the sultry (read: HOT) days of summer, typically July and August in the northern hemisphere. In Oregon, it is definitely August. It was 100.4 yesterday; 96 the day before. I didn’t sleep well, even with fans and cool evenings (low 60s). This is the hottest summer I’ve experienced in the 16 years I’ve been in Oregon–there were 10 days of hot weather in May, June, and July; and now so far in August. The Dog days are also when the star Sirius can be seen (assuming there is no cloud cover, a common phenomenon in Oregon–if not the winter rainy season, then summer thunderstorms.) Dog days were thought to be when “the Sea boiled, the Wine turned sour, Dogs grew mad, and all other creatures became languid; causing to man (sic), among other diseases, burning fevers, hysterics, and phrensies.” according to Brady’s Clavis Calendaria, 1813 (A Compendious Analysis of the Calendar; Illustrated with Ecclesiastica, Historical, and Classical Anecdotes 2 volumes). Continue reading →

Periodically I read Harold Jarche’s blog, Shedding Light on Workplace Transformation. The post on June 13, 2014 talks about the topic of “work is changing”. He offered (as he is wont to do) some insights he has gleaned from social media over the the past fortnight including a statement from Hugh MacLeod @gapingvoid. MacLeod also wrote about what is corporate culture. A relevant topic for evaluators.

It gave me pause, especially in light of evaluation and the work I do. MacLeod says, “The nature of work is changing. People’s relationship with work is changing. The changes to society will be vast.” Because the nature of work is changing, the nature of evaluation is also changing. The change in evaluation is evidenced by looking at the evolution of evaluation from the early 1960s to the present. That is over 50 years of evolution–I know that in the grand scheme of things, 50 years is nothing. For this profession, it is a lot. It kinda mirrors the leaps and bounds in technology. Of that 50+ years, I’ve been in the field over 30; I’ve seen so many changes. Recognizing the time to step out and let the next generation of evaluators work is critical. Reinventing yourself so that you can still play in the sandbox (so to speak) is also important.

I recently had a conversation with a long time friend who has been successful at reinventing self. At the age where most people want only to play with their grandchildren , my friend is jetting around the world looking NOT at evaluation rather another field entirely, albeit still professionally and scholarly invested; being invited as a consultant, a speaker, a presenter at various meetings. Another long-time friend of mine who has been “retired” for 10 years has just finished a new book and has not one, TWO monographs in press. This is retirement?

Not only are evaluators re-inventing themselves to match the times, evaluation is also evolving, changing, and thinking differently than 50 years ago. Is this the evolution of a profession? Is this the evolution of the workforce? Is this the evolution of work? Yes. Yes. Yes.

What are you doing to further the profession? How will your contribution to the field help make the profession better? Let me know.

my .

molly.

What? So what? Now what?

Sounds like an evaluation problem.

King and Stevahn (in press) tells us the first query requires thoughtful observation of a situation; the second query a discussion of possible options and implications of those options, and the third query calls for the creation of a list of potential next steps.

Yet these are the key words for “adaptive action” (If you haven’t looked at the web site, I suggest you do.) One quote that is reflective of adaptive action is, “Adaptive Action reveals how we can be proactive in managing today and influencing tomorrow.”( David W. Jamieson, University of St. Thomas). Adaptive action can help you

Understand the sources of uncertainty in your chaotic world
Explore opportunities for action and their implications as they occur
Learn a simple process that cuts through complexity
Transform the work of individuals, teams, organizations and communities
Take on any challenge—as large as a strategic plan or small as a messy meeting
Take action to improve productivity, collaboration and sustainability

Evaluation is a proactive (usually) activity (oh, I know that sometimes evaluation is flying by the seat of your pants and is totally reactive). People are now recognizing that evaluation will benefit them, their programs, and their organizations and that it isn’t personal (although that fear is still out there).

Although the site is directed towards leadership in organizations, the key questions are evaluative. You can’t determine “what” without evidence (data); you can’t determine “so what” unless you have a plan (logic model), and you can’t think about “now what” unless you have an outcome that you can move toward. These questions are evaluative in contemporary times because there are no simple problems any more. (Panarchy approaches similar situations using a similar model .) Complex situations are facing program people and evaluators all the time. Using adaptive action may help. Panarchy may help (the book is called Panarchy by Gunderson and Hollings .)

Just think of adaptive action as another model of evaluation.

molly.

Recently I came across some old note of mine, from some meeting several years ago. I though it would be useful in my writing so I saved it; actually there were two notes that were similar in content. They both relate to blogging, although at the time I didn’t know I would be blogging.

I lump them all under the title of taking a stand, although stance would probably be more descriptive.

The notes are these:

Know your audience.
Be proactive to anticipate needs.
Be reactive to meet needs.
Be authentic.
Be direct.
Be unapologetic.

Continue reading →

The US has been a country for 238 years. A long time. Perhaps it is an opportunity to reflect on what are the rights, privileges, and obligations of citizenship. Perhaps it is just another holiday. Perhaps it is just a time for blueberry pie and peach ice cream. Perhaps it is a…fill in the blank.

I’m not feeling particularly patriotic. I am feeling very evaluative. Recently I viewed a map indicating that on a US passport an individual could travel to 172 different countries. The only country passports which were more powerful (i.e., able to visit more countries) were UK, Finland and Sweden. I wonder to where (what country) can’t I travel on my US passport? That question requires evidence. That is evaluative. I value my US passport. My girls and I travel with them even though driver’s license would be easier. (Being able to fly to Paris at a Continue reading →

A colleague asked, “How do you design an evaluation that can identify unintended consequences?” This was based on a statement about methodologies that “only measure the extent to which intended results have been achieved and are not able to capture unintended outcomes (see AEA365). (The cartoon is attributed to Rob Cottingham.)

Really good question. Unintended consequences are just that–outcomes which are not what you think will happen with the program you are implementing. This is where program theory comes into play. When you model the program, you think of what you want to happen. What you want to happen is usually supported by the literature, not your gut (intuition may be useful for unintended, however). A logic model lists as outcome the “intended” outcomes (consequences). So you run your program and you get something else, not necessarily bad, just not what you expected; the outcome is unintended.

Program theory can advise you that other outcomes could happen. How do you design your evaluation so that you can capture those. Mazmanian in his 1998 study on intention to change had an unintended outcome; one that has applications to any adult learning experience (1). So what method do you use to get at these? A general question, open ended? Perhaps. Many (most?) people won’t respond to open ended questions–takes too much time. OK. I can live with that. So what do you do instead? What does the literature say could happen? Even if you didn’t design the program for that outcome. Ask that question. Along with the questions about what you expect to happen.

How would you represent this in your logic model–by the ubiquitous “other”? Perhaps. Certainly easy that way. Again, look at program theory. What does it say? Then use what is said there. Or use “other”–then you are getting back to the open ended questions and run the risk of not getting a response. If you only model “other”–do you really know what that “other” is?

I know that I won’t be able to get to world peace, so I look for what I can evaluate and since I doubt I’ll have enough money to actually go and observe behaviors (certainly the ideal), I have to ask a question. In your question asking, you want a response right? Then ask the specific question. Ask it in a way that elicits program influence–how confident the respondent is that X happened? How confident the respondent is that they can do X? How confident is the respondent that this outcome could have happened? You could ask if X happened (yes/no) and then ask the confidence questions (confidence questions are also known as self-efficacy). Bandura will be proud. See OR OR (for discussions of self-efficacy and social learning).

molly.

1. Mazmanian, P. E., Daffron, S. R., Johnson, R. E., Davis, D. A., Kantrowitz, M. P. (1998). Information about barriers to planned change: A randomized controlled trial involving continuing medical education lectures and commitment to change. Academic Medicine 73(8), 882-886.

Recently, I received the following comment: “In today’s world it’s virtually impossible to keep up with facebook, twitter, news, tv, movies email, texts, etc.”

It was in response to a blog post about making a difference. How do you know? Given that most of what was suggested happens in the virtual world, the play on words is interesting. How is it impossible–because there is too much information? because you are too distracted by the virtual part of all the information and get lost? because virtuality it is not clearly understood? because of something else? I personally find I can get lost when I spend all day on line (virtual). It isn’t real, actually. I have no sense of what is happening and what isn’t happening. Even with the feeds from news lines, I find I have to double check my facts. Yet even as I say this, the virtual is expanding (go here). I have heard about Web 2.0; hadn’t heard about IoE (Internet of Everything)…the CEO of Cisco (John Chambers) stated that the IoE depends on the architecture, the systems integration. Is virtual the way of the world? It certainly isn’t the future any more; it is now. I have to ask, though, what about people…Given that much evaluation is now being done with the use of virtual tools, are we really understanding what difference is being made? Or are there just connections?

The individual continued with the comment by saying, “Keep up your small voice. Some are listening.” Those “listening” are certainly reflected in the number of comments I received on the posts about making a difference in the last two days (over 45). This may certainly be a way of engaging; I know it is outreaching. It is only my small voice; it is rewarding to know that some are listening/reading. Even if they only stay a short while.

My two cents. (my small voice).

molly.

In a recent post, I said that 30 was the rule of thumb, i.e., 30 cases was the minimum needed in a group to be able to run inferential statistics and get meaningful results. How do I know, a colleague asked? (Specifically, “Would you say more about how it takes approximately 30 cases to get meaningful results, or a good place to find out more about that?”) When I was in graduate school, a classmate (who was into theoretical mathematics) showed me the mathematical formula for this rule of thumb. Of course I don’t remember the formula, only the result. So I went looking for the explanation. I found this site. Although my classmate did go into the details of the chi-square distribution and the formula computations, this article doesn’t do that. It even provides an Excel Demo for calculating sample size and verifying this rule of thumb. I am so relieved that there is another source besides my memory.

New Topic:

Continue reading →

I had a comment about last week’s post on Sustainability and Evaluation. I will share it here. I wonder what you readers think of this comment:

In preventive health/health promotion, ‘sustainability’ has generally been used to indicate that the intervention program, or elements of it, or benefits of it, are continued beyond the life of the funded program. It is about extending the value of the investment in a program, beyond the life of the funded program. So it’s about a legacy, about the continuation of things achieved, about leaving things, circumstances or communities better off than when you first arrived (more empowered, more resourceful, more able to continue improvements on their own).

I wonder how that fits with the definitions I provided? Is this a different sustainability? Does it speak to the future generations? Does that include equity and justice? Continue reading →

Evaluation is an Everyday Activity

Program Evaluation Discussions

Tag Archives: change

Evaluation, Maslow’s hierarchy of needs, and social media

Dog days of summer

Nature of work and evaluation

Adaptive action–evaluation by another name?

Taking a stand

Tomorrow is July 4th

Unintended? Unanticipated? Un…?

Virtual impossibility

30 is the rule of thumb…

Wednesday’s musings

Contact Info