Jul
13
Filed Under (criteria, program evaluation) by Molly on 13-07-2016

I want to talk about learning. Real learning. This week I am borrowing a blog from another writer intact. I have never done this. True, I have taken parts of blogs and quoted them. This blog post from the blog called “adapting to perpetual beta” by Harold Jarche is applied here in its entirety because I think the topic is important. I have added the visuals except for the Rodin, which was in the original post.

Yes, it relates to evaluation. We learn (those who value evaluation) throughout our careers. The various forms of learning are engaged (see: Edgar Dale who designed the learning cone though not with percentages that are usually attributed to the styles).Cone of learning(This particular version was developed by Bruce Hyland based on Dale’s work.) When you read the post below, think about how you learn. Engages? Reflective?

real learning is not abstract

Posted 2016-06-20

Are we entering an era that heralds ‘The End of Reflection’, as this NY Times article suggests?

Read the rest of this entry »

Apr
24
Filed Under (program evaluation) by Molly on 24-04-2015

About two years ago, I conducted a 17 month hybrid evaluation preparation program for the Western Region Extension Service faculty. There were over 30 individuals involved. I was the evaluation expert; Jim LindstromJames-Lindstrom (who was at WSU at the time) was the cheerleader, the encourager, the professional development person. I really couldn’t have done it without him. (Thank you, Jim.) Now, to maximize this program and make it available to others who were not able to participate, I’ve been asked to explore an option for creating an on-line version of the WECT (say west) program. It would be loaded through the OSU professional and continuing education (PACE) venue. To that end, I am calling on those of you who participated in the original program (and any other readers) to provide me with feedback of the following:

  1. What was useful?
  2. What needed to be added?
  3. What could be more in depth?
  4. What could be deleted?
  5. Other comments?

Please be as specific as possible.

I can go to the competency literature (of which there is a lot) and redevelop WECT from those guidelines.  (For more information on competencies see: King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22(2), 229-247.) Or I could use the Canadian system as a foundation. (For more information see this link.)

I doubt if I can develop an on-line version that would cover (or do justice) to all those competencies.

So I turn to you my readers. Let me know what you think.

my two cents.

molly.

Jan
23
Filed Under (program evaluation) by Molly on 23-01-2015

I was reading Harold Jarche’s blog, Perpetual Beta and he is talking about the interface of the digital and analog worlds; he is talking about connections. connections 2 I find that that concept applies to evaluators. Here’s how.

I was trained as an evaluator in the 1980s; we didn’t have access to the web, internet, email, FB, or many of the other high technology options available today. I did an NDE (wasn’t called that then) volume (Vol. 62) which was all done by hard copy and the USPS (a tedious and laborious process). I just completed another NDE (now called that) volume (Vol. 145) that was done electronically (no longer tedious, though still a laborious process). This last volume was quick. Although most of the authors entered the field after I did, my co-editor didn’t; he entered before I did. These authors had the luxury of electronics that we didn’t have. I have had to learn how to use electronics; I know my co-editor did, too. (I remember saying to myself and my colleagues, what will my secretary [yes, we used that title] do now that I’m composing on a key-board?) Now I do all my composing and other things on a keyboard; most of my work is augmented with electronics (i.e., the digital world). But I am truly a digital immigrant; learning how to use this new technology, to be in this digital world, is hard for and on me. (My children laugh at me and get exasperated; it is so simple to them.)

Today’s evaluators are highly connected, largely due to the electronic capabilitieselectronic connections. Yet there is still evaluator isolation. Makes me wonder if evaluators really connected? Sheila Robinson (the only evaluator in her organization bemoans this fact here). She advocates for connections through EvalTalk and AEA’s LinkedIn account. I still see these as digital, albeit, opportunities to connect. Social media is also mentioned for connections. Still I wonder–are you really connected? With colleagues scattered around the world, this may truly be the only way to stay connected today. Letters and phone calls are truly analog and perhaps passe. Still they are appreciated and sometimes welcome as the only way to connect. What will this world look like if the only way to connect is via digital?

Jarach advocates changing the way we organize. To me that talks about changing the way we work. Maybe evaluators will work in isolation. In order to “see as many possible roads ahead”, perhaps we need to “work in self-managing networks”. networks 2 “If those who are educated, knowledgeable, and experienced do not push for a better world of work, then who will? An effective knowledge network cultivates the diversity and autonomy of each worker. Knowledge networks function best when each person can choose with whom and when they connect. Solving problems together is becoming the real business challenge.” (From Jarach)

And that affects evaluators!

To whom are you connected? How do you connect? Are you caught in-between?

mytwo cents.

molly.

Sep
25
Filed Under (program evaluation) by Molly on 25-09-2014

Over the last several months, the Local Arrangements Working Group has been blogging at AEA365. One of ways evaluators can get ready for the upcoming annual conference is to read what the LAWG has to say about the conference. This year, the conference is once again in Denver. AEA was in Denver in 2008. Be forewarned–Denver is the mile high city. The air is rarefied and very dry. It may snow!

The LAWG has a lot to say about the conference and there are A LOT of links in these posts that are worth checking.  For those who have not been to AEA before or for those who have recently embraced evaluation, reading their posts are a wealth of information, as is the AEA website.

I will be presenting at two sessions this year–one on blogging (duh…) and one on capacity building. I see them related. I will also (like last year) be assisting with a professional development session (number 25) with my long time friend and colleague, Jim Altschuld. The professional development session occurs on Wednesday, October 15, 2014 from 8:00am MT – 3:00pm MT. It is titled Practical Ways to Link Needs Assessment (NA) with Asset/Capacity Building. (Just a little advertisement 🙂 ) It will draw from his new book, Bridging the Gap Between Asset/Capacity Building and Needs Assessment. Bridging the Gap-altschuld Read the rest of this entry »

Sep
04

Wow!  25 First Cycle and 6 Second Cycle methods for coding qualitative data.

Who would have thought that there are so many methods of coding qualitative data.  I’ve been coding qualitative data for a long time and only now am I aware that what I was doing was, according to Miles and Huberman (1994), my go-to book for coding,  miles and huberman qualitative data is called “Descriptive Coding” although Johnny Saldana calls it “Attribute Coding”.  (This is discussed at length in his volume The Coding Manual for Qualitative Researchers.) coding manual--johnny saldana  I just thought I was coding; I was, just not as systematically as suggested by Saldana.

Saldana talks about First Cycle coding methods, Second Cycle coding methods and a hybrid method that lies between them.  He lists 25 First Cycle coding methods and spends over 120 pages discussing first cycle coding.

I’m quoting now.  He says that “First Cycle methods are those processes that happen during the initial coding of data and are divided into seven subcategories: Grammatical, Elemental, Affective, Literary and Language, Exploratory, Procedural and a final profile entitled Themeing the Data.  Second Cycle methods are a bit more challenging because they require such analytic skills as classifying, prioritizing, integrating, synthesizing, abstracting, conceptualizing, and theory building.”

He also insists that coding qualitative data is a iterative process; that data are coded and recoded.  Not just a one pass through the data.

Somewhere I missed the boat.  What occurs to me is that since I learned about coding qualitative data by hand because there were few CAQDAS (Computer Assisted Qualitative Data Analysis Software) available (something Saldana advocates for nascent qualitative researchers) is that the field has developed, refined, expanded, and become detailed.  Much work has been done that went unobserved by me.

He also talks about the fact that the study’s qualitative data may need more than one coding method–Yikes!  I thought there was only one.  Boy was I mistaken.  Reading the Coding Manual is enlightening (a good example of life long learning).  All this will come in handy when I collect the qualitative data for the evaluation I’m now planning.  Another take away point that is stressed in the coding manual and in the third edition of the Miles & Huberman book (with the co-author of Johnny Saldana) Qualitative data analysis ed. 3 is to start coding/reading the data as soon as it is collected.  Reading the data when you collect it allows you to remember what you observed/heard, allows/encourages  analytic memo writing (more on that in a separate post), and allows you to start building your coding scheme.

If you do a lot of qualitative data collection, you need these two books on your shelf.

 

Jul
31
Filed Under (program evaluation) by Molly on 31-07-2013

The American Evaluation Association has opened its registration for the 2013 meeting in Washington DC.  This meeting promises to be attended by the most people yet.  Eleven years ago we were in D. C. and broke all attendance records to date.  I remember because that was my presidential year…the year that the evaluation profession started thinking that evaluation was a system; that everything we do is connected.  Several people have commented about AEA–that they didn’t know there was such an association; that they didn’t know about the conference; that they weren’t members.  So folks, here is the skinny on AEA (at least part of the skinny…).

The American Evaluation Association was officially founded in 1986 AEA logoas a combined organization of the Evaluation Research Society and Evaluation Network.  ERS was academic and EN was practitioner; merging the two was a challenge as each thought something would be lost.  This is a good example of where the whole is greater that the sum of its parts.  The differences were pronounced and debatable (now you only see AEA).  Robert (Bob) B. Ingle was the force behind the conference; he mounted the first EN/ERS conference in 1981 in Austin, Texas.  I was a graduate student.  I was in awe.  Although I had been to numerous  professional conferences before attending this first conference, I had never met any one like Bob Ingle.  His first comment to me once we connected after playing phone tag was, “You spell your name wrong!”  (Turns out he was the Scotland branch of the German house of Engel; my ancestors changed the spelling when they came out of Germany.)   I was a nascent graduate student in love with my studies and here comes this brusque, acerbic, and outrageous giant.  He became my good friend–I knew him from 1981 until he died in 1998.  He believed passionately in program evaluation.  I think he is smiling at the growth in the profession and the organization.  He knew a lot of us; he saw the association through the good times and the bad times.  I could end here and say, the rest is history…only there is so much to tell.  The association went from an all volunteer organization at its founding in 1986 to an organization of over 8,000 members run by an association management firm.  Susan Kistler (of Kistler Associates) was our executive director for the last 15 years.  (The association has transitioned to a new management firm [SmithBucklin] and a new executive director [Denise Roosendaal]).  Seeing the association transition is bittersweet;  growth is good, the loss of family feeling is sad.  The association is no longer feels intimate, family; it offers so much more to folks who are members.

David Bernstein is the co-chair of the local arrangements working group (LAWG) for this year’s conference.  He lead off a week of  AEA365 talking about the conference.  Read this post.  It tells you a lot about the conference. This week AEA365 is being written by the local arrangements working group.  The role of the local arrangements group is to make sure the folks who attend the conference have a good time, both at the conference and in DC.  DC is a wonderful city.  You cannot see it in a week; it is always changing.  Take a day if you have never been to see the city’s high points.  It is the nation’s capitol, after all, and there are many high points.

The members only AEA August newsletter also talks about registration with hyperlinks to the registration site, the conference program, and hotel accommodations.  (The members only newsletter is just one reason to join AEA.)  I’ve been going to AEA since 1981.  This is the first year I will not have a paper/poster/etc. on the program.  (I am doing a professional development session with Jim Altschuld, though; it is number 22).

Each year I attend AEA, I think of the three evaluative criteria that FOR ME makes a good conference:  See three long time friends; meet three new people who could become friends; and get three new ideas.  If I do all this, I usually come home energized.  I hope to see you there.

Mar
20
Filed Under (program evaluation) by Molly on 20-03-2013

Harold Jarche shared in his blog a comment by a participant in one of his presentations.  The comment is:

Knowledge is evolving faster than can be codified in formal systems and is depreciating in value over time.

 

This is really important for those of us who love the printed work (me) and teach (me and you).  A statement like this tells us that we are out of date the moment we open our mouths; those institutions on which we depended for information (schools, libraries, even churches) are now passe.

 

The exponential growth of knowledge is much like that of population.   I think this graphic image of population (by Waldir) is pretty telling (click on the image to read the fine print).  The evaluative point that this brings home to me is the delay in making information available.

O

Do you (like me) when you say, “Look it up”, think web, not press, books, library, hard copy?  Do you (like me) wonder how and where this information originated when the information is so cutting edge?  Do you (like me) wonder how to keep up or even if you can?  Books take over a year to come to fruition (I think the 2 year frame is more representative).  Journal manuscripts take 6 to 9 months on a quick journal turn around.  Blogs are faster and they express opinion; could they be a source of information?

I’ve decided to go to an advanced qualitative data seminar this summer as part of my professional development because I’m using more and more qualitative data (I still use quantitative data, too).  It is supposed to be cutting edge.  The book on which the seminar is based won’t be published until next month (April).  How much information has been developed since that book went to press?  How much information will be shared at the seminar?  Or will that seminar be old news (and like old news, be ready for fish)?  The explosion of information like the explosion of population, may be a good thing; or not.  The question is what is being done with that knowledge?  How is it being used?  Or is it?  Is the knowledge explosion an excuse for people to be information illiterate? To become focused (read narrow) in their field?   What are you doing with what I would call miscellaneous information that is gathered unsystematically?  What are you doing with information now–how are you using it for professional development–or are you?