Category Archives: Evaluation

STEAMing Ahead at Full Speed

Posted on by | Leave a reply

What do erupting volcanoes, learning to code, building bridges, creating art, and cooking have in common? They are all STEAM activities! This past year I started working as an external evaluator on Lincoln County School District’s (LCSD) 21st Century After School Program. There are seven sites throughout the county that are funded as 21st Century Community Learning Centers. The goal of the federal program is to “provide academic enrichment opportunities during non-school hours for children…offer students a broad array of enrichment activities that can complement their regular academic programs, and offer literacy and other educational services to the families of participating children.” LCSD’s program aligns with these goals, focusing specifically on providing students and their families STEAM-based activities.

Being a new project in the 2013-2014 school year, the 21st Century After School Program kept me on my toes as an external evaluator. All documents for data collection had to be written very quickly so they could be distributed as close to the start of the school year as possible. While the tools for collecting data did not fail they certainly could have been better. As the program moves into the second of it’s five years, I thought it best not only to redesigned some of the tools but also to communicate to the site coordinators why collecting this data is important. Site coordinators, one for each of the seven sites, are responsible for distributing, collecting, and turning in all of the instruments I use to collect data. The only part of data collection they are not responsible for are the standardized test scores, which are handled by the school district. Collecting data is no small task for the site coordinators because they already have a lot on their plate. Therefore, the program coordinator and I agreed that sharing results from the first year would showcase the importance of data collection.

Early in September I presented preliminary findings at the first site coordinator meeting of the second year of the program. You can view and listen to my presentation here.  The presentation ends with me addressing the four main instruments for data collection: a monthly recording sheet to document the STEAM activities and resources, the student STEM interest survey, a family literacy and STEAM night reporting document, and the parent survey.

Working as an external evaluator on these types of projects is always exciting for me. They push me to think in new ways and certainly make me a better researcher and evaluator. If you have any questions about the 21st Century After School Program or the documents I’ve produced, please feel free to be in touch.

How to Evaluate Jaw-Dropping STE(A)M Project-Based Learning

Posted on by | Leave a reply

As the school year comes to an end so too do the school-based projects I evaluate. What this means, first and foremost, is a mad rush to collect data. It’s also a time for those involved in the project to come together and share what they’ve been doing for the past 8 months. As an evaluator, I have been focused on the mad rush of data collection – writing surveys, distributing surveys, leading focus groups, and conducting site observations. All of this data is needed to prove these projects are doing great things; however, what I truly love is hearing about the activities educators are using to engage their students in STE(A)M.

As an evaluator I have to ask: how do you capture the amazing ideas these educators are coming up with and how do you evaluate the impact they’re having? And by impact I mean both the impact on students and the impact on other educators who are hearing what’s been done in other educational settings. What I’m actually asking is how do you evaluate jaw-dropping moments?

To put these questions into some context I need to clarify that I am, from here on, talking about my experience as one of the evaluators with the Oregon Coast Regional STEM Education Center . The STEM Center is a collaboration between Lincoln County School District, Tillamook County School District, and countless institutions and organizations up and down the Oregon Coast. A U.S. Department of Education Math Science Partnership grant funds the STEM Center, which offers professional development to teachers in both school districts. Over the course of the school year the teachers put Project-Based Learning (PBL) into practice.

Now to turn back to the questions I asked above. How do you capture the amazing ideas these educators are coming up with? We do collect and archive as much as we can, specifically PBL overviews, PowerPoints, assessments, and other resources teachers and students use during PBL and include them all on the website for others to use. 2013-2014 school year PBLs should be up this summer but you can peruse 2012-2013 PBLs here.

IMG_0194

Sean Bedell shows his colleagues a core sample he and his students took while looking for evidence of Oregon’s 1700 tsunami (project further discussed below).

How do you evaluate the impact these projects have? This question is more difficult to answer. For students, we distribute a STEM interest survey at the beginning and end of the school year and we use student test scores, but to me that can’t tell the whole story. The hard pill to swallow as an evaluator is that in order to capture what I would call the true impact on students and the whole story, this project would require longitudinal study (think 5 or 10+ years of collecting data and interviewing students). We also have teachers complete pre- and post-project surveys and have them write a reflection and those sources have proven to be useful in past projects to understand impact. We talked about running a focus group with the 2013-2014 STEM Center teachers to gauge how they incorporate all of the information delivered through professional development to plan and implement their PBLs. Anecdotal evidence shows that teachers are no longer taking a kit or pre-written lessons and using it as is in the classroom; instead, they are taking ideas from multiple sources and piecing together large scale projects. Essentially, their self-efficacy to do PBL and STE(A)M in the classroom is rising.

Most of the teachers presented their 2013-2014 PBL to their colleagues last Saturday. I was in the audience with my jaw on the floor for most of the day. I really appreciated the variety of presentations, which included posters, ignite presentations (i.e. short, sweet, and fast), and student voice-over presentations. In the afternoon some students came and presented PBLs from their perspective.  I can’t cover all of the projects here and encourage any readers to keep checking the website as we add the 2013-2014 PBLs. Here’s a selection of projects that caught my attention:

– Students at Newport Prep Academy studied marbled murrelets and corvids, specifically how the latter prey on the former’s eggs. Human interference (i.e. leaving trash at picnic sites) brings corvids closer to marbled murrelets. Check out the Public Service Announcements produced by the students using iPads and iMovie. QR codes the students created will soon be at picnic areas of state and national parks.

– Students at Eddyville Charter School focused on tsunamis. They designed, built, and tested their own tsunami structure at Hinsdale Wave Research Lab. Students also researched the earthquake and tsunami that hit Oregon in 1700 by taking core samples at five different locations to look for tsunami evidence. Check out their website, which contains videos and student wikis about the project.

– At another school, students had to engineer a closed forest ecosystem to gain an understanding on how we could sustain life on another planet. This was definitely a test-retest project as students had to monitor pH and water levels to keep plants alive. Many students had to re-engineer their plans and use different materials to meet the challenge.

IMG_0199 IMG_0200

Examples of student-designed forest ecosystems.

– In Tillamook, elementary students were given a challenge by the local utility company, which was really a fake letter written by the teacher with the company’s approval. Students had to evaluate different sources of renewable energy and where such sources could be placed within the landscape to be most efficient. At the end of the PBL, students presented their findings to an expert panel.

See? Jaw on the ground! And this is just a sample of what these amazing teachers in Lincoln and Tillamook County School Districts are working on with their students!

 

Nerd team assemble!

Posted on by | Leave a reply

I just got the news that my IRB application was approved, and I am now ready to proceed with play-testing! The prototype for Deme is currently occupying the space under our end table—a strange assortment of bits and pieces gathered into a plastic grocery bag.

The next step is to have my participants to play a round of an established tabletop strategy game. This will help me build my coding frame and observe how the group dynamic works with an established product. After that, we’ll take on Deme itself and discuss its mechanics afterward.*

Exposing a game to actual players can be a scary prospect. This thing has been twisting and turning in my mind for a long time now, and I’ve scaled it back significantly since I started. I want my rule system to be a skeleton, not a cage—this has become my development mantra. I remind myself that this is only the beginning of the process for Deme, and I have to keep the players in mind from the start. The game is about building and exploiting player-defined systems, after all.

Once I have a robust and enjoyable rule system in place, I want Deme to truly belong to anyone who wants to play it. It could succeed. It could fail. It could become something beautiful due to unforeseen changes—or even a complete overhaul—made by someone else.

That’s a long way off, though. Deme is still in the nursery, and I’m inviting a few people in to look it over and make sure it’s healthy. It won’t be moving out on its own until well after this project has concluded. I hope it knows how to do its own laundry by then.

*Oops! This is an idea that we had discussed, but it will not be part of this project. I’ll be building my coding frame and doing my analysis based on Deme alone. This will give me a chance to evaluate Deme more or less on its own terms first. It will also give me more time to analyze the data I have, as I’ll have less video data to crawl through. There’s some good existing literature on player discourse in established games.

Encouraging Communities of Practice

Posted on by | Leave a reply

Over the last couple of months, we’ve been reading about communities of practice (Wenger, McDermott, & Snyder, 2002) for our Friday theory meetings. In that time, I have been busy with Project SEAL. The Model Classroom Team and I were going out to the schools to check in on the teachers and hear about their projects. There are some wonderful teachers doing wonderful projects with their students. Teachers and students are creating a thought-provoking, interactive play space, learning about beach pollution and its impact on the environment, and building a school garden.

While it was exciting to hear about how teachers and students were using the iPad minis available with the grant and to see differences students were making in their school environment, there was something that seemed to be mentioned in each meeting: the teachers needed a community. A community to share and learn within. A community where they could figure out which apps to download, how to download them, and how to use them. A community to collaborate with.

Having this experience came at a great time with reading the book. All of a sudden, I had a framework in which to think about communities of practice. Something that I’ve struggled with as the evaluator of Project SEAL is how I (and the project team) can encourage teachers to develop a community of practice without it seeming like a top-down approach.

To help me think my problem through, I turn to the part in the book about the challenge of distributed communities. Project SEAL is a distributed community – teachers are at 9 different schools throughout Lincoln County. We have a webpage, where teachers post assignments and can find resources, but as the authors write,

“Members cannot see how many other people are reading – and benefiting from – a threaded discussion. Unlike in-person meetings, teleconferences and Web sites don’t offer easy opportunities for informal networking. Because of these barriers, it takes more intentional effort for members to consult the community for help, spontaneously share ideas, or network with other members” (p. 117).

I think the key word there is easy. It can be done but needs to be encouraged and supported. As an evaluator I would recommend the project team, including the Model Classroom Team, to act as coordinators and facilitators on the Web site “to shepherd the process of connecting, passing on the request to people in the network who are likely to have helpful information or insight… Once they have found each other, members talk by phone, in person, via email, or through the community bulletin board…” (p. 127).

Hopefully, we can help build a community of practice over the second year of the project.

 

Wenger, E., McDermott, R., & Snyder, W.M. (2002). Cultivating communities of practice. Boston: Harvard Business School Press.

SEAL takes over Newport Elementary School

Posted on by | Leave a reply

On Tuesday night I made the hour-long drive from Corvallis to Newport to cover an important event: Newport Intermediate School’s Family Literacy Night. In the past couple of months I’ve written about Project SEAL and the Family Literacy Nights schools are supposed to host as part of the project. I was thoroughly impressed with what I saw at Newport Intermediate School.

 

The principal (who also goes through Project SEAL training) brought in Roland Smith. Roland was an enthusiastic speaker, talking to both adults and kids and sharing his life experiences that led him to be an author. One of the best parts of his presentation was when he said he’d take 4 minutes to teach us how to write a book. His first step was to write about something that interests you. He travels a lot but when he gets home he still has research to do, so he asked, “where do you go to do more research?” The library of course! His next big step was to create a storyboard, attempting to map out as much of the book as possible. Then you have to write a rough draft. The final step he shared with us is relevant to all of us who write, including us graduate students. This was Roland’s secret of writing (shhh!): writing is revision. No matter how many times you make edits, the editors find something and your final product looks different from your first product. It’s not something to be scared of, it’s the process of writing.

 

After Roland’s talk, the parents and students were released to purchase books – many of them did – and to go to sessions taking place throughout the school, including poetry reading, student writer share-outs, and guest readers. I was casually observing each room and standing in the hall watching the flow of traffic when all of a sudden I noticed something… groups of students were walking around with iPad minis. Before I knew it, one 4th grader and her mom were standing in front of me and the girl asked, “Can I ask you a few questions?” When I responded “yes” she said, “you will be recorded, if that’s ok”. What a researcher! I gave consent and the student looked at the sheet in front of her (which had four questions that she came up with herself – I asked later on) and asked, “what do you know about the Japanese tsunami?” She asked me three other questions on the topic, specifically about marine debris and the dock that came on shore in Newport.

 

At the end of my interview, I saw one of the teachers whose student interviewed me and praised her on getting her students so involved and for using the iPads! I later found out that this is only a portion of the student project for Project SEAL.

What’s the Bleeding Edge for Museums?

Posted on by | Leave a reply

Last week, I talked about our eye-tracking in the science center at the Museums and the Web 2013 conference, as part of a track on Evaluating the Museum. This was the first time I’d attended this conference, and it turned out to be very different from others I’d attended. This, I think, meant that eye-tracking was a little ahead of where the audience of the conference was in some ways and behind in others!

Many of the attendees seemed to be from the art museum world, which has some different and some similar issues to those of science centers – we each have our generally separate professional organizations (American Association of Museums) and (Association of Science and Technology Centers). In fact, the opening plenary speaker, Larry Fitzgerald, made the point that museums should be thinking of ways that they can distinguish themselves from formal schools. He suggested that a lot of the ways museums are currently trying to get visitors to “think” look very much like they ways people think in schools, rather than the ways people think “all the time.” He mentioned “discovery centers” (which I took to mean interactive science centers), as places that are already trying to leverage the ways people naturally think (hmm, free-choice learning much?).

The twitter reaction and tone of other presentations made me think that this was actually a relatively revolutionary idea for a lot of folks there. My sense is that probably that stems from a different institutional culture that prevents much of that, except for places like Santa Cruz Museum of Art and History, where Nina Simon is re-vamping the place around participation of community members.

So, overall, eye-tracking and studying what our visitors do was also a fairly foreign concept; one tweet wondered whether a museum’s mission needed to be visitor-centric. Maybe museums that don’t have to rely on ticket sales can rest on that, but the conference was trying to push a bit that museums are changing, away from places where people come to find the answer, or the truth and instead to be places of participation. That means some museums may also be generally lagging the idea of getting funding to study visitors at all, let alone spending large amounts on “capital” equipment, and since eye-trackers are expensive technologies designed basically only for that purpose, it seemed just a little ahead of where some of the conference participants were. I’ll have to check back in a few years and see h0w things are changing. As we talked about in our lab meeting this morning, a lot of diversity work in STEM free-choice learning is happening not in academia, but in (science) museums. Maybe that will change in a few years, as well, as OSU continues to shape its Science and Mathematics Education faculty and graduate programs.