Application of Improvement Science to Scaling Up Research-Based Improvements in Science & Math Instruction
What problem does your project address? Why is this significant?
This project—titled Strengthening Yields in Mathematics through Metrics, Evaluation, and Teaching, Research to Improve Educational Systems (Symmetries)—will apply an improvement science process of networked improvement communities (NICs) to the problem of student performance in mathematics and, eventually, to other STEM subjects from elementary to college within a K-16 partnership. The target of our work is supporting successful mathematics transitions from elementary school through college (e.g., the transition from middle school to high school math). We argue that mathematics is the lynchpin for student academic success. Credible research has largely failed to have a scalable impact on classroom practices because it tends to emphasize adding knowledge to the field, not implementing practices at scale in highly variable conditions. NICs are composed of teams of educational professionals who blend knowledge from practice and research to improve long-term student achievement at scale. Iterative NIC processes and results are continuously assessed against quantitative and qualitative metrics. This approach has statewide implications for Oregon, especially the Portland metropolitan area, and can serve as a national model for scaling up educational improvement across diverse settings and environments.
Making transformative improvements in mathematics is one of the most significant problems that educators and researchers face today. One-third of students enter high school underprepared in mathematics. By graduation, 40 percent must take remedial mathematics courses in college before enrolling in credit-bearing courses. The sluggishness of this process begins early and grinds away at student motivation. The pressure of increasing student loan debt compounds this debilitating process. With only two-thirds of college students ever completing a degree, there is a stunning loss of human capital at college and in the economy. This loss manifests itself through (a) negative student attitudes and performance in school, particularly in middle and high school mathematics; (b) allocation of scarce resources toward noncredit developmental mathematics courses after high school; and (c) marginal competency in mathematics of college graduates. Implementation of the Next Generation Science Standards (NGSS) and the Common Core State Standards (CCSS) in Mathematics represents a major step in preparing students for employment in a competitive global economy. Still, the problems we face today will only worsen over time—even with new standards—unless there is a fundamental redesign of the system by which we educate students in mathematics.
Hamilton and Mackinnon (2013) estimate eighth grade proficiency in mathematics will drop from 66 percent to 33 percent when the new CCSS-based achievement tests are implemented. In Oregon, we anticipate that that decline will become public in summer 2016, which may further compound the deleterious impacts of retaining students. Yet, research on teaching and learning does exist, which could lead students toward CCSS’s world-class standards. Symmetries will leverage the NIC framework and processes to improve mathematics teaching and learning—the key to postsecondary educational attainment. This project addresses the key provisions of Teaching and Human Capital Management (1) by examining the problems of student success in mathematics at the system level and (2) by creating a NIC composed of teachers, administrators, counselors, and university faculty who collectively employ improvement science and iterative applications of research in highly variable environments designed to drive improved mathematics outcomes for students.
What do you intend to demonstrate or prove? What means will you use?
As a member of the Mid-Valley Mid-Coast Partnership (Partnership), Oregon State University’s College of Education is working with the Carnegie Foundation for the Advancement of Teaching in the Design Learning Lab Project to implement a NIC to guide improvements to the system by which students experience mathematics in and out of school. Our theory of change is that the NIC will lead to increases in high school graduation rates and successful transitions to college (1) by increasing student performance in mathematics in middle and high school and (2) by significantly reducing college enrollment in noncredit developmental mathematics.
Why should use of a NIC process claim to improve mathematics performance at scale where years of research on mathematics teaching and learning have not succeeded? The best classroom-based research is driven by theory and rigorous research design to show effects in well-defined settings. As discussed in a recent NSF report, researchers aim for reproducibility, replicability, and generalizability. These criteria produce results that expand knowledge and improve theory but fail to provide guidance on how to apply new knowledge and improvements at scale and in multiple settings. Research results only become scalable in the hands of knowledgeable, skilled practitioners (i.e., teachers, counselors, administrators) who understand the specific systems in which they seek to make changes. The NIC process is designed to bring all the players in a traditional research project together under new principles that address issues intentionally avoided in traditional research. Our NIC will engage participants in (a) understanding the educational systems that impact how students learn mathematics and (b) analyzing and adapting practices and innovations in different educational settings.
The NIC is composed of two organizational units—(1) the nodes and (2) the NIC hub. The nodes are operational teams of three to four individuals that may include teachers, administrators, community college instructors, and university research faculty. Nodes test research adaptations in order to fail fast, learn fast, and—ultimately—improve fast so as to discover what works in improving student achievement.  Each node reports its empirical results to the hub, which analyzes, adapts, and disseminates research-based interventions to other settings for further testing. (The hub works with the nodes to structure and support the distribution of successfully tested interventions.) Through its work with the nodes, the hub develops and maintains “the evolving framework that guides efforts among many different participants. [The hub] establishes the processes and norms governing how individuals and groups work together and the evidentiary standards for warranting claims.” Through the hub’s work, a NIC aggregates the collective experience from the nodes by testing research adaptations in different environments. The hub also keeps the entire NIC accountable to key metrics to which the community of partners have agreed. Symmetries will eventually guide mathematics improvements at scale for 45,000 K-12 students in five school districts, nearly half of whom are on free or reduced lunch across urban and rural settings.
A concrete example will further illuminate the dynamic structure of NIC-guided interventions and how they can increase student mathematical discourse to meet CCSS. Teachers in different nodes might, for example, employ the research-based practice of extended “wait-time” between a teacher prompt and the first student’s response. Representatives of the nodes would meet with the hub to discuss length and quality of student response, total number of different students responding during a class, and the number of responses students make to other students’ comments while using this intervention. Based on these data, successful implementations of wait-time are compared to less successful ones. The more successful approaches are then retried in more classrooms. Teachers in nodes work with the hub to extend the study of student mathematical discourse to new areas such as how to employ student ideas to leverage increased understanding and problem-solving. The NIC would also test fast other learning and teaching strategies—such as (1) having students take mathematics their senior year and (2) using real-time assessment where students provide feedback on teaching strategies—to improve students’ understanding and motivation in mathematics.
As the wait-time example illustrates, the NIC’s focus is to gather data on teacher implementation of research-based strategies. The active hub/node participants recruit other colleagues to deploy research-based interventions and to provide their own data. In general, one or two teachers may initiate an innovation while the hub facilitates dissemination of successful practices to many teachers and collects additional data across many different classrooms. Summer institutes will provide further opportunities to disseminate results and practices more broadly across the partnership. At each stage of dissemination, teachers will be expected to collect and submit data on effectiveness. The intention is to promote a mindset of getting better through the application of research-based innovations and data collection. Unlike other research projects, Symmetries is driven by the users of the research in collaboration with researchers—not the other way around. The project uses a scientific approach that blends research and practice to inform teacher, administrator, counselor, and faculty practices to solve fundamental problems in student academic performance. These procedures and structures build new capacity in the system to make empirically based improvement part of the entire educational system.
What outcomes do you expect for the project, both immediate and long term?
Symmetries’ expected outcomes are bulleted below.
- Immediate. Development of a permanent fully staffed hub, data analytics, representatives of nodes, and web and communication support to drive improvement in mathematics. Funds also will support increasing capacity for building nodes, gathering and analyzing data, dissemination, and scaling up.
- Long-term. In three years, an increase in the number of partnership students qualifying for college-level math when they enter college. We also expect to see an improvement in state mathematics test scores for students who have teachers involved in the project.
What is the estimated overall cost of the project over 3 years?
The estimated cost for the three-year project is approximately $1.45 million. A separately uploaded budget justification provides a line-itemization for these costs.
What other sources of support are you pursuing for this project?
Partners have identified local sources, state and federal grants, and private foundation support for potential funding. At present, partners are collectively providing in-kind and funds of $81,332 to fund a half-time director, part-time staff member, data management system, and the NIC Hub for 2015-16. State sources include Oregon’s Regional Achievement Collaborative funding, which could provide up to $50,000 per year for two years. The partnership is also investigating three National Science Foundation programs (Discovery Research PreK-12, the Advanced Technological Education, and ITEST) and two U.S. Department of Education programs (Supporting Effective Educator Development and the Teacher Quality Partnership). The partnership is also exploring opportunities with the Teagle, William T. Grant, Wallace, and Hearst foundations, as well as the Helmsley Charitable Trust.
List Personnel who will be engaged in the project along with their qualifications.
NIC Implementation Team members include the following:
- Christy Stevens, director: Linn-Benton Community College faculty, middle and high school teacher in rural and urban school districts, coordinated the PACERS Rural Science Program University of Alabama, director Teacher Support Services for Foxfire Fund
- Larry Flick, university leader: Dean of Education, professor science education, PI NSF grants, middle school science teacher, electrical engineer
- Rynda Gregory, K12 schools leader: Student Services Coordinator Corvallis School District, classroom teacher and elementary school principal
- Justin Smith, data analytics: Director of Institutional Research Linn-Benton Community College, data analytics, mandatory reporting, accreditation, and student surveys.
Mid-Valley Mid-Coast Partnership Members: Ed Ray, President, Oregon State University; Jim Golden, Superintendent, Greater Albany Schools; Larry Flick, Dean of Education, Oregon State University; Melissa Goff, Superintendent, Philomath Schools; Greg Hamann, President, Linn Benton Community College; Rob Hess, Superintendent, Lebanon Community Schools; Erin Prince, Superintendent, Corvallis Schools; Steve Boynton, Superintendent, Lincoln County Schools; Mary McKay, Superintendent, Linn Benton Lincoln ESD; and Christy Stevens, Proj
 Franke, M. L., Kazemi, E., & Battey, D. (2007). Mathematics teaching and classroom practice. In F. K, Lester (Ed.) (2007). Second handbook of research on mathematics teaching and learning, Volume 1, Charlotte, NC: IAP; and Zwiers, J, O’Hara, S., & Pritchard, R. (December 2013). Eight essential shifts for teaching common core standards to academic English learners, ALD Network, ALDNetwork.org.) A NIC is a scientific community that uses credible research to guide improvement in student performance.
 Hanushek, E. A., Peterson, P. E., & Woessmann, L. (May 2014). Not just the problems of other people’s children: U.S. student performance in global perspective, PEPG Report No.: 14-01, Taubman Center for State and Local Government, Harvard Kennedy School.
 Bryk, A. S., Gomez, L. M. Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America’s schools can get better at getting better. Cambridge, MA: Harvard University Press. Guided by a deep understanding of a particular problem and the underlying system that created the problem, a NIC focuses on a well-defined goal under a shared theory of change to develop, refine, and disseminate effective innovations across varied educational contexts. The processes developed within a NIC are disciplined and empirical.
 Sowers, N. & Yamada, H. (November 2014). Pathways impact report, Carnegie Foundation for the Advancement of Teaching, available athttp://cdn.carnegiefoundation.org/wp-content/uploads/2015/01/pathways_impact_report_2015.pdf
 Hamilton, L. & Mackinnon, A. (2013). Opportunity by design: New high school models for student success, New York, NY: Carnegie Corporation of New York.
 Jimerson , S. (2001). Meta-analysis of grade retention research: Implications for practice in the 21st century. School Psychology Review, 30, 420-437. Jimerson shows that students who have been retained in grade for just one year have a 50 percent greater probability of dropping out of high school.
 Schmidt, W. H. & Houang, R. T. (2012). Curricular coherence and the common core state standards for mathematics. Educational Researcher 41(8), 294-308.
 Bollen, K. et al. (May 2015). Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science, Report of the Subcommittee on Replicability in Science, Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences.
 Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked improvement communities in education. In Maureen Hallinan (Ed.) Frontiers in Sociology of Education, New York, NY: Springer Publishing.
 Under extended wait time, a teacher typically allows three seconds to pass before calling upon the first student for a response. Research indicates that this amount of wait-time opens opportunities for students to formulate more complex mathematical responses to the question.