Seeking Comment on New IES Research Topic, Continuous Improvement Research in Education

The National Center for Education Research in the Institute of Education Sciences is planning a new research program for Fiscal Year 2014 and is requesting comments on the ideas outlined below. This new research program will supplement the existing main IES research program, Education Research Programs (84.305A), not replace it.

The new research program, Partnerships and Collaborations Focused on Problems of Practice or Policy, will incorporate two extant research programs, Evaluation of State and Local Programs and Policies (84.305E) and Researcher-Practitioner Partnerships in Education Research (84.305H) and it will add a third new topic, tentatively called Continuous Improvement Research in Education. Evaluation of State and Local Programs and Policies and Researcher-Practitioner Partnerships in Education Research will remain substantially unchanged. It is this third new topic on which we are seeking comment. Directions for submitting comments are provided in the final section of this prospectus.

The new topic is intended to address two specific issues. The first key issue the RFA will address is the development of continuous improvement strategies that will affect adaptation, design, development and implementation of education interventions and result in a self-sustaining learning process within an educational system. The development of a continuous improvement strategy may include the implementation or adaptation of evidence-based strategies or the development and testing of new ones. Continuous improvement strategies require effective means for evidence use and analysis—not only for traditional formative evaluation designs, but also for quick experimentation, data analysis and system response. This approach usually involves multiple tests of small changes ("empirical tinkering") that can cumulatively result in large change (Morris & Hiebert, 2011). The work funded under this grant should also help build capacity among users within the system in order to sustain the continuous improvement strategies and the innovations after the researchers have completed their involvement in the effort. Careful and reliable measurement of key process and outcome variables is integral to the continuous improvement strategy. This work resembles various aspects of what has been called design-based research, design-based implementation research, improvement research, and continuous improvement (see, for example, Berwick, 2008; Bryk, 2010; Bryk, Gomez & Grunow, 2010; Gawande, 2010; Mehta, Bryk and Gomez, 2012; Penuel, Fishman, Cheng & Sabelli, 2011; Peurach & Glazer, 2012).

Recognizing that school improvement is a complex process involving multiple actors, policies and contexts, the RFA also addresses the importance of education systems as the second key issue. Researchers often focus on components of systems rather than whole systems; this RFA is directed at researchers interested in examining how components of systems work together to generate desired outcomes. For the purposes of this RFA, systems should be thought of in at least two different ways. First, a system is a set of multiple actors and decision makers—teachers, administrators, parents and central office staff—who work together for a common goal. Systems also include the array of policies, practices and tools designed to help teachers and school leaders affect a particular outcome or set of outcomes. In this RFA, we focus on systems aimed at addressing a limited number of topics as described below.

This new research topic is different from typical IES Development and Innovation (Goal Two) studies in three ways. First, projects funded under this topic must target systems, as defined above. Second, they must emphasize the collection and use of real time evidence gathered throughout the course of the project to assess and inform the adaptation, development and implementation of innovations. Successful applicants will propose innovative and promising methods to continuously adjust the intervention system/ improvement strategies in response to evidence. Third, successful applicants will demonstrate evidence of a deliberate collaboration between researchers and practitioners. Projects under typical IES Development grants can, but are not required to, incorporate these three aspects in their work. In order to emphasize our interest in encouraging this type of work, IES is specifically requiring these aspects under the proposed new RFA.

The research plan should include a variety of methodological approaches that are well suited to the proposed research questions. The project design and methods must be based on a strong and explicit theory of action. The approaches used in this research may include traditional iterative development approaches, as well as quick cycles that embed multiple small experimental or quasi-experimental studies within the project. The framework itself may follow a "plan, do, study, act" cycle. The research plan should describe how data will be collected, analyzed and shared among the participants in order to promote continuous improvement.

It should also indicate how the project will identify "evidence of promise" through pilot studies to be conducted in the final years of the grant. The pilot studies should be designed to determine whether the continuous improvement strategy enhances the intervention system and shows promise for improving proximal and distal outcomes relevant to the theory of action supporting the project design. Pilot studies should use a design that will allow a comparison of the revised system's continuous improvement strategy with a previous or different approach to improving outcomes. The pilot studies may also address the question of which features or combinations of features of the intervention system are most critical.

By their nature, these types of research projects must be collaborative, focus on specific problems of practice, and aim to develop capacity among policy makers and/or practitioners to adapt, design, develop, implement, continuously improve, test and sustain systems of practices.

In addition to the key issues noted above, this new research program incorporates the following principles:

• The work should address needs identified by practitioners and/or policymakers
• The work should be collaborative and reflect established partnerships between the researchers and practitioners
• The focus of the research should be on the interaction of people, context, and policies with processes, practices, tools, and programs
• System interventions should be sensitive to prevailing conditions and variability across settings and people, including professional staff, parents and students, and the policy environment
• Optimal development and implementation of system interventions depends on collection and analysis of real-time, high quality data on key process and proximal outcomes

The new research topic will fund research projects to create or improve system-focused strategies designed to improve specific outcomes through a continuous improvement approach. The system-focused strategies should be built around a strong theory of action. They should be planned so as to build capacity among users for continuous improvement through ongoing corrections and modifications guided by evidence generated through analysis of real-time data. This research is specifically focused on "how to make systems work and make them work better."

Researchers may partner with state or local school superintendents, networks of schools, educational management organizations, charter management organizations, professional organizations, etc. The partnership may include tool or strategy developers, including commercial ones. The intervention and continuous improvement strategy being studied must extend beyond a single school. Successful research proposals must contain evidence that a true collaboration between researchers and practitioners is already in place. Evidence of the collaborative nature of the work will include letters of commitment from all major stakeholders. The plan should include broad-based, committed advisory groups. Staff and budget allocations should demonstrate shared responsibility for the work among the partnering organizations.

As a new research topic in FY 2014, this RFA will focus on continuous improvement strategies related to limited number of broad objectives. They are described with examples below.

Creating a safe, orderly and supportive learning climate for students from preschool through high school. In this case, applicants might develop or adapt a range of practices to be interwoven into general classroom instruction and student-teacher interactions. These practices might also be supported by school-wide policies aimed at establishing uniformly high expectations for student achievement and behavior for all students and engaging parents and community members in positive and helpful ways. The implementation plan would include ongoing monitoring and data collection to support evidence-guided improvements in the activities, tools and practices being enacted. Professional development needs and time and resource needs would be considered. Researchers may compare different intervention approaches and continuous improvement strategies across schools, conducting multiple rapid experiments or quasi-experimental studies. Process and outcome variables may include student tardiness and attendance; teacher, parent and student survey responses; disciplinary referrals; and other indicators of a climate conducive to learning, for example.

Improving students' transition to high school. Even though many "early warning signs" predict high school graduation, the ninth-grade experience is very important for students. Many strong students falter in their first year and otherwise weak students may thrive. Proposals may address the creation of a range of tools and practices to guide high school teachers, counselors and school leaders in developing the academic mindsets, learning strategies and other "non-cognitive" skills aimed at promoting school engagement among first-time ninth graders as they transition into high school (Allensworth & Easton, 2007, for example). The strategies for system improvement may incorporate practices and policies targeted at improving general classroom instruction for all students (including those at risk of dropping out and those with disabilities), school climate, and parent and community supports. This could include strategies to foster professional development of teachers and principals as a strategy for ultimately improving outcomes for students.

Increasing access to college and post-secondary training. The third objective involves developing systems to support students in their transition from high school to post-secondary education and training. Decisions made during and after high school about post-secondary education and training are very consequential. Thus, the information they have available to guide those decisions can be as important as the actual access to education and training options. The proposal might develop data feedback and monitoring systems, such as weekly reports for students and their teachers. The project might compare interventions and improvement systems across different settings on outcomes such as employment, income, and enrollment in post-secondary education settings.

The maximum award for this grant is $1,500,000 over four years.

References

Allensworth, Elaine M., & Easton, John Q. (2007). What Matters for Staying On-Track and Graduating in Chicago Public High Schools. Chicago: Consortium on Chicago School Research.

Berwick, Donald M. (2008). The science of improvement. The Journal of the American Medical Association, 299 (10), 1182–1184.

Bryk, Anthony S. (2009). Support a science of performance improvement. Phi Delta Kappan, 90 i(8),597–600.

Bryk, Anthony S., Gomez, Louis M., & Grunow, Alicia. (2011). Getting ideas into action: Building networked improvement communities in education. Carnegie Foundation for the Advancement of Teaching. www.carnegiefoundation.org/sites/default/files/bryk-gomez_building-nics-education.pdf

Gawande, Atul. (2010). The checklist manifesto: How to get things right. New York: Metropolitan Books.

Mehta, Jal, Gomez, Louis M., & Bryk, Anthony S. (2012). Building on practical knowledge: The key to a stronger profession is learning in the field. In Mehta, Jal, Schwartz, Robert B., & Hess, Frederick M. (Eds), The futures of school reform (pp. 35–64). Cambridge, MA: Harvard Education Press.

Morris, Anne K., & Hiebert, James. (2011). Creating shared instructional products: An alternative approach to improving teaching. Education Researcher, 40, pp. 5–14.

Penuel, William R., Fishman, Barry J., Cheng, Britte, & Sabelli, Nora. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40 (7), 331–337.

Peurach, Donald J., & Glazer, Joshua L. (2012). Reconsidering replication: New perspectives on large-scale school improvement. Journal of Educational Change, 13 (2), 155–190.

Directions for Submitting Comments

Please email your comments to ContinuousImprovementRFA@ed.gov.

We are seeking input on the overall clarity and promise of the proposed research topic; on the specific three broad objectives that the RFA will address; and on the development of "evidence of promise" through pilot testing for the continuous improvement innovation to be studied.

All comments are welcome.

Top