Having a good map, a coherent plan for instruction, can certainly help teachers find their way on the road to reform. But it is no guarantee. Actually implementing that curriculum in the classroom is where the rubber hits the road. And sadly, this is often where the change process breaks down.

For many teachers, what follows is an all too familiar scenario. The school or district launches a new program with a great deal of energy and fanfare. A small group of innovative teachers eagerly jumps on the bandwagon. Another group watches and waits to see how they fare. Still others, hoping that "this too shall pass," are determined to sit this one out.

A year or two or three later, the administration starts to wonder, "Whatever happened with that new curriculum? Did it improve student learning? What did we get for our time and money?" Those questions, however, are meaningless without first knowing if and how the change was ever implemented. For lasting change to take hold, reform leaders have to pay close attention to the process of change and to the people implementing it.

Researchers and practitioners have learned a lot from decades of making and studying change. Loucks-Horsley, Hewson, Love, and Stiles (1998) offer several general principles about change that emerge from the research.

Implementation Data:
Your Tool for Change

While general principles about change are useful, the real trick is applying them to a specific situation. To do that well, you, as a member of a school reform team, have to stay closely attuned to how teachers are responding. What are they actually doing with the new curriculum? How are they feeling about the change? What kind of support do they need most now? Once a change effort is launched, it is not too early to ask if your program is going down a road that is likely to lead to your desired outcomes. That’s better than waiting until the momentum has stalled because of problems that have not been addressed.

Implementation data can be a great aid to your team. It can provide vital planning information, such as which components of the
program teachers are using, or not using,
and why. It can help you diagnose and even prevent problems and target the best interventions. For example, if you find out early on that teachers are not comfortable with the
science content they are now expected to teach, you can offer appropriate professional development up front, cutting the problem off at the pass.

Tracking implementation also provides baseline data from which to measure progress. This is important not only for planning but also for morale and public relations. For instance, a school board is getting impatient with lack of improvement on mathematics test scores. With implementation data, you can show that while scores have not increased yet, progress is being made. For example, you may be able to report that this year 35 percent more teachers are using research-based
mathematics problem-solving strategies with their students.

Defining the Change: Will You Know it When You See It?

"Nothing blocks communication, inhibits evaluation, hampers staff development, and thwarts improvement more than a program that is not clearly defined."
(Crandall and Loucks, 1982, p. 1)

A first challenge in assessing classroom implementation of reform is defining what you are looking for. Even though you may have a beautifully written curriculum document, that
doesn’t mean everyone agrees on what the
curriculum looks like in action. For example, educators can have wide-ranging opinions about what is "inquiry-based instruction." For some, it is an occasional, teacher-directed experiment. For others, it is students asking their own questions and designing their own investigations on a weekly basis. Clearly, defining a desired practice is important both for building teachers’ understanding and commitment to the change and for assessing progress.

Horizon Research, evaluators for the National Science Foundation’s Local Systemic Change sites, uses video clips and discussions to help build consensus among its evaluators on what "good practice" in mathematics or science classrooms looks like. Tapes such as Teaching Math: A Video Library, K—4,
available from the Annenberg/CPB Math and Science Collection, can be a good catalyst for discussion. (See resources list, page 5.)

Another approach is to create (or borrow) your own detailed checklists or rubrics of the key features of your mathematics or science program. These can also stimulate discussion and build common understanding. When written in concrete, observable terms, checklists can also serve as yardsticks against which to measure actual classroom practice.

A third approach is to use the Practice Profile methodology, a tool developed by Susan Loucks-Horsley and David Crandall to help schools develop precise descriptions of a program or practice. Practice Profiles describe key components of a practice, possible variations ranging from ideal to unacceptable, and a list of implementation requirements. Practice Profiles are useful both for helping developers and implementers gain clarity about what a program actually looks like and for monitoring implementation.

Choosing Your Indicators

Once your team has reached a reasonable degree of clarity about the desired change, you will be ready to choose indicators for tracking progress. What indicators will you use? Recent national and state studies of mathematics and science education reform offer some ideas. Think of the possible indicators listed below as a menu from which your team can select a few items that will be most satisfying for your work. Trying to track too many indicators can be as paralyzing
as overeating!

Time

For elementary schools, simply looking at how much time teachers spend teaching science and mathematics relative to other subjects can be informative. For elementary schools where little science instruction is taking place, increasing time spent teaching science may be an important measure of progress. At the high school level, the number of years students study mathematics and science versus other subjects can be revealing.

What Happens During Lessons

Of course, it is also important to consider what actually happens during science and mathematics lessons. For example, the 1993 National Survey of Science and Mathematics Education, a study of 6,000 teachers in 1,250 schools nationwide, found that in science classes the largest proportion of time was devoted to lecture/discussion–38 percent of class time. Hands-on/laboratory work took 23 percent; individual seat work, 19 percent; non-laboratory group work, 10 percent;
and non-instructional activities, 10 percent (Weiss, 1997, p. 3).

Inquiry for Everyone?

Teacher Qualifications

Do all students have equal access to mathematics and science instructors with majors in the fields? Researchers have found that classes with high percentages of minority students tend to have less qualified teachers
(Weiss, 1997).

Instructional and Assessment Materials

Determining what materials teachers and students are using and how is another
possible measure of curriculum implementation. For example, some school districts have developed systems for tracking use of science kits or mathematics manipulatives. Portfolios, performance tasks, journals,
and other alternative assessments provide additional evidence of progress toward
standards-led curriculum reform.

Levels of Use

Reform teams may want to consider a widely used tool for assessing implementation of any particular innovation known as Levels of Use. Developed by the CBAM Project at the University of Texas in the 1970s, Levels of Use looks at what teachers are actually doing with a new program or practice such as a mathematics replacement unit or a kit-based science program. Typically, as teachers work with a new program over time, they progress in their skill and comfort. Through a series of interviews, evaluators can determine where teachers are in their use, ranging from early levels of preparation to more sophisticated integration and renewal.
(See Figure 1.)

This information can help planners target support appropriate to teachers’ needs. Training and materials are available for teams that want to learn more about this approach. (See resources list, page 5.)

Stages of Concern

Research on change tells us that it is important to pay attention to how teachers feel as they implement a new program or practice. Another tool developed by the CBAM Project, Stages of Concern, enables change facilitators to better understand and measure
teachers’ concerns ("perceptions, feelings, motivations, frustrations and satisfactions") about an innovation (Hall and Loucks, 1978, p. 38). CBAM researchers learned that
individuals go through predictable stages in their perceptions and feelings about change, starting with concerns about self, progressing to concerns about the task itself, and eventually moving on to concerns about impact. (See Figure 2.) They developed three tools for assessing concerns: a 35-item Stages of Concern questionnaire, an
Open-Ended Statement of Concerns, and an informal interview. By assessing concerns, reform teams can target interventions to help teachers resolve their most pressing concerns. Training in the use of these instruments is recommended. (See resources list, page 5.)

Views About Reform

In addition to concerns, your team may want to look at teachers’ and administrators’ views about mathematics and science education reform. Underlying and driving what people do is what they believe. Changing beliefs can be a good indication that reform is progressing. For example, teams may want to know if support for the use of hands-on activities is growing, if staff believe that computers or calculators are important for instruction, or if they think ability grouping is the best way for students to learn.

Practice Profiles

Practice Profiles, descriptions of key components of a particular practice and their variations (ranging from unacceptable to ideal), provide another set of indicators by which teams can monitor implementation of a new curriculum. Through interviews and observations, reform teams can assess how these components are being implemented. This information can then be used to help you determine what aspects of your program are not being implemented well and direct your support for teachers to those areas.

Collecting the Data

Once your team has identified what indicators you will use to track implementation of curriculum reform, you are ready to choose data collection methods. Methods include teacher and student surveys, classroom observations, interviews, classroom artifacts, materials use, and self-assessments. (Page 5 lists books and organizations that can help you select appropriate data collection methods and tools. Many tools are included in the guidebook "Using Data: Getting Results," forthcoming from the Regional Alliance.)

Remembering the Goal

While defining the change, choosing indicators, and collecting the data, it is important to keep focusing on the goal–improved
student learning. For student learning in mathematics and science to improve, what we teach, how we teach it, and how learning is assessed have got to change. Data about the implementation of reform as well as its impact can help reform teams make better decisions, keep reform on track, and deliver on student learning gains.

Crandall, D., & Loucks, S.F. (1982). The
practice profile: An all-purpose tool for program communication, staff development, evaluation, and improvement. Andover, MA: The NETWORK, Inc.

Hall, G. E., & Loucks, S.F. (1978). Teacher concerns as a basis for facilitating and
personalizing staff development. Teachers College Record 80(1), 36-53.

Hall, G.E., Loucks, S.F., Rutherford, W.L., & Newlove, B.N. (1975). Levels of use of the innovation: A framework for analyzing
innovation adoption. The Journal of Teacher Education 26(1), 52-56.