Professional Development Design: Alliance evaluates workshops for Using Data/Getting Resutls

By Molly Singsen Richter

To measure the effectiveness of the workshops and to help in the planning of future training sessions, the Alliance conducted a follow-up study. While the study was concerned with workshop design and guidebook implementation, the recommendations may prove useful to professional development providers developing similar programs. (See sidebar.)

Workshop Goals

Alliance staff hoped participants would leave the workshop prepared to create the conditions for safe and effective use of data; make work with data a positive, collaborative learning experience; engage in data-driven dialogue; and use state assessments and other data effectively as part of a process of continuous improvement. Approximately 200 educators attended the five two-day workshops. Participants included school and district groups of teachers, administrators, and curriculum coordinators, as well as representatives from universities, colleges and state departments of education, and other technical assistance and professional development providers.

The Study

Evaluators wanted to determine the extent to which participants felt prepared, after they left the workshop, to implement the processes of collaborative inquiry outlined in the guidebook and, more importantly, the extent to which they did implement the processes. The Alliance sent participants a written survey and conducted three in-depth telephone interviews. The data revealed that while some educators returned to the schools and began to apply what they had learned, many felt they had not been able to do as much as they would have liked. The important question to ask, then, is why not?

Time back at work. Not surprisingly, the most commonly cited obstacle to implementation was time. This was expressed in the familiar sense that there is never enough time to do everything and that other school priorities have a tendency to take precedence. Respondents also related concerns about the amount of time needed to engage in the collaborative inquiry approach prescribed in Using Data–Getting Results.

Administrative leadership and support. Lack of administrative support or leadership was another reason participants gave for their inability to implement the strategies they learned. One survey respondent reported, "We [our team] have reviewed the materials and were very impressed with the quality. We've talked with administrators on two levels about the importance of using data. However, we feel that the whole school has to be involved and that administrators should...be taking a leading role. To date they have not." Lack of support from school or district leadership, including not providing adequate time, can impede implementation in multiple ways.

Authentic purpose and opportunity. Participants who reported the most progress in implementing the collaborative inquiry process shared two key factors in addition to time and administrative support. These participants attended the workshop with a specific and authentic purpose for acquiring new skills in using data, and their job situation provided opportunity to apply what they learned. For some, their authentic purpose was externally determined by a district or state mandate to use data in planning. Whether externally or internally derived, they were seeking a process with which to investigate and problem-solve.

Perhaps as important, these participants held positions in which working with school-wide or system-wide data is a natural part of their job, not an added responsibility. For example, these were school and district curriculum coordinators in charge of leading planning committees, not teachers participating on committees in addition to their teaching responsibilities. Those preparing to introduce the collaborative inquiry approach to other teachers worried that without their own internal motivation and job scope, the other teachers might not be prepared or able to devote the necessary time outside of their classrooms to understand and complete the process. Attending the workshop with members of their planning committee also seemed to increase the likelihood that participants would have opportunity to apply what they learned.

Preparation. Another factor that emerged as possibly related to implementation is how much the participant knew about the workshop content before attending. This is linked to a sixth factor, the degree of initiative taken to attend the workshop. Those who reported implementing the most after the workshop found out about the workshop on their own, knew specifically what it was about, and requested permission to attend. Many who did not implement had been recruited or assigned to attend, and weren't necessarily sure what exactly they were attending until they arrived. Certainly, researching workshop options and taking initiative to attend are more likely when there is a specific purpose for attending.

Each of the influencing factors–time; administrative leadership and support; specific and authentic purpose; opportunity to apply new learnings; prior knowledge of the workshop; and initiative in attending–relates to clarity of purpose (for the individual attending and for their administrative leadership) and participant readiness and capacity to implement the collaborative inquiry approach. These factors are outside the control of workshop presenters. However, participants named two other factors, which workshop presenters can influence.

Workshop length. Some participants who had not implemented anything from the Using Data–Getting Results workshop explained that there was not enough time at the workshop for them to process the amount of material presented, and therefore they did not feel adequately prepared to implement or lead others in the collaborative inquiry approach or the strategies presented.

Complexity of the material. Some participants reported that the guidebook was too complex, daunting or involved for them to deal with in the time allotted, or to tackle at all. This could indicate that the guidebook was not suited to their purposes, that the workshop simply did not provide enough time for them to process the content, that the content needs to presented in a different way, or that the content presented needs to be scaled back.

To learn how one school district is using the guidebook see Collaborative Inquiry Uses Data to Get Results.

Recommendations

Many of the stumbling blocks to implementation that survey and interview respondents cited are not new to Alliance staff, but they serve as reminders of the challenges professional development providers must address in their program designs. The following suggestions may be useful for anyone planning a similar professional development effort.

Ensuring Clarity of Purpose: Determining Participant Readiness & Capacity

The following steps might improve the appropriateness of audiences selected to attend Using Data or similar workshops:

–For workshops organized through a third party, as the pilot workshops were, provide descriptive materials to the third party or handle registration directly in order to assure quality and accuracy of description disseminated.

–Target registration to teams composed of members in roles that deal directly with data (i.e. not solely classroom teachers).

–Limit registration to pre-existing teams, or teams with a defined application in mind; require a statement of intended purpose from the entire team as part of registration.

–If these precautionary steps are not possible, begin the workshop with an assessment of participant knowledge and purpose for attending, and clarify as necessary.

Ensuring Adequate Understanding

Judging from the survey responses and interviewee reflections on content, it seems that the two-day introduction to the Using Data –Getting Results guidebook and its processes may not provide enough time for participants to gain the understanding they need to apply what they have learned. In addition to ensuring that the audience is appropriately positioned, motivated and supported, it may be necessary to lengthen the workshop, reduce the amount of material or concepts covered, and/or add follow-up mechanisms to facilitate further learning.