Main Content: Evaluation
The State Professional Development Grantees must describe how the SEA will assess, on a regular basis, the extent to which the strategies implemented have been effective in meeting the prescribed performance goals. Below are journal articles, resources and tools developed by SPDG evaluators to assist projects in evaluating initiatives.
Assessments / Instruments / Checklists
Behavior Interventions and School Climate
Data Collection, Display, and Use
Comparison Group Evaluation Design
Goal Attainment Scale
Fidelity of Implementation
Implementation
Logic Models
Measuring Collaboration
Reports
Response to Intervention and Multi-Tiered Models of Intervention
SPDG (SIG) Program Evaluation
Assessements / Instruments / Checklists
inQsit Online Assessment System (Fortiede, D. and Draper, V., Ball State University) inQsit is a comprehensive online assessment instrument.. You can create, distribute, and administer tests, quizzes and surveys via the intranet or internet, then retrieve, record, optionally grade and assess responses; all online, without the need for any HTML or programming knowledge.
Keywords: online assessment instrument, online surveys, online quizzes
The Effect of an "Unsure" Option on Early Childhood Professionals; Pre- and Post-Training Knowledge Assessments (American Journal of Evaluation, published online, September 3, 2010, Wakabayashi, T. and Guskin, K.)
Keywords: knowledge assessment, measurement, methods, early childhood professionals, true/false test
PaTTAN Leadership Mentoring to Mastery (M2M): Self Assessment Tool (DOC, January 2009)Developed by Pennsylvania Department of Education for those Based on Council for Exceptional Children, Council for Administrators of Special Education (CEC-CASE), Advanced Knowledge and Skill Set for Special Education Administrators, and Aligned with Pennsylvania Inspired Leadership (PIL) Standards.
Keywords: knowledge assessment, leadership, administrators, mentoring
CYFERnet Children, Youth, Families Education, Research Network. CYFERnet's evaluation section includes practical tools that you can use to evaluate community-based programs; information on how community programs can be sustained; and assessments of organizational support for work in the areas of children, youth and families. This site is hosted by North Carolina State University.
Keywords: community-based programs, children and families, organizational assessment
Rhode Island Modified Universal Design for Learning Educator Checklist (DOCX, September 24, 2009, Adapted from CAST UDL Guidelines: http://udlguidelines.wordpress.com/introduction/)
Keywords: UDL, Universal Design for Learning, Educator Checklist
Advancing Parent-Professional Leadership in Education (A.P.P.L.E.) Example Post-Course Survey (PDF, Spring 2009) Includes: course evaluation items, post-test items on course content and general participant feedback items.
Keywords: parent survey, parent leadership
Behavior Interventions and School Climate
Vermont PBiS Website - with evaluation resources
VT PBiS Team Implementation Checklists The Team Implementation Checklists (monthly and quarterly) are tools designed for school teams to monitor their progress through PBIS implementation. For the first few months following PBIS implementation, we recommend that school teams use this to guide your monthly PBIS Leadership Team meetings.
VT PBiS School-Based Universal Coordinator Self-Assessment (May 2011) This self-assessment is designed to assist coordinators in identifying current strengths and professional development goals.
Keywords: checklist, self-assessment, behavior, PBIS, team implementation,
Developing Training Capacity for Statewide Implementation of Michigan's Integrated Behavior; Learning Support Initiative (MiBLSi): A Response to Intervention Model for Reading; Behavior Support (DOC, 2010) The purpose of this document is to develop statewide training capacity for the implementation and sustainability of school-wide models of reading and behavior support (RtI), through the application of the Implementation Drivers to create an effective Trainer Capacity Development Model. In the Appendix are evaluation resources, which include: Participant Training Evaluation, Rubric Feedback Forms, Trainer Feedback Form, Pre-Post Survey of Readiness for Training, Trainer Work Day Evaluation and TAP Trainer Support Evaluation.
Keywords: Rubric, Trainer, Survey, Coach
Building Effective Schools Together through Positive Behavior Supports Implementation Rubric (XLS, July 2010, CalSTAT) This interactive template is used by school teams to record their implementation level for 10 key elements. After entering their data teams can generate reports.
Keywords: rubric, practice profile, implementation rubric
Comparison Group Evaluation Design
Data Collection, Display and Use
DATA COLLECTION
New Mexico State Personnel Development Grant State Performance Plan Improvement Project Data Management System (PDF, 2009) This resource highlights New Mexico SPDG's State Performance Plan Improvement Project (SPPIP) Online Consultant Log for collecting training and coaching data. The New Mexico SPDG project uses a web-site to disseminate information about project activities, register participants for centralized events, manage school improvement plans (Educational Plans for Student Success), administer surveys for feedback and evaluation, and collect information on training, coaching and other technical assistance provided to schools by the cadre of consultants.
Keywords: online training log, online coaching, online database, State Performance Plan, data collection
New Mexico's iPad Training and Coaching Data Collection Tool: Measuring implementation; fidelity of implementation; challenges; solutions for data collection; data management using web-based tools including a new iPad tool. To view the app go to the Apple App Store under Training and Coaching Log. Anyone can download the free app but you have to have an account on the system to sync data. Otherwise, the data just stays on your iPad. For more information contact Carlos Romero - romero@apexeducation.org
Keywords: iPad app, mobile data collection, online training and coaching data collection
NCSIP II Fidelity Observation Forms and Professional Development Data Collections Forms and Procedures North Carolina's State Personnel Development Grant - NC SIP II - Evaluation Tools. The site includes: student progress data evaluation collection instructions and forms, online surveys, professional development data collection forms and procedures, and reading and math model fidelity observations forms.
Keywords: data collection, fidelity, reading, math, student progress data
DATA DISPLAY
Creating Customized Evaluation Reports for Stakeholders (Session Recording WMV, June 2009. Walter, C. and Wood, A., California SPDG Evaluators) CalSTAT's process for summarizing; displaying evaluation data for multiple; shared tools they developed and the software used to create their reports.
Keywords: stakeholder reports, data display, customized reports
DATA USE
Washington State: Using Data to Drive Statewide Improvement Efforts (PPT, November 2011, Pyper, L., Washington SPDG Director and Roberts, G., Ph.D., Center on Instruction, Special Education Strand)
OSPI Efforts Inventory Instructions and Reference Sheet (DOC, October 2010) The purpose of this inventory is to identify those efforts within OSPI [Washington] that are supportive of RtI components. This document was developed in conjunction with the Northwest Regional Comprehensive Center.
Keywords: Data Analysis, Data Use, Statewide Improvement, Systems Improvement
Using and Implementing Goal Attainment Scales as a Way to Measure Progress (PPT, June 2010. Gaumer Erickson, A., Ph.D., KS and MO SPDG Evaluator, University of Kansas, Center for Research on Learning; Ballay, M., Ed.D, LA Evaluation and Site Liaison, Louisiana State University) Session Recording: WAV
Goal Attainment Scale Templates and Data Displays
Goal Attainment Scale (GAS) Template (DOC, Ballay, M., 2010)
Modified Goal Attainment Scale Template (DOC, Ballay, M., 2010)
Goal Attainment Scale Form (DOC, Gaumer Erickson, A., 2010)
GAS Sample Data Displays (PDF, Gaumer Erickson, A., 2010)
The Evaluation Forum Blog - 'Fiddling with Fidelity?". The author provides a unique description of fidelity, using a bread pudding metaphor. It goes on to describe the key questions evaluators should ask for process evaluation purposes.
National Center on Response to Intervention Resources on Fidelity to RTI
NCSIP II Fidelity Observation Forms and Professional Development Data Collections Forms and Procedures North Carolina's State Personnel Development Grant - NC SIP II - Evaluation Tools. The site includes: student progress data evaluation collection instructions and forms, online surveys, professional development data collection forms and procedures, and reading and math model fidelity observations forms.
Advancing Parent-Professional Leadership in Education (A.P.P.L.E.) Example Post-Course Survey (PDF, Spring 2009) Includes: course evaluation items, post-test items on course content and general participant feedback items.
Creating Customized Evaluation Reports for Stakeholders (Session Recording WMV, June 2009, Walter, C. and Wood, A., California SPDG Evaluators) CalSTAT's process for summarizing and displaying evaluation data for multiple shared tools they developed and the software used to create their reports.
PRESENTATIONS
SPDG Evaluator's Webinar - Problem Solving and Response to Intervention SAPSI Training PPT, Session Recording (January 2011, Castillo, J., Ph.D. and Dorman, C., Ed.S., University of South Florida, Tampa) Drs. Castillo and Dorman describe in-depth how they are using the Self-Assessment of Problem Solving Implementation (SAPSI) too formatively evaluate their project activities. In addition, they share other evaluation instruments that would be of interest to the community. Resourses shared during the session:
Florida Statewide Problem Solving/Response to Intervention Evaluation Tool Technical Assistance Manual (PDF)
Example - Self-Assessment of Problem Solving Implementation (SAPSI) Administration Summary, 2010-2011 School Year (DOC)
Florida's PS/RtI Project Program Evaluation Website
Florida's PS/RtI Project Year One Evaluation Report
Florida's PS/RtI Project Year Two Evaluation Report
Keywords: problem solving, fidelity of implementation, self-assessment
Overview of Stage-Based Measures of Implementation Components (PDF, SISEP, 11/2011). This table outlines the stage-based assessments of implementation activities in practice, to be used (modeled) by Regional Implementation Team members during District or project work with District Implementation Teams or Project Teams in the State Department of Education. Subsequently useful to District Leadership and Implementation Team members as they work with groups of schools or building leadership and implementation teams.
ImpleMapping. Exploration Stage Assessment of Implementation Capacity (PDF, SISEP, 11/2010). This is part of SISEP's series of Stage-Based Measures of Implementation Components.
Installation Stage Assessment (PDF, SISEP, 2/2011). This is part of SISEP's series of Stage-Based Measures of Implementation Components.
Installation Stage Action Planning Guide for Implementation (PDF, SISEP, 2/2011). This is part of SISEP's series of Stage-Based Measures of Implementation Components.
Full Implementation Stage Assessment (PDF, SISEP, 2/2011). This is part of SISEP's series of Stage-Based Measures of Implementation Components.
Stages of Implementation Analysis (PDF, SISEP, Adapted 1/2012). This tool provides the team with the opportunity to plan for and/or assess the use of stage-based activities to improve the success of implementation efforts for EBPs or evidence-informed innovations. The tool can be used to assess current stage activities (e.g. “We are in the midst of Exploration”) or past efforts related to a stage (e.g. “We just completed most of Installation? How did we do? What did we miss?).
Michigan's Integrated Behavior and Learning Support (MiBLSi) Initiative - Evaluation Resources At this link you will find resources on MiBLSi's Model imlementation evaluation, measurement manual, measurements, measurement schedule and their support training sequence.
Keywords: implementation, measurement, training, multi-tiered model of intervention
ERIA: Effective Reading Interventions Academy, A Pathway Towards RTI2 (PDF Program Guide, December 10, 2010) This 16-page booklet explicitly addresses both implementation and intervention practices to guide the design of a site-based program.
Keywords: practice profile, RtI, Response to Intervention
ERIA Implementation Rubric (XLS, July 2010, CalSTAT) This 10-item instrument provides a framework for trainers, coaches, site team members and teachers to evaluate and discuss implementation, fidelity and next steps.
Keywords: implementation rubric, RtI, Response to Intervention
Building Effective Schools Together through Positive Behavior Supports Implementation Rubric (XLS, July 2010, CalSTAT) This interactive Excel template is used by school teams to record their implementation level for 10 key elements. After entering their data teams can generate reports.
Keywords: implementation rubric, behavior interventions
Missouri/KU Regional Coach Level of Implementation and Collaboration Interview Questions (DOC, September 2010)
School Staff Survey (PDF, March 2011) The Missouri Integrated Model (MIM) School Staff Survey is designed to gain input from all school staff (teachers, administrators and instructional staff) regarding the level of implementation of the essential features of the MIM. This survey was developed with input from the MIM Evaluation Work Group and asks participants to rate their own behavior in implementing the 11 essential features (i.e., Shared Vision and Commitment, Leadership, Collaborative Environment, Ongoing Professional Development, Mentoring and Coaching, Culturally Responsive Practices, Resource Mapping, Family and Community Involvement, Evidence-based Practices, Data-based Decision Making, Monitoring of Student Progress). To view Dr. Gaumer-Erickeson's presentation about this survey go to: http://signetwork.org/wikis/5/page/Evaluators%27_Webinar_March,_2011:_Measuring_Implementation:%0B%0B%0BSchool-Wide_Instructional_Staff_Perspective_.
Keywords: school staff survey
PRESENTATIONS
Evaluating the Implementation Drivers (PPT, SPDG Evaluator's Webinar, May 2010, Gaumer Erickson, A., Ph.D., KS and MO; Morrison, J., Ph.D., OH; Mueller, P., Ed.D., NH and MS)
2011 SPDG Directors' Professional Development Webinar #1: Models of and Evalution of Professional Development To view this recording: https://tadnet.ilinc.com/join/ppjxfkm
Professional Development Series #1 - Evaluation. (PPT, January 2011, Morrison, J., Ph.D., Ohio SPDG Evaluator) Julie Morrison explored how NIRN's implementation drivers framework has prioritized staff competence for effective programs; practices; examined how Guskey's five critical levels for evaluating professional development can be used as a framework in designing effective training.
2011 Professional Development Webinar: Models of and Evaluation of Professional DevelopmentTools as Implementation Drivers: An Example from California SPDG's ERIA - Effective Reading Interventions Academy (PPT, January 2011, Walter, C., Ph.D. and Wood, A., California SPDG Evaluators, CalSTAT)
Logic Models (2010 Screencast Recording) The Center for Evaluation and Education Policy (CEEP) created this Voice-Over PowerPoint for OSEP on: (1) How to create and use logic models and (2) How to create high quality objectives and performance measures.
The following Web site provides more information on logic models and lists multiple online resources: http://www.cdc.gov/eval/resources.htm.
Measuring Collaboration Among Grant Partners (American Journal of Evaluation 2006; No. 27; pg. 383)
Keywords: collaboration; grant partners; cooperation; visual displays.
Transition Collaboration Scale and Rubric Process (PDF, January 2011, Noonan, P., Ph.D., University of Kansas, Center for Research on Learning)
Transition Collaboration Survey (PDF, January 2011, Noonan, P., Ph.D., University of Kansas, Center for Research on Learning) The Transition Collaboration Survey is designed to gain input from individuals regarding their level of collaboration in providing transition education and services.
Creating Customized Evaluation Reports for Stakeholders (PPT, June 2009, CalSTAT, Walter, C.L., Ph.D. and Wood, A., Ed.D., California SPDG Evaluators) Calstat's process for summarizing and displaying evaluation data for multiple and shared tools they developed and the software used to create their reports. Session Recording (WAV)
CalSTAT's Annotated Materials List (PDF) Includes links to SIG 2/CalSTAT Activities and Outcomes Evaluation Summary Report, Professional Development TA and Collaborative Sites Survey Summary Report, ERIA Summary Report 2007-08 and ERIA Team Implementation Checklist (TIC) Charting Program. To see additional reports developed by CalSTAT go to: http://www.calstat.org/evaluation.html
Wisconsin: http://wispdg.org/wispdg_evaluation.html
This link provides access to Wisconsin SPDG's current efforts related to Fidelity of Implementation and Sustainability Evaluation Architecture and also provides examples of evaluation reports for fidelity of implementation and sustainability.
Response to Intervention and Multi-Tiered Models of Intervention
National Center on Response to Intervention Website (NCRTI) RTI Screening, Progress Monitoring and Other Tools. Use their Advanced Search function for best results.
Keywords: RtI, response to intervention, progress monitoring, screening
Response to Intervention Progress Monitoring Resources for Grades K-12 (PDF, March 2009) Southeast Comprehensive Center
Keywords: progress monitoring
I-ASPIRE Evaluation Tools At this site you will find the following tools: Data Protocol Form, Self-Assessment of Problem Solving Implementation, Technical Assistance Log Forms, Fidelity Checklist, Parent Survey and Institutes of Higher Education Checklist. The evaulation tools were developed by Loyola University, Center for School Evaluation, Intervention and Training. for the Illinois State Personnel Development Grant, I-ASPIRE.
Illinois ASPIRE Project Evaluation Tools Overview (PPT, October 5, 6, 2009, Cox, K., I-ASPIRE Director) An overview of two evaluation tools and data collected on levels of implementation for 2007-2008. The primary focus of the I-ASPIRE project is reading at the K-3 level.
Illinois ASPIRE Project Problem Solving Fidelity Checklist (PDF, 2009) Allows external evaluation by I-ASPIRE regional staff of implementation in school sites. The primary focus of the I-ASPIRE project is reading at the K-3 level.
Illinois Self-Assessment of Problem Solving Implementation (SAPSI) (DOC, V2.4) This self-assessment is used to assess the degree of implementation of the problem solving process at the building level as self-reported by the schools. Increases in maintaining implementation levels over time can also provide a measure of fidelity and degree to which practices are being sustained.
Keywords: fidelity of implementation, checklist, fidelity, self-assessment, problem-solving
Methods & Instrumentation Implementing Multi-Tiered Literacy & Behavior Models (PPT, Mueller, P.H., Ed.D., Evergreen Evaluation Consulting) This presentation highlights the methods & instrumentation employed by New Hampshire's SPDG Leadership Team and evaluators to capture the extent to which the intervention is being implemented with a high degree of fidelity. The primary outcome of the NH RESPONDS initiative is to develop and implement a multi-tiered, braided literacy and behavior support program which result in improved student outcomes in five demonstration school districts across New Hampshire. The intent is for the demonstration schools to have a highly developed integrated 3-tier system of literacy and behavior support at the end of the grant period that is aligned to school improvement efforts and supported by district leadership activities.
New Hampshire Literacy Universal Team Checklist (DOC, 2002, Adapted from: Sugai, Horner, Lewis-Palmer with content from NH Literacy Action Plan) The Universal Team completes this checklist at two checkpoints during the school year to monitor activities for implementation of RTI for Literacy Instruction: Early Fall and Early Spring.
Keywords: checklist, fidelity of implementation, TIC
Montana's RtI Implementation Survey [PDF, February 2011] Montana's RtI Implementation Survey has been used in identifying where schools and districts are in the process of scaling up RtI in their local areas. Target Audience: Participants who have identified RtI or PBIS as one of their SPDG Goals.
Keywords: RTI, response to intervention, survey, implmentation
Montana District Data Audit Tool (PDF, September 2010) As Montana starts the braiding of RtI and PBIS, this Data Audit Tool has been designed to assist local sites in reviewing and discussing data in the areas of academic achievement, discipline & special education placement over a three year period.
Keywords: audit, district, multi-tiered model, integrated model
Evaluation of the State Program Improvement Grant (SIG) Program: Final Report (PDF, December 2007, Fiore, T.A.; Munk, T.E.; Langham, A., Westat and Magliocca, L.A., Ohio State University) Includes a systems change model and an elegant way of designating outcomes as robust, solid, probable and possible. Outcome evidence is provided to support the categorization of the outcomes. There is also a very interesting section on administrative leadership. Please note that this is a SIG rather than SPDG report. As such, some of the outcomes are a bit dated (e.g., participation in assessment) given the passage of NCLB.
Findings from Westat's Evaluation of the State Improvement Grant Program: Implementation Issues, Systemic Change and Project Outcomes (PPT, February 2010, Fiore T.A.; Langham, A., Westat and Magliocca, L.A., Ohio State University) This presentation describes how results from Westat's SIG Program evaluation can be used to inform SPDG evaluations.
Categories: Main Content