Executive summary as issued Friday by William Sheldrake, president of Policy Analytics, and John Grew, executive director for state relations and policy analysis at Indiana University:
Summary of Process Undertaken
To gain an understanding relative to the accountability system, the authors conducted interviews with current and former IDOE staff who were involved in system development as well as representatives of educational organizations. Through these interviews, a narrative description of the process of developing and implementing the accountability rule from early 2010 through October, 2012, was prepared and is included in Section IV of the report.
Also key to the project was the ability to independently run results for the accountability system and its two models and compare output (school grades) to the final grades issued by the IDOE in October, 2012. With the assistance of Legislative Services Agency staff, a program was developed that permitted use of IDOE data to produce various reports that enabled the authors to determine if the 2011-12 grades were awarded in accordance with the Bennett administrative rule. This process and findings are detailed in Section V of the report.
1) The authors found that IDOE underestimated administrative and technical challenges associated with developing the new administrative rule, computer programming and testing necessary to implement the new rule, and obtaining feedback relative to 2011-12 grades.
2) Because the Bennett administrative rule did not contemplate all of the numerous school configurations in place during 2011-12, it was necessary for IDOE to make certain interpretations including the decision to eliminate HS. scores from the Christel House Academy’s grade. The authors found that this interpretation was consistently applied to 16 other schools which had analogous situations.
3) The removal of the EMS “subject matter growth caps” impacted the final scores and grades of 165 schools. According to former IDOE staff, lan-guage in the final approved rule intended that there be no EMS subject matter caps, however, these caps were erroneously included in the computer programming of the model. This mistake was discovered and corrected prior to the September 19, 2012 embargoed release of school grades and related data.
4) With regard to the final disposition of the grade for the Christel House Academy, the authors heard from both Dr. Bennett’s critics and supporters that his focus on the Christel House Academy was because of its widely accepted reputation as an excellent school, which functioned as a quality control indicator. However, when the school’s grade was initially found to be a C, Dr. Bennett expressed surprise and dissatisfaction. These expressions prompted an energetic response to find solutions to what was perceived to be an unfair and inaccurate result. From the emails, it is apparent that IDOE staff worked diligently, over a period of several days in an effort to respond to this situation.
In the end, the Authors found that the two adjustments administered to determine Christel House Academy’s final grade were plausible and the treatment afforded to the school was consistently applied to other schools with similar circumstances.
5) Although efforts were made by Dr. Bennett and his staff to interact with educational stakeholders and practitioners, a significant portion of the educational community did not understand or trust in the accuracy or fairness of the Bennett Rule’s Metrics, did not believe the that the metrics represent-ed essential accountability constructs, and did not believe that the Rule treated different school formats
6) IDOE’s ability to finalize the accountability system, perform quality control simulations, and to produce final output was clearly compromised by the loss of several key technical staff beginning in summer, 2011 through summer, 2012.
7) In part due to the loss of key IDOE technical staff, there was inadequate time to complete final pro-gramming and perform quality control work, prior to release of each school’s final grades.
1) Disposition of 2011-12 Grades: The grades issued for 2011-12 should be corrected for errors discovered relative to high school grades for several schools and IDOE should publicize various IDOE interpretations of the administrative rule that were necessary to make grade determinations in cases in which the rule did not provide complete clarity.
2) Reporting of 2012-13 Grades: As required by statute, IDOE should proceed with finalizing and reporting results for 2012-13 as quickly as possible after reasonably assuring school administrators and the public regarding the integrity of the test results. Adequate time should be provided for vetting preliminary results, for schools to file appeals, and for IDOE to review and respond.
For the 2012-13 school year and subsequent years until the new accountability system required by HEA 1427-2013 is implemented, state policymakers should consider not subjecting a school to state interventions described in IC 20-31-9-4 due to a sixth consecutive year of placement in the lowest category or designation of school performance.
3) Developing a Revised Accountability System Under HEA 1427-2013: The authors observe and recommend:
A. The authors observe that the recently announced memorandum of understanding between the Governor, the General Assembly, and the Superintendent for establishing a collaborative process for development of a new accountability rule is an excellent step towards increasing support by the educational community and the public.
B. The process of development of a new system should:
1. Provide for extensive involvement by experts and practitioners from the education community.
2. Provide for transparency in all decision-making.
3. Result in development of a new system that is as simple as possible, more easily understood, and equitable.
C. In compliance with HEA 1427 – 2013, the new accountability system should incorporate measures that involve less reliance on standardized tests passage rates and more reliance on individual student growth based on criterion-referenced measures.
4) Further Recommendations regarding the Revised Accountability System:
A. Additional measures for the EMS model should be included, besides the two student test measures, that provide additional indicators of school performance.
B. Because of the complexity involved in implementing any new accountability system, the system should be piloted prior to implementation, if possible, permitting IDOE to solicit and receive extensive feedback from schools, adequately perform programming tests, and evaluate policy components incoporated into the system.
C. In order to ensure that the legislative branch has the capability to perform analyses on the new accountability system, LSA staff should be provided with ongoing access to all data and computer programming necessary for it to replicate results and respond to various inquiries from legislators about the system.
- Charter’s grade change ‘plausible’
- Report: Put school interventions on hold until A-F system is fixed
- Indiana Chamber: Rely less on standardized tests
- Ritz makes accountability review panel appointments
- David Long: A-F findings will help immensely
- Democrats for Education Reform: What happened is unacceptable
- Ritz: Indiana needs a system that is fair, transparent, trustworthy
- Indiana Democrats: These aren't just grades on paper