Schools, school districts, and states looking for help in developing a comprehensive improvement plan can turn to New American Schools, a nonprofit consulting organization. Harold C. Doran of New American Schools recently talked with Education World about his views on testing, reform, and the No Child Left Behind Act. Included: Descriptions of comprehensive school reforms.
Education World: What factors could delay or dilute the No Child Left Behind (NCLB) Act's goal of achieving educational accountability at all our nation's schools?
Harold Doran: Two factors could delay the NCLB goal of educational accountability. First, the development of high-quality instructional standards that define what students know and are able to do was a part of the original Elementary and Secondary Education Act (ESEA). Many states have already developed standards and are well positioned to begin the development of statewide assessments to measure those standards. However, some states have not yet developed content or performance standards as required by the original legislation. States that have not developed those standards will be unable to produce assessments, which are the cornerstone of the NCLB test-based accountability plan.
The second factor is the ability of states to develop and administer assessments that are aligned with state-defined content standards for instruction. For example, an off-the-shelf norm-referenced test will not align with state standards unless it is augmented in some manner. States may choose to develop criterion-referenced assessments that specifically measure their own standards.
In addition, two factors may dilute NCLB progress toward educational accountability. The first is the ability to measure adequate yearly progress (AYP) and define what it means to be proficient. Tests are not designed to measure how much growth is enough. Educational measurement experts and policymakers must determine what adequate growth is. When defining AYP, states must struggle with many technical issues, such as which test metric is most appropriate (percentile ranks, normal curve equivalent scores, or scale scores); what student level data there is access to and how that data can be used; whether the test is designed to measure student progress over time and, if so, which value-added method is best.
Second, it is imperative that states not set the "proficient" bar too low. Obviously, an easily attained standard does not result in an improved educational system. If states set the bar too low when defining proficiency, then the full intention of NCLB will be watered down.
EW: How could NCLB be improved?
Doran: Although NCLB makes significant progress in its intentions toward educational accountability, the act does not include a clear mandate for the most appropriate method of measuring student achievement -- value-added analysis. It is imperative that student progress be tracked over time and that educators not be held accountable for factors that are out of their control. Traditional methods for assessing school effectiveness -- such as comparing the average of different cohorts of students -- simply are not defensible in the world of high-stakes educational accountability. More emphasis should be placed on year-to-year improvement than on comparison of different groups of students over time.
EW: What concerns about implementing the act do you hear from the schools you work with?
Doran: Primarily, states are concerned about AYP, value-added analysis, and comprehensive accountability design. States are asking for help in constructing comprehensive accountability strategies. They also ask us how to assemble a rigorous measure of AYP so they're in compliance with the accountability provisions of the law.
Additionally, states want more than simple test scores -- and rightfully so. Test scores cannot tell the entire story of a school. We help states identify how such additional qualitative information as comprehensive school reviews and diagnostic assessments can complement accountability provisions and provide a rich, detailed picture of a school's effectiveness. Using multiple measures of effectiveness prevents schools from being labeled as under-performing when results may be due to random fluctuations in test data.
States also are looking for guidance on what type of assistance and improvement strategies are offered to schools that are designated as under-performing or failing. We provide them with strategies that can help improve the governance structure and student achievement for those schools.
EW: What are the first things you ask when you meet with a school's administrators?
Doran: We want to know to what extent administrators are using data to track achievement over time; what their current capacity to warehouse and manage student achievement data is; what assessments they are required to use and how those assessments can be used in an accountability plan; and what their current capacities, incentives, and opportunities for creating a coherent educational environment focused on school improvement are. Additionally, we want to know the purpose and the target audience of the analysis; that is, will the evaluation be cumulative or formative, and whom is the data being collected for?
EW: What do you see as the pros and cons of high-stakes testing?
Doran: That's an important question -- one that too often focuses on the con side. Often missing from the debate are the many positive consequences affiliated with high-stakes testing programs. Those may include improved professional development, improved quality of educational tests, and more focused instructional programs. Many educators have participated in misguided professional development sessions that were not sustained over time or that were based on inadequate research. High-stakes testing provides an opportunity for educators and parents to engage in professional development targeted at disaggregating student achievement data and at action planning for improved instruction based on test analysis.
Properly constructed tests can provide valuable information to classroom teachers about the quality of instructional programs and the learning of individual students. [The value of that information], of course, rests upon the timely return of test results to teachers. It makes no sense to provide classroom teachers with information about last year's students. That is why I am so excited about computer adaptive testing (CAT). The potential applications of CAT include tests that take less time to administer, are more reliable, and provide immediate feedback.
My chief concern among the cons of high-stakes testing is the use of the same test each year. It is certainly less expensive to reuse a test, but getting by on the cheap will not improve the quality of education or meet the full intentions of NCLB. Certain techniques can be used that equate old forms and new forms of a test so that scores among the different test forms are comparable. When tests are reused each year, test score gains may not be a result of actual gains in student learning. Many other factors not due to an increase in school effectiveness -- such as teaching to the test, narrowing the curriculum, coaching, and cheating -- may produce apparent test score gains.
Tests in and of themselves do not improve the quality of education; teachers do. Unless testing programs are coupled with comprehensive strategies for disaggregating and interpreting data and implementing curricular and instructional approaches to school improvement, it is unlikely anything will change. It would be like repeatedly taking the temperature of a sick baby and then not treating the baby's illness.
EW: What are the most important factors in improving student achievement?
Doran: Schools and districts must create a coherent educational environment in which all educators are committed to high-quality professional development, allocate resources focused on improving instruction, use and interpret educational data in appropriate ways, and involve parents and the community in all facets of schooling. Additionally, I would add that strong leadership, and governance models that focus on strategic planning, accountability, and the decentralization of decision making are critical factors. The school's key stakeholders, including parents, teachers, and the school principal, should have the authority to make important decisions about the school and its programs. Granting more autonomy in exchange for greater accountability and results permits schools to be innovative and to focus on the primary purpose of schooling -- teaching and learning.
This e-interview is part of the Education World weekly
Wire Side Chat series. Click here
to see other articles in the series.