Reducing Behavior Problems in the
Elementary School Classroom

Appendix A:
Postscript from the
Institute of Education Sciences

WHAT IS A PRACTICE GUIDE?

The health care professions have embraced a mechanism for assembling and communicating evidence-based advice to practitioners about care for specific clinical conditions. Variously called practice guidelines, treatment protocols, critical pathways, best practice guides, or simply practice guides, these documents are systematically developed recommendations about the course of care for frequently encountered problems, ranging from physical conditions, such as foot ulcers, to psychosocial conditions, such as adolescent development.(1)

Reducing Behavior Problems In the Elementary School Classroom This guide is intended to help elementary school educators develop and implement effective prevention and intervention strategies that promote positive student behavior. The guide includes concrete recommendations, ways in which each recommendation could be carried out, and recognized roadblocks to implementation of each recommendation. Introduction Overview Scope of This Practice Guide Recommendation 1: Identify Specifics of Problem Behavior Recommendation 2: Modify the Learning Environment Recommendation 3: Teach and Reinforce New Skills Recommendation 4: Draw on Relationships With Colleagues, Families Assess School-Wide Behavior Problems Appendix A: What Is a Practice Guide? Appendix B: About the Authors Appendix C: Potential Conflicts of Interest Appendix D: Technical Information on the Studies References See more articles from the U.S. Department of Education in our DOE article archive.

Practice guides are similar to the products of typical expert consensus panels in reflecting the views of those serving on the panel and the social decisions that come into play as the positions of individual panel members are forged into statements that all panel members are willing to endorse. Practice guides, however, are generated under three constraints that do not typically apply to consensus panels. The first is that a practice guide consists of a list of discrete recommendations that are actionable. The second is that those recommendations taken together are intended to be a coherent approach to a multifaceted problem. The third, which is most important, is that each recommendation is explicitly connected to the level of evidence supporting it, with the level represented by a grade (strong, moderate, low).

The levels of evidence, or grades, are usually constructed around the value of particular types of studies for drawing causal conclusions about what works. Thus, one typically finds that a strong level of evidence is drawn from a body of randomized controlled trials, the moderate level from well-designed studies that do not involve randomization, and the low level from the opinions of respected authorities, buttressed by theory or research that does not meet criteria for high or moderate ratings (see table 1). Levels of evidence also can be constructed around the value of particular types of studies for other goals, such as the reliability and validity of assessments.

Practice guides also can be distinguished from systematic reviews or meta-analyses, such as the What Works Clearinghouse (WWC) intervention reviews or statistical meta-analyses, which employ statistical methods to summarize the results of studies obtained from a rule-based literature search. Authors of practice guides seldom conduct the types of systematic literature searches that are the backbone of a meta-analysis, although they take advantage of such work when it is already published. Instead, authors use their expertise to identify the most important research with respect to their recommendations, augmented by a search of recent publications to ensure that the research citations are up-to-date. Furthermore, the characterization of the quality and direction of the evidence underlying a recommendation in a practice guide relies less on a tight set of rules and statistical algorithms and more on the judgment of the authors than would be the case in a high-quality meta-analysis. Another distinction is that a practice guide, because it aims for a comprehensive and coherent approach, operates with more numerous and more contextualized statements of what works than does a typical meta-analysis.

Thus, practice guides sit somewhere between consensus reports and meta-analyses in the degree to which systematic processes are used for locating relevant research and characterizing its meaning. Practice guides are more like consensus panel reports than meta-analyses in the breadth and complexity of the topic that is addressed. Practice guides are different from both consensus reports and meta-analyses in providing advice at the level of specific action steps along a pathway that represents a more-or-less coherent and comprehensive approach to a multifaceted problem.

PRACTICE GUIDES IN EDUCATION
AT THE INSTITUTE OF EDUCATION SCIENCES

The Institute of Education Sciences (IES) publishes practice guides in education to bring the best available evidence and expertise to bear on the types of systemic challenges that cannot currently be addressed by single interventions or programs.

Although IES has taken advantage of the history of practice guides in healthcare to provide models of how to proceed in education, education is different from healthcare in ways that may require that practice guides in education have somewhat different designs. Even within health care, where practice guides now number in the thousands, there is no single template in use. Rather, one finds descriptions of general design features that permit substantial variation in practice guides across subspecialties and panels of experts.(2) Accordingly, the templates for IES practice guides may vary across practice guides and change over time and with experience.

The steps involved in producing an IES-sponsored practice guide are first to select a topic, which is informed by formal surveys of practitioners and requests. Next, a panel chair is recruited who has a national reputation and up-to-date expertise in the topic. Third, the chair, working in collaboration with IES, selects a small number of panelists to co-author the practice guide. These are people the chair believes can work well together and have the requisite expertise to be a convincing source of recommendations. IES recommends that at one least one of the panelists be a practitioner with experience relevant to the topic being addressed. The chair and the panelists are provided a general template for a practice guide along the lines of the information provided in this preamble. They also are provided with examples of practice guides. The practice guide panel works under a short deadline of 6-9 months to produce a draft document. The expert panel interacts with and receives feedback from staff at IES during the development of the practice guide, but they understand that they are the authors and, thus, responsible for the final product.

One unique feature of IES-sponsored practice guides is that they are subjected to rigorous external peer review through the same office that is responsible for independent review of other IES publications. A critical task of the peer reviewers of a practice guide is to determine whether the evidence cited in support of particular recommendations is up-to-date and that studies of similar or better quality that point in a different direction have not been ignored. Peer reviewers also are asked to evaluate whether the evidence grade assigned to particular recommendations by the practice guide authors is appropriate. A practice guide is revised as necessary to meet the concerns of external peer reviews and gain the approval of the standards and review staff at IES. The process of external peer review is carried out independent of the office and staff within IES that instigated the practice guide.

Because practice guides depend on the expertise of their authors and their group decision-making, the content of a practice guide is not and should not be viewed as a set of recommendations that in every case depends on and flows inevitably from scientific research. It is not only possible but likely that two teams of recognized experts working independently to produce a practice guide on the same topic would generate products that differ in important respects. Thus, consumers of practice guides need to understand that they are, in effect, getting the advice of consultants. These consultants should, on average, provide substantially better advice than an individual school district might obtain on its own because the authors are national authorities who have to reach agreement among themselves, justify their recommendations in terms of supporting evidence, and undergo rigorous independent peer review of their product.

Go to Appendix B:
Institute of Education Sciences: About the Authors

Footnotes
1. Field and Lohr (1990).
2. American Psychological Association (2002).

Publication posted to Education World 07/06/2009
Source: U.S. Department of Education; last accessed on 07/06/2009 at
http://ies.ed.gov/ncee/wwc/pdf/practiceguides/behavior_pg_092308.pdf