State reforms to develop K-12 academic standards and to assess the performance of all students on these standards have resulted in a substantial increase in the number and scope of contracts with testing companies for statewide assessment programs. Many of these assessments are high-stakes for students (e.g., graduation or grade promotion tests) and/or educators (e.g., accountability programs). With high school diplomas, monetary awards or federal funding for schools and school systems dependent on test results, it is imperative that state assessments be of high quality, meet professional standards for best practice, be delivered in a timely manner, and be scored accurately. With increasingly tight budgets, it is similarly imperative that assessment programs be developed and implemented in an efficient and cost-effective manner without sacrificing quality.
Creating a high-quality state testing program requires both cooperation and accountability. It recalls the arms control motto, "trust but verify." The main participants in this relationship are state agency staff and test vendor staff. To support these efforts, the participating states in developing these Model Contractor Standards and State Responsibilities for State Testing Programs must communicate more clearly today's expectations for the development and administration of high-quality, efficient, and defensible, high-stakes state testing programs.
For vendors, commitment to following the "vendor standards" described herein can be cited as evidence of self-regulation and adherence to best practices. Of course, such standards are also designed for use by states in designing contractual relationships with vendors and in managing and overseeing those relationships. For states, the outlined "state responsibilities" are intended to provide a model for what is necessary to create a high-quality testing program and to serve as guidelines for policymakers enacting reforms in state testing programs. Some of the state responsibilities also describe important requirements for legal defensibility of high-stakes components within a state testing program.
The Preplanning Standards address antecedent activities and decisions critical to the production of a comprehensive Request for Proposal (RFP) describing the products and services the state wants a vendor to supply to its testing program. Assuming the criteria specified in the Preplanning responsibilities have been met, the Development and Administration sections provide guidelines for the specification of contract activities and execution of the contracted work. The Uses of Data section deals activities subsequent to the administration of the test, but directly connected with the interpretation of test data and score reports. Each section begins with a brief introduction that provides background and explanatory information.
Many of the standards herein have both a vendor and state corollary. The purpose of this document is to clarify ideal roles and obligations in a typical relationship between a state and vendor, either with respect to test development or scoring and administration. The "typical" relationship is assumed to be that the state, in response to action by its legislature, has developed academic standards and is contracting with one vendor to purchase a custom-built assessment based on its own academic standards and with another vendor to handle test administration and scoring.
Assumptions of some kind are clearly necessary for the design of a "model" document of this type. Of course, there are few "typical" states that precisely, or even nearly, mirror all of the arrangements assumed here. Among the many possible variations are:
- A state agency buys access to a commercially developed and published test.
- A state agency hires a vendor to develop a test that is then owned by the agency. The state requires the vendor to provide materials, scoring services, and reporting services.
- A state agency hires a vendor to develop tests that will be owned by the state and then hires a second vendor to do the test administration, scoring, and reporting activities.
- A state agency buys access to a test publisher by one vendor but then hires another vendor to administer, score, and report the results.
- A state agency develops the test with the assistance of local districts and state universities. It then hires a vendor to administer, score, and report the results.
Evaluating the acceptability of a process or contract between a state and vendor should not rely on the literal satisfaction of every standard or responsibility in this document, nor can acceptability be determined through the use of a checklist. Further, while retaining decision-making authority, a state may benefit from seeking the advice of the vendor regarding alternative methods for satisfying a particular guideline. Similarly, a responsible vendor will seek the state's advice or feedback at every point along the way where important decisions must be made. Regardless of how roles are defined and tasks are delegated, states retain ultimate responsibility and authority for state testing programs.
Throughout this document, the terms "testing companies" and "industry" apply generically to refer to all providers of test content, printing, scoring, validation, and other testing related services, whether for-profit or not-for-profit, public or private. The term "state" applies to the educational enterprise of the fifty states, territories, and other appropriate jurisdictions and includes state education agencies (SEAs), state boards of education, and other official educational entities.
For a state testing program to follow standards of best practice, several preconditions should be met before an RFP is developed, a contract is signed with a vendor, and test development begins. These preconditions are important in enabling a vendor to produce a quality test that will satisfy the state's expectations. These preconditions include actions to be taken by the state legislature (or other responsible entity) as well as the state agency with authority to implement the testing program. The purpose of these preconditions is to support production of an RFP that specifies in detail the services and products to be provided by the vendor given reasonable timelines and resources, to ensure that the state has knowledgeable and adequately trained staff competent to assign all required activities to either itself or the vendor, to ensure adequate planning and funding to produce a quality testing program, and to ensure that staff is in place to competently supervise the vendor relationship going forward.
The state legislature should enact reasonable timelines and provide adequate funding for the testing program. The legislation should also include: statements of purpose; designation of authority for important tasks (e.g., standard setting); responsibilities of the state agency and the local districts; types of reports of results; uses of the data (e.g., school or district accountability); contracting authority.
In general, the lead time for developing a new, high-stakes assessment includes a minimum of 38 months: 6 months planning, preparation, & development of test blueprint; 6 months item writing & editing; 6 months item tryouts & analyses; 6 months preparation of field test forms & supporting materials; 6 months field testing, research studies (e.g., opportunity to learn surveys in the case of high-stakes tests) & analyses; 6 months development of final test forms, edit supplementary materials, set passing standards, finalize security, accommodations, & reporting policies; 2 months administer final tests, score, equate, and report results. This timeline begins with the signing of a contract with a vendor and assumes that state content standards for the subjects being tested have already been adopted.
Preplanning activities leading to the development of a comprehensive RFP, vendor bid time, proposal evaluation, and negotiations for the award of a final contract will often add at least another 6 months. Unanticipated complications that often accompany implementation of a new testing program will also add additional time. Thus, legislation that creates a new testing program should allow approximately 4 years from the time of passage until the first live tests are administered. States with well-established testing programs and experienced staffs in place may be able to reduce the time required to develop additional tests (i.e., development limited to certain grades not previously covered).
Three to four years of lead time is also consistent with legal requirements for an adequate notice period and opportunity to learn (OTL) for tests with high stakes for individual students. OTL requires sufficient time for implementation of curricula and instruction that provides an adequate opportunity for students to have been taught the tested content before their initial attempt to pass high-stakes tests. If a state has developed sufficiently clear academic standards, notice requirements for OTL may be triggered by the publication date of the standards if a strong communication effort is undertaken and the state can demonstrate that schools have aligned instruction with the standards. When offered opportunities for input, potential vendors should alert states to unreasonable timelines and propose alternatives reasonably calculated to meet professional standards for best practice.
In addition to providing sufficient development time, legislation for a new testing program must provide adequate funding for agency preplanning activities, test development, test administration, and other program activities necessary to produce a quality test for each of the mandated grades and subjects. The required activities may be conducted by the state or an outside vendor, but funding must be sufficient so that no important steps are left out.
Development costs do not depend on the number of students to be tested. Therefore, a small testing program with limited funds and a relatively small number of students over which to spread the cost may only be able to develop tests for fewer grades and/or subjects than larger testing programs. Options for state cooperation to improve efficiency and lower costs are described in the accompanying innovation priorities document.
Where custom tests are to be designed on the basis of state academic standards, special care should be taken to develop high quality standards that are rigorous, clear and specific, consistent with sound research on curriculum and instruction, and well-organized to ensure that the lower levels serve as a sound foundation for the upper levels.
While sound standards-based tests must be well aligned with state standards, the standards should be designed so that they serve that purpose well. Consensus-building within a state is important in order to develop support for broad implementation, but consensus does not always or necessarily lead to quality. Where, for example, consensus is reached as a result of agreement on broad or vague standards statements, schools may focus excessively on the test itself for guidance on what to teach; tests are typically not designed to carry such a heavy load, leading to criticisms that teachers are "narrowing the curriculum" to what is being tested. More detailed criteria and models for high-quality standards exist and have been identified by other organizations.
The state agency responsible for testing should include staff with adequate knowledge and training in psychometrics, curriculum, communication, policy, special populations, and contracting to develop a comprehensive RFP, to complete all necessary activities not assigned to the vendor, and—especially—to monitor vendor performance throughout the contract period.
Agency staff must play an active role in the development of a quality testing program. In order to decide which services and products should be included in an RFP, agency staff must thoroughly understand the test development process and the requirements for a psychometrically and legally defensible test. Where knowledge and training are inadequate or lacking, staff should seek training and assistance from outside experts.
Agency staff must be prepared to complete all required steps and activities not specifically delegated by the state to a vendor and to competently monitor all contract activities. It is possible for a state agency to outsource some of the steps and activities required prior to the selection of the main test development vendor, though adequate expertise should always exist on staff to monitor the performance of all such additional vendors.
The staffing needs of the state agency to support a statewide assessment program are significant. This is going to be one of the more serious tasks confronting states. At the minimalist end of the continuum, a state could theoretically have one person be the assessment coordinator and simply allow the contractor to do everything. At the other end, a state agency can hire sufficient number of staff members to coordinate the work of multiple contractors, assume primary responsibility for quality control work, and provide data analysis and dissemination/training activities within the state.
State legislatures do not have to create a lot of permanent positions for the state bureaucracy if the agency molds together an office with a critical mass of permanent employees and out-sources tasks requiring more personnel (e.g., test development, editing, and test form assembly).
The state agency responsible for testing should develop a comprehensive RFP that describes clearly and in detail both the end products to be provided by the vendor and the development process.
The RFP is the roadmap for the creation of a quality testing program. It should contain detailed specifications for all key areas, including, but not necessarily limited to, the totality of services and products to be provided, timelines for performance, quality criteria, responsiveness criteria, mid-course correction opportunities, and the process for evaluating proposals. When developing the RFP, agency staff should be aware of all required activities for a defensible testing program and should specifically assign responsibility for each activity to itself or to the vendor. It is occasionally the case that the state is not sure how to accomplish a certain goal of the testing program and wants vendors to propose a solution. In this case, the RFP should clearly separate those requirements that are firm, those that are aspirational, and those that are simply unknown. It should be very clear to vendors whether they are responding to specific requirements or proposals for implementing general requirements. The state should also clearly spell out timelines in the development process, both for test development and for scoring/administration.
To provide a fair comparison of proposals received in response to an RFP, states should require all vendors to either: (a) provide costs for a fixed set of services and products specified in detail in the RFP; or (b) specify in detail what services and products they could provide for fixed incremental costs. The method should be chosen in advance by the state and clearly specified in the RFP.
When vendors bid on different combinations of services and products, their proposals are not comparable and it is difficult for the state to evaluate cost effectiveness. The lowest bid may have to be accepted when essential activities are missing or incomplete. When all vendors use the same method, a fairer evaluation of their proposals is possible. States with precise knowledge of the test products and services they will need are more likely to benefit from option (a), while states with less precise knowledge, or whose plans may change, are more likely to benefit from option (b).
Clearly, this document will apply differently in each of these different scenarios. It is hoped, however, that the model is sufficiently clear and well-defined that a state using a different arrangement would be able to determine the necessary adjustments in those sections that require adjustment. Even in cases where the relationship or tasks in a given state appear to fit perfectly with the model described here, this document is only intended to provide a framework to ensure that relevant issues are addressed. Important issues need to be resolved in a way that is consistent with a state's political process.
Jeff C. Palmer is a teacher, success coach, trainer, Certified Master of Web Copywriting and founder of https://Ebookschoice.com. Jeff is a prolific writer, Senior Research Associate and Infopreneur having written many eBooks, articles and special reports.
Source: https://ebookschoice.com/state-reforms-to-develop-k-12-academic-standards/
