The standards and responsibilities in this section describe activities necessary to administer psychometrically and legally defensible high-stakes tests efficiently and to a high standard of quality. These activities are typically assigned to the vendor but responsibility may be shared with the agency. If the agency decides to retain responsibility for an activity, the agency may seek advice from the vendor but should clearly indicate that expectation in the RFP (Request for Proposal) and resulting contract.
For each activity or portion of an activity assigned to a vendor, the RFP and resulting contract should describe in detail what is expected of the vendor, any special conditions or limitations, and the compensation to be paid. If a state requests changes or delegates additional responsibilities to the vendor after the contract has been signed, the state may have to renegotiate the price.
Where the state has delegated such responsibility to the vendor, a plan for developing and maintaining a database of student and school testing information shall be created. The plan should provide mechanisms for tracking student movement, keeping track of retests, collecting demographic information needed for data analyses and reporting, ensuring confidentiality of individually identifiable student data, correcting student identification numbers as needed, and updating files when errors are uncovered.
With multiple subjects, multiple grades, and retests, it is essential that test data be organized in a format that is accessible, accurate, provides all data needed for state and federally-mandated analyses, and tracks the testing history of students, items and test forms. Because most of the data collected will involve confidential or secure information, detailed policies for protecting the confidentiality of data collected and retained must be developed.
The RFP and resulting contract should clearly specify vendor expectations in this area. Creation and maintenance of electronic databases is expensive and the cost may be prohibitive for some small testing programs. If the state chooses to maintain or collect its own data, the contract should clearly specify the form and content of data files the vendor is expected to provide to the agency.
The state has the responsibility to collect and report useful data to a variety of constituencies, including satisfying federal requirements. Where permitted by state law, a database of student and school information can be highly useful. The state is ultimately responsible for ensuring that such a database of student and school information is maintained properly; where it has elected to delegate this responsibility to the vendor, the state is responsible for monitoring the work. States choosing not to use a state level database possess other means for carrying out this function that a vendor does not, such as requiring school districts to provide the data.
When a state chooses not to contract with a vendor to maintain a state database, the state must assume the responsibility for collecting and maintaining assessment data in a form that will produce usable information for various constituencies and that satisfies applicable law. Appropriate procedures must be implemented to satisfy confidentiality requirements and to ensure proper use and access to all data. While a state has options other than creation of a statewide database, such options limit the usefulness of the available data.
The vendor proposal and resulting contract shall specify procedures for determining quantities of materials to be sent to districts (or schools), tracking test materials that have been sent, and resolving any discrepancies. A mechanism shall be developed for ensuring the accuracy of enrollment data supplied to the vendor and for updating school requests for additional or replacement materials. Instructions for handling test materials and for test administration (e.g., Administrator’s Manuals) shall be shipped to districts at least one month prior to testing to allow time for planning and staff training.
Valid and fair test results require adherence to all standard test administration conditions and security procedures by all test administrators. Test administrators are best prepared for this task when sufficient quantities of materials are received prior to testing and training has been provided using the actual instructions to be employed during testing. In order for districts to receive sufficient quantities of materials, accurate and timely enrollment information must be supplied to the vendor and a mechanism must be established for efficiently responding to requests for additional or replacement materials. Administrator’s manuals and instructions for handling test materials are important communications for district planning and test administrator training and should be available for study prior to the receipt of test materials. By making such procedures and communications responsive to the needs and concerns of districts and schools, greater cooperation should be achieved.
Timelines and procedures for receipt and return of test booklets and answer sheets shall be consistent with an agreed upon test security policy and specifications in the RFP and resulting contract. Generally, test materials should arrive in sealed containers no earlier than one week prior to testing, should remain in a secure, locked storage area while in district/schools, and should be repackaged and picked up within two days after test administration has been completed.
The security of test materials and the accuracy of state test data depend on the timely receipt and return of test materials by schools. The RFP and resulting contract should provide detailed descriptions of all security procedures to be followed by the vendor, including procedures for distributing, tracking, and returning test materials.
The state may wish to delegate the responsibility for training of school and district personnel to the vendor.
States must develop and implement a policy for ensuring that schools and districts comply with the policies enumerated in the RFP and contract. When non-compliance is an issue, the state must be able to impose sanctions or otherwise compel action on the part of the local education agency. In addition, the state is responsible for the training of school and district personnel in the security policies.
The state retains responsibility for training, monitoring, and investigating local education agencies’ compliance with established test security procedures. Administrative rules or statute should enumerate educators’ responsibilities, proscribed activities and sanctions for violators. The state also has a duty to monitor contractor activities and to assist in the resolution of unforeseen circumstances (e.g., school closing on test week due to a major flood or storm damage).
Reliability for any high-stakes exam should be at the highest levels. Where open-ended response items or essays are included in an assessment, two raters shall score each response with at least 70% agreement on initial scoring. When raters disagree on initial scoring, resolution (re-scoring) by a senior or supervisory rater is required.
Tests that are to be used for high stakes for either educators or students should attain high standards of reliability, as may be exemplified by an overall internal consistency rating of at least 0.85 to 0.90 on a 0-1 scale. Such overall reliability will not be attained unless hand scored items, typically essays or other open-ended items, also attain adequate levels of inter-rater reliability. Trained raters using detailed scoring rubrics who are periodically rechecked for accuracy should be able to score responses with a high degree of agreement. When two raters disagree and the test is being used for high-stakes decisions about individual students, fairness dictates that an experienced third rater resolve the discrepancy. (In cases of items with a large number of score points, "agreement" may consist of adjacent scores.) Alternative procedures for computerized scoring of open response items can include one trained rater serving as the second rater, with similar procedures for resolving discrepancies. For assessments that do not include high-stakes for students, a single rater may be sufficient as long as proper procedures are in place for checking samples for rater drift.
Quality control procedures for checking the accuracy of all item information, student scores and identification, and summary data produced by the testing program shall be developed and implemented. The standard for the error rate of data reports provided by a vendor to an agency for review is zero.
The vendor has a duty to formulate and implement quality control procedures for data generation that have as their goal the production of error-free reports and summary data. All data operations should be subject to multiple checks for accuracy before being released to the state. The vendor should document its quality control procedures for state review and create detail logs that trace the application of those procedures to the state data reports.
Data reports released by state agencies must also be error free. The state must develop its own quality assurance policy to monitor the work of the vendor. Data reports should be examined before general release. Effective techniques prior to release include: running score and summary reports on "dummy" data to ensure that the output is correct; close examination of a sample of the reports; sending preliminary data to select schools or districts for review; or having the state TAC (Technical Advisory Committee) or an outside consultant examine a sample of the reports.
When erroneous data is released publicly, the testing program loses credibility and incorrect decisions may be made. It is imperative that all reasonable procedures be used to check the accuracy of all testing program data before report distribution or public release. The vendor has primary responsibility to find and correct errors, with agency staff acting as a final check. The expectation of zero errors is contingent upon the state providing all necessary information. Nontrivial vendor errors may trigger financial penalties in states that include such provisions in their contracts.
When an item error, scoring error, or reporting error is discovered, the vendor shall notify state staff immediately. Vendor staff should then work closely with agency staff, and technical advisory committee members or outside consultants where appropriate, to develop a comprehensive plan for correcting the error. The plan should include the provision of timely and truthful information to the affected stakeholders.
The way in which an error becomes public and the actions taken to correct it can have a major impact on public perceptions. Straightforward communication of information as it becomes available and immediate corrective action can help restore public confidence in the vendor and the state testing program. Error does not include reasonable differences of opinion.
Testing report forms shall be received by the district or other responsible entity (e.g., charter school) no later than the end of the semester in which testing occurred. Individual student reports for multiple-choice tests should be received within 2 weeks of the date on which answer documents were received by the vendor. School, district, and state reports should be produced within 2 weeks of the cutoff date for return of answer documents. For tests containing open-ended items or essays requiring ratings, individual student reports should be received within 6 weeks of the date on which answer documents were received by the vendor. School, district, and state reports should be produced within 6 weeks of the cutoff date for return of answer documents. Where an assessment is composed entirely, or almost entirely, of essays or other open-ended items, more time is likely to be necessary for scoring. The contract should specify any antecedent conditions that must be met by the agency for reports to be delivered on time.
For data to be useful for instructional improvement and for making decisions about enrollment in remedial classes or summer school, it must be received prior to the beginning of the next instructional semester following the date of testing. Turnaround time will vary depending on program complexity but should be kept as short as possible while maintaining accuracy. If state staff with expertise believe that these timelines do not reflect their needs, they can elect to deviate from them; however, a rationale should be provided. It is understood that there are tradeoffs inherent in the timeline process, and state policymakers should be able to explain their reasoning for allowing vendors to go beyond these timelines, if they elect to do so.
Plans should include rules for scoring of late arriving papers, particularly with regard to calculating summary statistics. (E.g., how long should one school be allowed to hold up the state summary statistics?) Clear guidelines in this area are especially important for tests that include open-response items; in such cases, a contractor will typically have only a limited window of time to implement the work of the human raters. The beginning date of the 2-week or 6-week scoring window should be clearly defined in the contract. Further, the scoring timeline for the contractor should be defined to include all activities that the contractor needs to perform (i.e., including all of those required to ensure the integrity of the data, not just the scoring itself once these activities have been completed).
When the RFP and resulting contract provide reasonable timelines for scoring and reporting, and the agency has met its obligations, states may wish to include contractually agreed upon incentives for performance by the vendor. Incentives may include a bonus for early completion or a penalty for late performance or errors. Administration activity timelines may well exceed typical annual state appropriations; states may benefit from multi-year funding plans and contracts across fiscal years (which may be cancelled if the budget must be reduced or the program is eliminated). States must, of course, stay within statutory constraints imposed by their respective legislatures.
The RFP and resulting contract should contain workable timelines that allow sufficient time for scoring and quality control. When delays occur, timely communication is vital for resolving the problem expeditiously and dealing effectively with those affected. If bonus or penalty clauses are included in contracts, timelines for agency staff to complete prerequisite tasks should also be specified. States may want to consider contract payment schedules to vendors based upon the delivery of specified products and services rather than on the basis of calendar dates alone.
The majority of state testing programs choose a spring test administration that results in demands on vendors to produce reports for multiple programs during the same narrow time frame at the end of the school year. States able to schedule scoring during nonpeak periods may have greater flexibility in turnaround time and may gain a cost savings. Programs with bonus or penalty contract provisions may likely be given priority in such circumstances (though other considerations are also likely to come into play). The contract should contain the same scoring deadlines contained in the RFP. States may wish to attach to these deadlines specific liquidated damages for each day of non-delivery. In such cases, the contract should include provision for performance bonds against which the agency can claim the damages.
Funding is not a simple issue of obtaining annual appropriations. Activities for any given assessment administration from start to finish require approximately 18 months. This means that the typical fiscal year of 12 months and the assessment "year" of 18 months will conflict unless special provisions are made in the funding. One would not want to be in the position of having to write a contract for the first 12 months of activities and then another contract for the last 6 months of work. Furthermore, there is the likelihood that the fiscal year will not coincide with the RFP/contract/implementation cycle. The solution is to create multiyear funding plans and permit the agency to contract across fiscal years. Contracts can be cancelled if budgets must be reduced or the program is eliminated. Contracts should allow for necessary audits if required by the state comptroller.
When a delay is likely, the vendor should notify agency staff immediately and provide a good faith estimate of its extent.
Immediate notification of the state when a delay is likely is always best practice for the vendor. Quick notification allows all parties involved to assess the scope of the problem, its impact, and any necessary actions.
Generally, use of the data is the responsibility of the state and the LEA (Local educational agency). Some of these activities might be delegated to vendors, however. It is important that the RFP and the resulting contract make it clear what is expected of the vendor. If the state requests changes or delegates additional responsibilities to the vendor after the contract has been signed, the state may have to renegotiate the price.
Clear and understandable reports must be developed for communicating test results to educators, students, parents, and the general public.
Clear communication and guidelines for interpretation are essential to appropriate use of test data. Interpretative guidelines should be reported for both individual and school level reports. Cautions on over-interpretation, such as using tests for diagnostic purposes for which they have not been validated, should be made clear.
The state is responsible for communicating the test results to educators, students, parents, and the general public. An important part of this responsibility is the design of reports of test data. The state might choose to do this itself, or delegate it to the vendor. If the state delegates the design of reports to the vendor, the state shall be responsible for clearly sharing with the vendor its expectations about the audience for the reports, the purpose of the testing program and the uses to which the data will be put. The state shall also make clear, in writing, its requirements for the languages of reports to parents and the community and whether the reports should be graphic, numerical or narrative. The state shall be responsible for approving report formats in a timely manner as described in the contract.
The state is in the best position to determine how the test results will be used and what data will best communicate relevant and important information to the various audiences. It is also the prerogative of the state to determine report formats, types of scores to be reported and appropriate narrative information to accompany each report. Final report formats should be approved by the state before actual reports are printed. The state may also choose to provide access to data on a website designed by the state or its vendor.
If specific responsibility for monitoring the use of the test data is a part of the vendor’s contract, the vendor shall develop detailed policies and procedures for promoting and monitoring the proper interpretation of test data and implement those plans. Regardless of delegation of responsibility in this area, the vendor shall have a system for compiling any information of which it becomes aware regarding the improper and/or incorrect uses of data and relaying that information to the state.
The vendor, just like the state, bears responsibility for supporting and encouraging the ethical and proper implementation of the assessment system. Where the vendor has become aware of inappropriate practices in the course of its work on the assessment system, these should be reported to the state.
The state shall determine how the test data are to be used, and develop detailed policies and procedures for the proper use of the data. The state shall use the resources of the vendor or other qualified individuals (such as the Technical Advisory Committee) as needed to ensure the proper use of the test data for the purposes for which the test is intended, and make all reasonable attempts to prevent the improper use and interpretation of the data.
The only purpose of the testing program is to provide data that meets the goals of the program. Improper interpretation and use of the data negate all of the activities that led to the creation of that data, wasting money and time and perhaps causing serious disservice to students in the state. Since the vendor knows the test well and often has the capabilities to assist in interpretation and dissemination, the state may want to include in the contract the use of the vendor’s resources in conducting workshops around the state for teachers and administrators, joining and assisting the state personnel in presenting the data to stakeholders, such as legislative committees and the press, or assisting in the dissemination of the data. The state should use its greater knowledge of schools and districts in the state and their needs to help the vendor in these functions. The complementary expertise of the vendor and state should be utilized to ensure that the data is use in an appropriate manner.
Jeff C. Palmer is a teacher, success coach, trainer, Certified Master of Web Copywriting and founder of https://Ebookschoice.com. Jeff is a prolific writer, Senior Research Associate and Infopreneur having written many eBooks, articles and special reports.
Source: https://ebookschoice.com/procedures-and-communications-responsive-to-the-needs-and-concerns-of-districts-and-schools/
