Descriptive, Narrative, Persuasive, & Informative – Prompts with Passages
TABLE 32: SCORING OF 2007-2008 FIELD TEST ESSAYS
NUMBER OF STUDENTS
NUMBER OF READS/ PROMPT/ESSAY
THINKING SKILL LEVEL
188.8.131.52.1.3 Operational Tests 2008-2009 through 2013-2014
Vendors will be supplying materials to West Virginia’s 55 counties and two special districts, (the School for Deaf and Blind and Institutional Schools), parochial schools, private schools and home schools. At the time of this RFP, approximately 710 schools are participating in West Virginia testing. Enrollment figures in Table 29 reflect projected enrollment figures with a 5% overage for the first operational year. Vendors must provide appropriate quantities of overage for all test materials.
TABLE 33: GRADES 3-11 OPERATIONAL TEST - 2008-2009 THROUGH 2013-2014
THINKING SKILL LEVEL
TABLE 33: GRADES 3-11 OPERATIONAL TEST - 2008-2009 THROUGH 2013-2014 (Continued)
THINKING SKILL LEVEL
184.108.40.206.2 Prompt/Passage/Form Development
WV requires the vendor provide a detailed plan for meeting all Prompt/Passage/Form Development requirements. The following sections outline required development tasks. Vendors must elaborate on these where appropriate and provide and delineate new ideas regarding the development process when possible. All prompts/genres require a Thinking Skill Level of Four (4).
All elements of the development process including procedures, processes, and products used by the vendor to complete contract work are subject to final approval by Agency. Agency is to participate fully in all form construction. Any prompts/passages developed specifically for Agency will be owned by the Agency. The vendor will work closely with Agency and groups of West Virginia educators to complete all work tasks. Throughout the contract period, the vendor will confer with the Agency on a continuing and consistent basis and will be involved in frequent face-to-face meetings with the Agency as needed.
220.127.116.11.2.1 Test Construction Specifics
The specifics for test construction for Writing Assessment grades 3-11 include the following:
evidence that the results of the Writing Assessment are reliable and valid
evidence that test forms align to the test blueprints
schedule of second copy reviews for paper-based administration or screen captures for online administration
schedule of camera copy reviews for paper-based administration or screen captures for online administration
schedule of blueline/final reviews for paper-based administration or screen captures for online administration
description of process to flag prompt/passages based on Differential Item Function (DIF) analysis and what happens to these prompt/passages
If an online option is selected, all materials must go through a similar review process. 18.104.22.168.2.2 Test and Prompt/Passage Specifications
Beginning in 2007, the vendor will work with the Agency to plan a system for accomplishing the task of prompt/passage specification development. Agency has the final approval on all prompt/passage specifications. The vendor will conduct a review of the existing Writing Assessment specifications, the current WV Content Standards and Objectives for Writing, and the required distribution of thinking skills to be provided in the test item selection per grade level. The vendor will follow the Agency specifications for all test prompt development for grades 3-11. The Agency will expect the vendors to align prompts to the final DOK to each standard and objective per grade level. All DOK determinations will use the Webb alignment model and definitions of DOKs. The successful vendors will be provided with the DOK distribution charts for the 21st Century West Virginia Content Standards and Objectives by grade level. The newly developed assessments will require a large number of higher thinking skill levels; therefore, all proposals must demonstrate higher levels of thinking. Based on the review, the vendor will provide and distribute an updated version of the specifications for all grades 3-11.
It will be the vendor’s responsibility to electronically maintain an updated electronic version of the prompt/passage specifications. These specifications will be used throughout the development cycle by writers and reviewers and at prompt/passage review meetings with West Virginia educators and Agency staff.
The test design document will include the approximate number of prompts/passages to be included on each field test form of the assessment. The test design will also describe the measurement model and procedures implemented for scoring Writing Assessment grades 3-11. This document will form the basis for developing the first draft of the prompt/passage specifications and the scoring specification documents.
The review will be attended by Agency staff and may involve West Virginia educators. Based on the review, the vendor will create and distribute specifications for all grades 3-11. It will be the vendor’s responsibility to maintain an updated electronic version of the prompt/passage specifications for use throughout the development cycle.
22.214.171.124.2.3 Prompt/Passage Development/Selection Process
The vendor’s proposal will describe and specify the technical quality of their test development procedures in terms of alignment to the validity and reliability of assessment testing principles as per APA Standards. The test development procedures must result in prompts/passages created to generate reliable and valid performance. During development, consideration of additional technical factors must include, but not be limited to, test bias and fairness of prompts/passages to all students, readability of test directions and prompt/passage analysis.
After the field test, the vendor is to review appropriate prompt/passage statistics with select Agency staff. In the proposal, the vendor will describe the type of prompt/passage statistics to be used in determining appropriateness of student performance data for use in final test forms. In the final Technical Report, the vendor shall provide assurances that students with disabilities, limited English proficiency, and the West Virginia NCLB race/ethnic subgroups were appropriately included.
The vendor shall describe in detail the procedures to be used in detecting prompt/passage bias and other technical quality principles to be used. The determination of whether or not each field tested prompt/passage goes into the pool to become eligible for use on operational tests is to be a joint decision between the vendor and the Agency with Agency having the final approval. Agency plans to participate in the prompt/passage selection process and form construction.
Writing prompts/passages for all grade levels must measure the writing skills listed in the 21st Century West Virginia CSOs. The writing prompts/passages are to be written so that students in the designated grade level mode will respond to descriptive, narrative, Informative or persuasive writing. The prompts will be designed to a Thinking Skill Level of Four (4). All prompts must be field tested prior to actual administration. A total of sixteen (16) prompts will be field tested per grade level except for Grade 3 where eight (8) prompts will be tested, providing adequate prompts for six operational years.
The test format, directions for administering and prompt/passage development shall use the following guidelines:
All test material/answer documents must be formatted in a manner assuring ease of reading. The style of type used shall be large enough to be easily readable and have physical characteristics that research has indicated facilitate reading.
Obtaining permission for using copyrighted reading passages of published works will be the responsibility of the development vendor.
Prompts/passages must not elicit any information that would be considered in the realm of “values and/or morals” and so the student should not be led to discuss religious or political values or any issue that would invade personal or family privacy.
Verbal information in the prompt should be appropriate in vocabulary and general readability for the grade level tested.
Content should focus on students’ everyday or life experiences.
Handwritten, paper-based operational writing tests (Option # 3 or Option # 4 – grades 5, 6, 8 and 9) will be administered on only one day per grade throughout the state and administered in roughly the same time period. A second day will be designated for makeup testing.
If an online option is selected, a two-week testing window will be determined for each grade level for those grades taking the test online.
The test will be untimed; however, students must complete and submit their essays within one sitting. This also applies to online administration.
126.96.36.199.2.4 Universal Design Principles
The implementation of the Universal Design Principles is a critical feature of the NCLB legislation. The vendor must document the procedures used to assure that Universal Design Principles have been used in development of test items and must describe in detail how principles of Universal Design are implemented in the creation of items, forms, and all other student assessment materials. Vendor will provide samples of the Universal Design protocols for committee review.
The vendor shall be responsible for coordinating and conducting the Writing Content Review Committee meetings and the Bias Review Committee meetings to be held in West Virginia. The Face-to-Face Review will be performed at a location mutually agreeable to both West Virginia and the vendor. Plans must include activities, timelines, anticipated number of days for completions, and number of participants required for conducting these meetings. The vendor must submit their protocols for Prompt/Passage Review, Bias Review and Face-to-Face Reviews for committee review. The vendor shall have appropriate staff on site to successfully conduct all reviews. The vendor will be responsible for all costs and arrangements related to the review meetings for vendor personnel and meeting materials. The Agency will assume the cost for the facility, refreshments and meeting materials. The Agency will also assume the cost for the facility, refreshments, and lunch as well as travel reimbursements (hotel, mileage, meals) for participants.
188.8.131.52.2.6 Prompt/Passage Review Meetings
The vendor will outline a plan for a review of prompts/passages and their scoring rubrics by the West Virginia Writing Content Review Committee. The plan will include content reviews of all prompts/passages with particular emphasis on the congruency of prompts/passages with test specifications, readability requirements, technical quality, content match, and continuity and articulation of skills across the assessed grade levels. Procedures and materials for orienting reviewers must be described. Each committee includes approximately five members per grade area. Content Review Committees will be established to ensure the prompts/passages accurately reflect the specifications and test blueprints. The vendor shall provide a minimum of one qualified prompt/passage writer to facilitate the content reviews for each grade. The vendor shall be responsible for preparing all meeting minutes and developing monthly reports for the Agency.
184.108.40.206.2.7 Bias Review
The vendor will be responsible for providing a plan for a bias/sensitivity review of prompts/passages. The Writing Assessment Bias Review Committee will review all the prompts/passages to ensure prompts/passages are fair and free of bias for ALL students. The committee will meet prior to field testing and again for the purpose of reviewing prompts/passages prior to development of the final test forms. All proposals must provide examples of vendor protocols and procedures for committee review.
The vendor shall provide a minimum of one qualified prompt/passage writer to facilitate the bias reviews for each grade. The Agency may conduct a separate bias review, with the results being provided to the vendor. Agency personnel will be in attendance during review sessions.
220.127.116.11.2.8 Face-to-Face Review
The vendor will provide a plan for face-to-face prompt/passage review meetings with the Agency staff following the field test. The vendor must have sufficient prompt/passage writers available to successfully conduct face-to-face review meetings. The review will cover all grade levels. Agency expects each vendor to provide recommendation concerning the timeline and number of participants required for this activity. Two weeks prior to the face-to-face review, the vendor will provide the Agency a written report including revisions of the prompts/passages, as well as comments and suggestions on the content and editorial issues. The vendor shall have a method of electronically tracking all changes.
18.104.22.168.3 Form Development Process
22.214.171.124.3.1 Field Test Development
The vendor shall be responsible for developing and producing field test forms, answer documents, manuals and ancillary materials that resemble the operational assessment materials. Braille and large print versions are required for the field test if paper/pencil tests are done. For online testing, the vendor must propose a method for participation of these students.
The vendor will design the field test to provide information to evaluate test prompts/passages and to evaluate the overall test design. The vendor will describe the characteristics of the field test using calibration results, scoring procedures, and other data from the 2008 field test. The reliability of scores and dimensionality of the proposed operational tests to include multiple methods for estimating the underlying dimensionality of test data should be provided. A written report of the results of this investigation will be presented to the Agency. The vendor will make recommendations for scaling approaches in the event of appreciable multidimensionality. The vendor will communicate frequently with the Agency during the field test period.
It is the responsibility of the vendor to group the newly developed prompts/passages into prompt/passage sets to be included in the field test forms. Using prompts/passages that have been revised, as determined after the review meetings, the vendor will develop a plan that indicates which of the prompts/passages should be field-tested. Agency will review the vendor’s plan and approve or request revisions. The determination of whether or not each prompt/passage goes into the pool for use on operational tests is to be a joint decision between the vendor and the Agency with the Agency having final approval. All technical data requested in this section must be included in the Field Test Technical Report.
All prompts will be field tested in the format in which the student will be tested. In other words, if an online option is selected the prompts will be field tested in an online environment.
126.96.36.199.3.2 Operational Form Development
The requirements for constructing operational tests include creating test form construction specifications. Vendors must detail their specifications/processes for form building. In addition, test construction must be supported by sophisticated computer software that will generate test characteristics.
The vendor will design and produce camera-ready art for the test forms. Operational test forms will be selected (along with ancillaries) and moved through the vendor’s form development process for each operational administration. A modification of this process for online form construction should be provided for committee review. Agency also desires attractive, high quality, user friendly, and easy to navigate online environments with appropriate type size, graphics, and layout.
Agency elects to participate in the WESTEST/Writing Assessment operational forms development. Agency may choose to modify the design of the test answer documents, online testing environment (if applicable) and ancillaries prior to any test administration, within the constraints of the specifications for such materials provided, and reserves the right to change this configuration, if necessary, through change orders or appropriate contract amendments. Throughout the forms development process, the Agency is committed to reviewing products submitted by the vendor as efficiently as possible.
188.8.131.52.4 Ancillary Product Development
The vendor will develop and print all publications, materials and forms in compliance with Agency printing specifications. The specifications and quantities for the major products to be printed are found in Table 35 for Field Test 2008 and Table 36 for Operational Tests 2009 - 2014.
The vendor will also print any additional materials needed to implement the project, such as transmittal memoranda, labels for packing, and packing lists. The vendor will be responsible for all aspects of production for publishing printed products, including formatting, graphics, and key entry. For each publication in Table 35 and Table 36, the vendor will submit for approval, printing plans that identify type size and style, ink and paper color, paper quality, and layout. Agency desires attractive, high quality printed materials of reasonable cost. Agency also desires attractive, high quality, user friendly, and easy to navigate online environments with appropriate type size, graphics, and layout.
184.108.40.206.5 Art and Production
Interesting, attractive and technically defensible design is required for all test products developed by the vendor. These designs include the organization, format, page layout, and covers required for test books, reports of assessment results, information publications and other printed materials. The same quality is expected for online solutions. The vendor will produce all graphics, charts and illustrations for the products for which it is responsible. All graphics, charts and illustrations will be age appropriate and approved by Agency. The vendor must provide examples of test products created for other programs as part of the proposal.
Braille and large print test booklets and answer documents will be offered as accommodations for students with disabilities. All products created for use by students will have Braille and large print versions at each grade level for visually impaired students. Established publishers of Braille and large print materials approved by the Agency will produce the large print and Braille versions of the test books, answer documents, and other documents at the vendor’s expense. These documents will be produced so they will be delivered to districts in the same shipment with the regular format versions of the products.
The vendor will describe and provide appropriate accommodations for students with disabilities when providing documentation for online solutions.
All products created for use by students will have Braille at each grade level. Approximately 10 booklets per grade are required. The vendor is responsible for production of camera-ready formats and files to provide to the Braille subcontractor.
The vendor should designate a person in the proposal as knowledgeable with regard to brailing, will give the Agency assurance all prompts/passages are modified correctly, and will assume final responsibility for the accuracy of Braille test instruments. The Braille tests/answer documents will be made available for review by the Agency prior to reproduction. The Agency reserves the right to employ the services of a Braille proofreader. In addition, the vendor is responsible for having the Braille materials proofed by an independent party.
Test administrator notes and scripts to accompany Braille test versions will also be developed. Supplemental directions for transferring of responses must be provided as needed to the test administrators.
220.127.116.11.6.2 Large Print
Large print tests/answer documents will need to be created. Approximately 35 tests/answer documents and other pertinent material per grade will be required. Student responses can be recorded in the test/answer document; therefore, the test/answer document may be combined into one document. The format will be horizontal or full-view format with at least 18-point font. Black print will be used throughout the test. The closed book size will not be larger than 14" by 17" on approved paper. Reformatting of documents may be necessary to meet these specifications. The binding must allow the test to open flat. The test booklets and answer documents will be made available for review and approval by Agency prior to reproduction.
18.104.22.168.7 Content/Form Management System
The vendor shall have the ability to acquire and manage items, statistical data, item codes, test booklet formats, and other pertinent information from their own (and subcontractors’) item development, scoring, and psychometric systems. The Agency will have continuous access to these banks on a twenty-four hour seven days per week basis. The data (the items and related information) will be placed into a secure content management system that the Agency can access using either web-based technology or other methods proposed by the successful vendor. The proposal must contain a detailed plan to continually acquire and inventory all West Virginia content and item/form data across the life of the contract. At an absolute minimum, it will be the responsibility of the vendor to update and maintain the actual data in the management system after each field test, or operational administration. Item information will be updated each time an item is used either as a field test or operational item. If applicable, transferring existing information between vendors’ management systems will be the responsibility of the primary vendor as designated by the Agency. The plan must also provide details of how the items and related material will be converted to the vendor’s content management system, if applicable. Past experience suggests that transfer of items, their accompanying graphics, and their statistical information from one system to another is problematic and time consuming. Historical information will be retained for all items, including any developed prior to this project. Vendors must consider this in preparing their proposals.
The vendor shall provide assurances/solutions in the proposal to ensure this level of availability. The resolution images of items, item graphics, and reading passages in the system must be considered. Vendors must provide adequate quality controls on the accuracy of the system and the data. Quality control checks of the data in the content management system are mandatory and vendors must specify the procedures to be utilized for this purpose.
The content management system must be in a form meeting the requirements of this section and shall maintain the ability to use historical information about all items. Items and passages developed and utilized during the course of the project described herein will be added to the system according to the schedule approved by the Agency, and these items will become the property of West Virginia.
22.214.171.124.7.2 Descriptive Informationfor Content/Form Management System
Descriptive information associated with each item to be included in the Content/Form Management System should include when applicable:
content standard and grade level objective alignment
thinking skill level
item identification and classification codes
answer key (scoring rubrics)
images of items
item graphics and other art or stimuli associated with items
status (field-test or operational) and
history of use including test form(s), page, and item number
history/tracking of changes/edits made to items during reviews, etc.
identify the person who requested or authorized changes/edits
126.96.36.199.7.3 Psychometric Information for Content/Form Management System
Of particular significance is how all of the psychometric data will be collected, analyzed, verified, and stored for future use in building test forms. The content/form management system must be capable of retrieving and utilizing both traditional and IRT item parameter values for use in test item selection and test construction procedures. The plan must include what information will be included, how missing parameters or statistics will be acquired or computed, any transformations needed of parameters using historical data files, and any limiting issues anticipated, along with proposed solutions.
The content management system must be able to incorporate item data that includes, if applicable:
maximum item information
location of maximum information
IRT parameters and statistical values
fit index, chi-square values
classical item analysis for distractors
Differential Item Function (DIF) statistics, including contrast values for Mantel-Haenszel comparisons
The Agency requires a Technical Report to include the above mentioned data for the field test as well as for the operational administrations.
188.8.131.52.7.4 Software/Hardware Concerns for Content/Form Management System
Security of the content/form management system is of utmost importance to the Agency. The Agency must have twenty-four hour, seven days per week continuous access to this secure site. All update costs for software/hardware must be defined in the proposal and the costs must be honored throughout the life of the contract. The costs will be evaluated in the cost proposal.
The proposal must contain a detailed plan to continually acquire and inventory all West Virginia content and prompt/passage/form data across the life of the contract. At an absolute minimum, it will be the responsibility of the vendor to update and maintain the actual data in the content management system after each field test or operational administration. Prompt/passage information will be updated each time a prompt/passage is used either in a field test or operational form. The system will also track any changes/edits made by review committees. If applicable, transferring existing information between vendors’ content management systems will be the responsibility of the primary vendor as designated by the Agency.
The plan must also provide details of how the prompts and related material will be converted to the vendor’s content management system, if applicable. Past experience suggests that transfer of items, their accompanying graphics, and their statistical information from one system to another is problematic and time consuming. Historical information will be retained for all prompts/passages, including any developed prior to this project. Vendors should consider this in preparing their proposals.
184.108.40.206.7.5 Optional Prompt/Passage Content Management System Tasks
Vendors may propose additional prompt/passage content management system tasks or activities if they will substantially improve the results of the project. The additional tasks or activities must be described in this section, yet separated from the required items in the cost proposal.
220.127.116.11.8 Copyright Issues
The prompt/passage development plan must include a schedule for acquiring copyright permission as needed for new development and uses, as well as for future uses of any existing passages. Vendors should describe in detail their experience with obtaining and retaining copyrights, the processes used to perform this, and a description of the specific personnel to be assigned to this task and their experience.
The vendor is responsible for maintaining copyright agreements obtained by any means and for securing agreements with copyright holders for continuing use of passages for a period of 10 years, for a variety of potential purposes as follows:
publication in the grades 3 – 11 Writing Assessment over the life of the contract
publication in any interpretive or public usage products
use in interpretive or public usage products in the form of electronic media distributed to districts or other parties
use in the form of electronic media for Internet or any future electronic access
The vendor is liable for assuring all copyright permission acquisitions. The vendor’s proposal should explicitly make assurance that the Agency will be held harmless should a vendor fail to acquire copyright permissions.
18.104.22.168.9 Examples of Prompts/Passages
For Options # 1, 3, and 5, vendor must provide at least one independent prompt for each grade level (grades 3-11). For Options # 2, 4, and 6 the vendor must provide one prompt with accompanying reading passage for each grade level (grades 3–11) for committee review.
22.214.171.124.10 Examples of Page Layouts
Vendor must provide one page layout for all grades for each option; screen captures are acceptable for online options.
126.96.36.199.11 Prompt/Passage Alignment to the 21st Century WV CSOs
Writing Assessment prompts/passages will be in alignment with West Virginia 21st Century Reading/Language Arts CSOs. Vendors must submit their processes to ensure the development of all prompts/passages will be in direct alignment with West Virginia’s CSOs.
Vendors should be aware that schedules and future development might be required. In this event, the existing blueprints and prompt/passage specifications will need to be updated to reflect any changes in the skills or the classification system for the objectives.
188.8.131.52.12 Psychometric Research and Technical Services
Maintaining test validity, reliability, and the equivalence of tests and score scales across years is a fundamental priority of the Writing Assessment program. The vendor will be responsible for psychometric services related to the development of valid, reliable, generalizable, equitable (by race, ethnicity, gender, and all other applicable criteria), bias-free, and legally defensible assessments.
These services should include planning and coordination of data design and analysis; producing descriptive statistics; using a Item Response Theory (IRT) model to scale the field test prompts/passages; standard setting; providing measurement consultation; and conducting any additional special studies.
The IRT model used for the field test data must be the same as that used for the operational tests. Vendors should note the technical specifications outlined in this section reflect minimum requirements. Vendors must provide a detailed plan of all proposed psychometric research and technical services necessary to deliver quality products. All test design activities must be conducted according to the most recently published version of the Standards for Educational and Psychological Testing (AERA, APA, and NCME). The successful vendor must provide a Technical Report for the field test and each operational test administration.
184.108.40.206.12.1 Descriptive Statistics
Following each field test or operational administration, the vendor will provide the Agency with a detailed prompt/passage item analysis. The vendor will provide item response theory-based and classical statistics for all field test prompts/passages. The analysis will demonstrate the proportion of students in each total test performance category achieving each score point. For all prompts/passages the analysis will indicate the correlation between the prompt/passage and performance on the relevant subscore category and the total test.
The vendor will provide, at a minimum:
difficulty estimates, p-values, and coefficient alpha reliability for every prompt/item
alpha reliability estimates for each test form and standard
inter-rater agreement indices
differential prompt/item functioning (DIF) statistics (using at least two procedures)
DIF analyses to detect possible prompt/passage bias are to be conducted by Caucasian, African-American, Asian/Pacific and Hispanic racial/ethnic groups, low Socio-economic Status as well as by gender. Values for prompts/items resulting from these (and all other pertinent) analyses will be included in the vendor’s content management system. Changes in DIF values across administrations will be analyzed.
Other descriptive analyses will include, but not be limited to, means and standard deviations for the population and by each demographic subcategory (i.e., gender, ethnicity, disability status, low Socio-economic Status and English-proficiency status), prompt/passage total correlations, and frequency distributions. IRT analyses of field test data will include Item Characteristic Curves (ICC) for all the parameters for each prompt. Vendors are encouraged to propose additional analyses, based on their experience and emerging statistical theory.
In alignment with the Agency’s requirements for technical excellence, the vendor will be responsible for establishing and documenting various evidences of validity. This will include, but not necessarily be limited to, the following:
evidence of content validity: the alignment to WV CSOs, test blueprints, prompt/passage specifications and prompts/passages, item specifications, and test items (See Section 3.1.2)
evidence of the interrelationship among standards
evidence that all operational and alternate forms cover the same content
evidence test prompts/items/passages formats measure the intended content
evidence prompt/passages were chosen based on test specifications
Vendors must provide a plan for demonstrating the validity evidences mentioned above, along with any others, based on their experience and emerging statistical theory.
The vendor will be responsible for establishing and documenting methods to collect evidence of the reliability of test scores. This evidence of test score reliability will include, but not necessarily be limited to the following:
classical and IRT measures of reliability and standard errors for totals and subscores
analysis of model fit
analysis of the reliability of classification decisions in relation to achievement levels
frequency distributions of student scores on prompts/items
internal consistency of total scores
generalizability estimates of standard errors
reliability and standard errors of group mean scores and classifications of students by group
reliability and standard errors of year-to-year changes in school means
Vendors must provide a plan for demonstrating the reliability evidences mentioned above. In addition, the vendor must provide reliability evidence regarding any applicable surveys, technical studies, contrasting groups studies, descriptions of norm groups, test form equating, test level equating and field tests.
220.127.116.11.12.4 Scale Scores – Calibration Scaling and Equating Procedures
The processing and scoring of the Writing Assessment answer documents require the calibration, scaling, and equating of student responses. Vendors must provide a detailed plan that articulates the scaling procedures used for assessments. The vendor must provide the standard setting method to be used and evidence of reliability and validity of that the combine scores. The vendor must also provide a method for combining the Reading/Language Arts score with the Writing Assessment score to provide a combined student score and a score for AYP calculations.
Responses to test prompts/items will be analyzed using a prompt/item response theory (IRT) model. In West Virginia, multiple-choice item/passages are to be calibrated and scaled using the three-parameter, logistic model (3PL); and all constructed responses are to be calibrated and scaled using the two-parameter, partial credit model (2PPC). All prompt/item parameters are to be placed on a common scale. Student total scores are based on prompt/item parameter estimates and are to be obtained using pattern scoring. These analysis procedures will be implemented using a probability sample of West Virginia schools. The vendor is responsible for designing a sampling plan, calibrating the current test, and equating the score scale of the current test to the base year of the assessment. This “calibration sample” will include, at a minimum, 5,000 students per grade appropriately representative according to West Virginia student demographics for NCLB subgroups (i.e. Caucasian, African-American, Asian/Pacific, students with disabilities, and limited English proficiency).
The first operational year, 2009, will serve as the base-year scale for prompt/passage parameters and scores. Future operational test forms will be equated to the base-year scales using the Stocking and Lord Procedure. Braille versions of the Writing Assessment will contain the same prompt/passages as the regular tests and are to be calibrated and scaled as such.
18.104.22.168.12.5 Statistical Software
The vendor may utilize proprietary software for calibration, scaling, and equating, but should provide a fully licensed copy (or a copy of implemented “shelf” software) to the Agency for this confidential review. This software must be transferable to an Agency subcontractor, if required. Vendors must describe hardware prerequisites and the provided training associated with this software within their proposal. In their proposal, vendors are to provide the name, historical usage, and an overview of the software to be used to complete this task. Additionally, vendors are to provide a description of the technical support for the software and transition plans should the software need to be upgraded, or replaced. No cost technical support must be provided for the length of the contract.
22.214.171.124.12.6 Vertical Scaling
Student achievement on the WESTEST and the Writing Assessment will be reported using scale scores and vertical scale scores, among others. Vendors must create a vertical scale for the Writing Assessment that provides for reporting growth continuously from grades 3 to 11 which is combined with the WESTEST scale. The collection of vertical scaling data will embed prompts/passages as part of the field test on the 2008 field test. In addition to this method, vendors may propose alternative strategies, if desired. The vendor will describe, in writing, the procedures and requirements for conducting vertical scaling prior to the selection of prompts/passages and construction of forms. Any alternate procedures must be presented to the Technical Advisory Committee and approved by the Agency.
126.96.36.199.12.7 Standard Setting
Vendors must provide a detailed plan regarding the procedure recommended for the Agency to determine balanced and well-articulated cut scores, as well as cut score ranges for performance standards for each grade level 3-11. The vendor will be responsible to develop specifications for the achievement level standard-setting process. The specifications will address the nature of the proficiency level standards, methods for determining the standards, and procedures for validating and analyzing the quality of information reported using the achievement levels. The Agency will have final approval of all cut scores and will have the final approval on the methodology used to derive such scores. All standard setting procedures must be considered valid and reliable. In addition to describing the general standard-setting strategy, the proposal will describe in detail the procedures, data, and materials to be utilized.
The vendor is responsible for facilitating the Agency’s process to establish achievement level standards in consultation with West Virginia educators and citizens. The WV CSOs have associated performance descriptors that provide the basis for assessing overall student competence of grade level standards. With the ultimate goal of “learning for all,” these descriptors allow the teacher, students and parents to judge the level of student proficiency in each 21st century learning standard. The five West Virginia Performance Levels are defined below:
■ Distinguished: A student at this level has demonstrated exceptional and exemplary performance. The work shows a distinctive and sophisticated application of knowledge and skills that go beyond course, or grade level expectations.
■ Above Mastery: A student at this level has demonstrated competent and proficient performance and exceeds the standard. The work shows a thorough and effective application of knowledge and skills.
■ Mastery: A student at this level has demonstrated fundamental knowledge and skills that meet the standard. The work is accurate, complete, and fulfills all requirements. The work shows solid academic performance at the course, or grade level.
■ Partial Mastery: A student at this level has partially demonstrated fundamental knowledge and skills toward meeting the standard. The work shows basic but inconsistent application of knowledge and skills characterized by errors and/or omissions. Performance needs further development.
■ Novice: A student at this level has not demonstrated the fundamental knowledge and skills needed to meet the standard. Performance at this level is fragmented and/or incomplete and needs considerable development.
Performance standard setting methods must include both test-based methods and student-based methods and must include the use of impact data. The vendor will be responsible for developing specifications for the achievement level standard setting process and utilizing IRT item values in a bookmark-based procedure. The specifications will address the nature of the proficiency level standards, methods for determining the standards, and procedures for validating and analyzing the quality of information reported using the achievement levels. Vendor’s proposed timeline for completing proposed services must respect the Spring 2008 field test administration and the Spring 2009 test operational dates.
The vendor will be responsible for organizing and implementing the standard setting process, based on the achievement level specifications, and for conducting the standard setting meetings (in conjunction with the Agency). The standard setting process will involve meetings of standard setting committees, consisting of West Virginia teachers and instructional leaders.
The vendor will also be responsible for developing a Standard Setting Technical Report that outlines the processes, procedures, materials, etc. used in the standard setting. Tech Report should include but not limited to:
Standard Setting Overview
Standard Setting Agenda
Median Results and Impact Data
Graphical Presentations by Content and Grade
Standard Error Tables
Interpolated Cut Scores by Grade (Including performance levels)
Evidence for Procedural Validity
The vendor will be responsible for providing needed data and materials, for organizing, and implementing the standard-setting process related to Writing Assessment, based on the achievement level specifications and based upon the agreed upon plan.
188.8.131.52.12.8 Statistical Analyses for Special Populations and Other Studies
Vendors may feel free to propose other appropriate studies in the event the RFP has not covered every eventuality. In addition, the Agency may, over the life of the program, request additional studies to meet a particular need, in consultation with the vendor.
184.108.40.206.13 External Quality Control
The Agency and the vendor shall operate separate quality control operations. In so doing, the Agency may utilize the services of one or more vendors to assist in verification of the quality and accuracy of the statewide assessment results. If implemented, these vendors will work under the direction of the Agency and will perform data verification checks at times and places so designated. The vendor will be obligated to provide data, information, explanations, and workspace, if necessary, for Agency staff and data verification vendors working with the Agency. The objective of the quality control processes is to triangulate analyses and verify the data being reported are correct. The Agency will review all quality control findings and will provide timely permission for the vendor to score and distribute test results.
220.127.116.11.14 Psychometric Support
A high level of communication is absolutely essential in the area of psychometric support. While not every element of this communication can be defined here, Agency expects the utmost in consultation and availability of the vendor’s psychometric staff.
This section of the proposal must describe procedures to provide measurement consultation services to Agency staff (and others) and describe procedures for providing measurement consultation in a collaborative manner. These support services may include problem-solving discussions, consultation with outside experts, participation in content/bias reviews, review of field test and operational data, exploratory analyses, unexpected measurement or technical issues or more formal studies. The successful vendor must be prepared to provide psychometric support to address these issues by working closely with the Agency to resolve all issues.
Attendance of selected psychometric staff at Agency TAC meetings is mandatory. Vendors should provide strong evidence and identification of experienced key psychometricians, research and measurement specialists, data analysts, legal experts, and other key technical staff assigned to the program.
18.104.22.168.15 Technical Reporting
22.214.171.124.15.1 Analyses Reports
The vendor will be responsible for providing a hard copy and an electronic copy of all data analyses to the Agency. The vendor will also provide the data files and a record layout to the Agency. The electronic format and data layout will be mutually agreed upon by the vendor and the Agency at the first technical meeting.
126.96.36.199.15.2 Final Reports
The vendor will be responsible for designing, writing, and producing Technical Reports to provide documentation of all technical work associated with each field and operational test. The content of the reports shall include detailed narrative descriptions of content and bias reviews, prompt/passage review and selection, validity and reliability studies, scaling, and prompt/passage statistics. These reports will provide sufficient information to allow for an independent evaluation of the quality of the assessments.
Following each field and operational administration, the vendor will produce both draft and final technical documents (incorporating recommended revisions) based on an overall analysis of the administration. The draft technical document will be reviewed by the Agency and by the Technical Advisory Committee (TAC) prior to completion of the final copy of the report. Recommendations and changes will be incorporated into the final draft. All Writing Assessment technical data will be included in the WESTEST Technical Manual.
Technical manuals must include the following sections. Others may be proposed, as the vendor deems necessary and when applicable:
Overview of test design
Test development procedures employed to construct the test forms
Handscoring reliability and validity
Online reliability and validity
Description and analysis of sampling procedure
Description and analysis of calibration and equating
Model fit; local dependence
Scaling and equating procedures
Reliability and validity of individual and group scores
Reliability of classification decisions, including SEMs
Reliability of year-to-year changes in school means
Generalizability for all relevant sources, such as variability of groups and internal consistency of item responses
Description and analysis of equating procedure
Vertical scaling procedure
A comparison of the characteristics of the current test administration to previous administrations
The Agency will work with the vendor to determine the final contents of the Technical Report and any supplements. The vendor will provide ten printed copies of the final report and an electronic version in the format determined by the Agency. The report will include tabular and graphic displays of data to illustrate the characteristics and quality of test scores. Hard copy reports must be professionally bound and labeled.
Agency will work with the vendor on the delivery date for the technical manuals and documents, but in no case shall it be longer than 60 working days from dissemination of student written reports to the counties.
188.8.131.52.16 Materials Production
The vendor shall be responsible for developing and producing all test materials (e.g., test forms, answer documents, manuals, and header sheets, etc.) for the administration of the annual Writing Assessment in compliance with Agency printing specifications. Development and production must include, but is not limited to writing, copy editing, proofreading, graphic design, layout, and print or electronic publishing of documents.
The vendor will also print any additional materials needed to implement the project, such as transmittal memoranda, labels for packing and packing lists. The vendor will be responsible for all aspects of production for publishing printed products, including formatting, graphics, and key entry.
For each publication, the vendor will submit for approval printing plans that identify type size and style, ink and paper color, paper quality, and layout. Printing examples that show type size and style will be included. The Agency desires attractive, high quality printed materials of reasonable cost.
See Table 35 and Table 36 for information regarding additional material specifications. A description of the test documents comprising the field test of 2008 and operational of 2009 is provided in Table 28 and Table 30. The vendor can expect the configurations in 2010 and beyond will be similar to those shown for 2009, however, the Agency reserves the right to change this configuration, as needed.
184.108.40.206.16.1 Tests/Answer Documents
Writing Assessment Tests/Answer Documents for grades 5, 6, 8 and 9 are to be constructed for each administration if the paper version is selected. Vendors shall propose a defensible plan for the final test form construction process. This must include a description of the final documents. The Agency may choose to modify the design of the tests/answer documents prior to any test administration within the constraints of the materials specifications. The vendor shall submit to Agency in a timely manner the final proofs of the test/answer documents.
The Agency must receive a minimum of 10 copies of each test/answer document before distribution to local counties. After distribution to counties is completed, all overage is property of Agency.
If a paper-based, handwritten response is the method of testing, an answer document will be required. This document will be at the minimum a four-page document, consisting of a machine scannable front cover with appropriate information (e.g., title, name, birth date, grade, logo, date, school name, county name, WVEIS number). Remaining pages will include machine scannable preslugged information, the Writing Prompt/Passage, Directions, a Writing Checklist, and a sufficient number of lined pages for final draft writing. A hand-written response would require an answer document for each student attempting the test. (See Table 29 and Table 30 for number of students/essays and number of reads per essay for field test and operational tests.)
The Agency requires each answer document be on quality paper and include certain student information across all grade levels and subject areas. This includes such items as name, identification number, gender, race/ethnicity, date of birth, classification as to exceptional education services, low Socio-economic Status or limited English proficiency, etc. The answer documents shall also include appropriate fields for Agency and county use. Vendors should be aware that, in accordance with policy changes from the federal level, the Agency may alter the current design of the race/ethnicity fields to be in proper compliance with federal guidelines. The nature of these changes cannot be specified at this time. Online solutions must be able to collect and reproduce the same information.
Tests/answer documents must have customized covers. The cover must dovetail with a selected color scheme and have a similar look to test material from WESTEST and other programs. The vendor may provide options to customized covers for grades 3 - 11. The vendor shall submit to Agency in a timely manner the final proofs of the answer documents.
The tests/answer documents will have the following general characteristics:
Reading passages, if utilized, are published works for which copyrights have been obtained.
Universal Design Principles have been considered when selecting graphics and pictures.