Request for Proposal (rfp)



Download 5.9 Mb.
Page17/38
Date09.08.2018
Size5.9 Mb.
TypeRequest
1   ...   13   14   15   16   17   18   19   20   ...   38

3.2.2.6.5.3 Reporting


Any and all reports requested by the paper/pencil exam must be replicated, but not be limited to those requested by the assessment system in general. The basic reports names are (for detailed descriptions see Section 3.2.2.6.11):

  • INDIVIDUAL STUDENT SUMMARY

  • SUMMARY CONTENT

  • ITEM ANALYSIS SUMMARY

  • PERFORMANCE GROUPING SUMMARY

  • STUDENT LONGITUDINAL REPORT

  • STUDENT PORTFOLIO



3.2.2.6.5.4 Disaster Recovery Plan


The vendor shall provide a description for committee review of the plan to backup all systems, applications, and databases routinely. Additionally, the vendor shall detail the plan for data recovery in the event a disaster is declared where the data is maintained and stored. Database transaction logs should be archived and maintained online for 48 hours. Back-ups will be maintained offline for the length of the contract.
3.2.2.6.5.5 FTP Site

Vendors must include provisions for a secure File Transfer Protocol (FTP) site for data, documentation and report transfer from the vendor to the Agency and from the Agency to the vendor. This site must be maintained by the vendor who will ensure its continued availability throughout the life of the program.


3.2.2.6.6 Software/Hardware Security Concerns

Security of the content/form management system is of utmost importance to the Agency. The Agency must have twenty-four hour, seven days per week continuous access to this secure site. The vendor must provide evidence that the system is secure. The system will meet federal compliance regulations for both FERPA (Federal Educational Rights and Privacy Act) and COPPA (Children’s Online Privacy Protection Act of 1998). The Agency requires a system that has a locked-down Desktop that will not allow students to leave and re-enter the program, or go to other Internet sites, programs, or files.


3.2.2.6.7 Accommodations

The online Formative Assessment program must be accessible to visually impaired students and students with disabilities. Vendors should refer to Web Content Accessibility Guidelines http://www.w3.org for specific guidelines and recommendations for making Web content more accessible. “Accessible” means usable to a wide range of people with disabilities, including blindness and low vision, deafness and hearing loss, learning difficulties, cognitive limitations, limited movement, speech difficulties, photosensitivity and combinations of these. The vendor must document in detail, the adjustments that will be made to the online system that will allow equal access to visually impaired students and students with disabilities. The vendor must provide screen captures. Screen resolutions must not compromise the test results. All printable forms and ancillaries must be able to print to hard copy without additional white space between items.


3.2.2.6.7.1 Large Print

The online Formative Assessment program must be accessible to visually impaired students for online administration. The vendor will enable the testing materials to be enlarged online to meet, or exceed, the font requirements suggested by the American Printing House for the Blind (minimum of 18-point type) without distorting, or compromising, the testing materials. The enlarging of the exam will not affect the validity and reliability of testing principles, nor technical factors such as test bias, fairness of items for all students, readability of test directions and test items and item analysis.


In addition, the online program must have the functionality to allow the enlarged-print test to be printed to hard copy if the teacher chooses to exercise the pencil/paper test format. All printable forms and ancillaries must be able to print to hard copy without additional white space between items. The screen resolution must not compromise the test results. The vendor must provide screen captures for review and evaluation for this proposal. Also, an exam hard copy must be printed directly from the online platform and submitted for committee review.
3.2.2.6.8 Optional Content/Form Management System Tasks

Vendors may propose additional content/form management system tasks, or activities, if they will substantially improve the results of the project. The additional tasks or activities must be described in detail within this section, yet separated from the required items in the cost proposal. Any associated costs must be submitted along with the vendor’s proposal.


3.2.2.6.9 Copyright Issues

The item bank system must include a schedule for acquiring copyright permission as needed for new development and uses, as well as for future uses of any existing passages/items. The vendor is liable for assuring all copyright permission acquisitions. The vendor’s proposal should explicitly make assurance that the Agency will be held harmless should a vendor fail to acquire copyright permissions. The Agency maintains the right to view any, or all, copyright permission acquisitions at any time throughout the life of the contract.


3.2.2.6.10 Psychometric Information

Of particular significance is what specific psychometric data will be collected, analyzed, verified, and stored for future use.


The content management system must be able to incorporate item data that may include:

  • difficulty values

  • classical item analysis for distractors

  • Differential Item Function (DIF) statistics, including contrast values for Mantel-Haenszel comparisons.

The vendor may suggest additional data. The vendor must document and explain any suggested psychometric component to be included in the proposal specific psychometric components to be included within the report.


3.2.2.6.10.1. Validity

In alignment with the Agency’s requirements for technical excellence, the vendor will be responsible for establishing and documenting various evidences of validity. The vendor will specify the elements to be included within a Technical Report and will submit to the Agency evidence of item validity based on their experience and emerging statistical theory.


3.2.2.6.10.2 Reliability

The vendor will be responsible for establishing and documenting methods to collect evidence of the reliability of test scores. The vendor will specify the elements to be included within a Technical Report and will submit to the Agency evidence of item reliability.


3.2.2.6.10.3 Performance Levels

Performance levels describe in narrative format how students demonstrate achievement of the WV CSOs. These performance descriptors provide teachers with more information on the knowledge and skills that students have within specific core content areas, and they help to categorize and explain student performance on statewide assessments. West Virginia uses five performance levels:


Distinguished: A student at this level has demonstrated exceptional and exemplary performance. The work shows a distinctive and sophisticated application of knowledge and skills that go beyond course, or grade level expectations.

Above Mastery: A student at this level has demonstrated competent and proficient performance and exceeds the standard. The work shows a thorough and effective application of knowledge and skills.

Mastery: A student at this level has demonstrated fundamental knowledge and skills that meet the standard. The work is accurate, complete, and fulfills all requirements. The work shows solid academic performance at the course, or grade level.

Partial Mastery: A student at this level has partially demonstrated fundamental knowledge and skills toward meeting the standard. The work shows basic but inconsistent application of knowledge and skills characterized by errors and/or omissions. Performance needs further development.

Novice: A student at this level has not demonstrated the fundamental knowledge and skills needed to meet the standard. Performance at this level is fragmented and/or incomplete and needs considerable development.

The vendor will, to the greatest extent possible, customize performance levels on the formative assessment exams to fit the performance levels used by the state of West Virginia for its statewide assessment system. The vendor will provide to the Agency for committee review detailed written information on its system performance levels and how those can be customized to fit West Virginia’s existing performance levels.



3.2.2.6.11 Scoring/Reporting

The Formative Assessment system will have the capability to produce online reports, which can be printed to hard copy, at the Student Level, Teacher/Classroom Level, Grade Level, School Level, County Level, and State Level. All reports will meet federal compliance regulations for both FERPA (Federal Educational Rights and Privacy Act) and COPPA (Children’s Online Privacy Protection Act of 1998). The vendor must provide examples of these online reports for committee review.


Each of these levels must contain, but not be limited to, the following report types:

  • Individual Student Summary

  • Summary Content

  • Item Analysis

  • Performance Grouping Summary

  • Student Longitudinal Report

  • Student Portfolio

The Formative Assessment reports for the program must include, but are not limited to:



1. Individual Student Summary:

    • The Individual Student Summary will include the following:

      • Student name (first name, middle initial, last name), grade, date of birth, WVEIS number, school, county, test date, purpose of assessment

      • Reporting (online and with hard copy) of overall score, performance level, mastery/non-mastery of content objectives for the individual student

      • Percent correct for every WV CSO tested

      • Thinking skill levels

For the individual student test taker, s/he will receive a copy of this report information. The classroom teacher will also be able to print this report for review. In addition, the student will be able to view each test item answered incorrectly along with its correct response and accompanying WV CSO. The student will then be provided with a brief instructional activity aligned to that specific CSO. This online instruction will not replace in-class teacher instruction, but simply re-teach the information in a alternative way.


2. Summary Content:

    • The Summary Content Report will include the following:

      • Student names (first name, middle initial, last name), grade, dates of birth, WVEIS numbers, school, county, test date, purpose of assessment

      • Reporting (online and with hard copy) of each student’s (within a specific class) percent correct for each item tested

      • Each student’s points possible, correct answers, points obtained, responses to items, performance levels, and thinking skill levels

      • Information on skills and needs (tied to WV CSOs) of individual students

      • Disaggregating of test data by demographic categories defined within NCLB


3. Item Analysis Summary by Subgroup

    • The Item Analysis Summary will include the following:

      • Content area, item number, and CSO number and description

      • Number of students within the group

      • The percentage of students who responded to each item and the percentage of students who selected each response

      • The percentage of students with each item correct, percentage of students with each item incorrect, percentage of students with each score point and the number of points possible

      • Results (within group) disaggregated by demographic categories defined within NCLB

      • A method of identifying skills and strengths (tied to WV CSOs) by student group

      • Thinking skill levels


4. Performance Grouping Summary

    • The Performance Grouping Summary will include the following:

      • School, county, grade, test date, number of students

      • Content area, item number, CSO number and description, thinking skill levels

      • Grouping of students into performance ranges for each CSO tested

      • Specific instructional/intervention strategies to be presented to groups of students exhibiting weakness, or need, in a specific CSO(s)


5. Student Longitudinal Report

    • The Student Longitudinal Report will include the following:

      • Student name (first name, middle initial, last name), grade, date of birth, WVEIS number, school, county, test date, purpose of assessment

      • Criterion-referenced test results reported by points obtained compared with possible points

      • Report of and graphing of percentage results

      • Student’s relative strengths and needs for improvement (tied to WV CSOs) using raw scores.

      • Evidence of growth in knowledge of CSOs


6. Student Portfolio

    • The Student Portfolio will include the following:

      • Student name (first name, middle initial, last name), grade, date of birth, WVEIS number, school, and county

      • History of all test results for this specific student within a selected date range

      • Date of test administration, school and/or class in which the student was enrolled at time of test, and test name and/or form

      • Scores reported by content area and specific CSOs with percent points obtained, points obtained over points possible, and scale scores (if available)

      • Thinking skill levels

      • Indication of student improvement over time

The vendor must provide evidence that the online platform is capable of:



  • providing student longitudinal tracking reports at all levels

  • providing immediate scoring/reporting of student results

  • providing reports that aggregate scores for federally designated AYP subgroups

  • providing secure web-based Student/School/County/State Reports

  • providing an item analysis report for each test generated

  • providing a tracking tool for the data reports in a longitudinal year-to-year manner

In order to assure that the vendor’s delivery system is reliable, the vendor, in conjunction with the Agency, will conduct a pilot test in a variety of schools in approximately 15 counties. The vendor will assure reliable delivery of the operational test. The vendor will provide documentation that the platform has built-in features to prevent the loss of student assessment data both during administration and during scoring/reporting.


3.2.2.6.12 References

Black, P. & Wiliam D. (October 1998). Inside the black box. Phi Delta Kappan, V80, 2, p. 139—148.


Boud, D. (2000). Sustainable assessment: Rethinking assessment for the learning society. Studies in Continuing Education, V22, 2, p. 151—167.
Pine, S., Church, A., Gialluca, K., & Weiss, D. (1979). Effects of computerized adaptive testing on black & white students. (Research Report 79—2). Minneapolis, University of Minnesota, Department of Psychology, Psychometric Methods Program.



College Predictive/

Entrance Exams

(Contingent upon funding, the Agency reserves the right to activate this section as tests by grade levels at any time during the life of the contract)

3.2.3 College Predictive/Entrance Exams

3.2.3.1 Vendor Requirements

WV requires the vendor provide a detailed plan of how they propose to meet all requirements listed in this section. The following sections outline required tasks and services. Vendors should elaborate on these where appropriate and provide examples/samples/protocols/procedures regarding the section requirements.



3.2.3.1.1 Test Specifics


The Agency requires a “off-the-shelf” testing program with the following components. A college predictive exam for grade 8 and grade 10 and a college entrance exam component for grade 11 with a make-up or re-test for eligible students in grade 12. The college predictive assessments in grades 8 and 10 must at a minimum collect individual student educational and career plans, determine career interests, student aptitudes, and provide reliable predictions for how a student should perform on the college entrance exams.
The 8th and 10th grade results and predictions must be linked activities and must link to the college entrance exam results. The vendor needs to provide evidence of these linkages and how the predictions are used to reliably project a score linked to the proposed 11th grade college admissions tests.
The vendor must provide evidence that the results of the college entrance exam component for grade 11 with a make-up/re-test in grade 12 are currently being accepted and will continue to be accepted by colleges and universities for admission in West Virginia and elsewhere.
The college predictive exams for grades 8 and 10 will be administered in the fall. The college entrance test date is yet to be determined. The make-up/retest option for 12th grade will be a fall national test date during the first semester/block.
3.2.3.1.2 Agency Participation

All elements of the project including procedures, processes, and products used by the vendor to complete contract work are subject to final approval by Agency. The vendor will work closely with Agency and groups of West Virginia educators to complete all work tasks. Throughout the contract period, the vendor will confer with the Agency on a continuing and consistent basis and will be involved in frequent face-to-face meetings with the Agency as needed.


3.2.3.1.3 Vendor Item Development Process

The vendor will provide the test documents, to be held confidential, to the Agency for review of content alignment, quality of item construction/answer selection, thinking skill levels, page layout, art work, and Universal Design Principles for all content areas. The requested information will be made available for committee review for all review participants who sign a security agreement with the Agency. The vendor’s proposal will describe, specify, and provide documentation of the technical quality of their test development procedures in terms of their alignment to validity and reliability of testing principles. The test development procedures must result in passages and items created to generate reliable and valid performance. The consideration of additional technical factors must include, but are not limited to, test bias and fairness of items to all students, readability of test directions and items, item analysis, and inclusion of higher order thinking skills. The vendor must provide examples of their technical quality via hard copy for committee review.


In the proposal, the vendor will provide documentation of the type of item/prompt statistics to be used in determining appropriateness of student performance data used in final test forms. The vendor shall describe and provide in detail the procedures used in detecting item/prompt bias and other technical quality principles.
The test format, directions for administering, and item development should follow these guidelines:

  • The time limits will be reflected in directions read orally to students by the test administrator. The directions will indicate to students the testing schedule.

  • Items must not elicit any information that would be considered in the realm of “values and/or morals” and so the student should not be led to discuss religious or political values or any issue that would invade personal or family privacy. Verbal information in the item/prompt should be appropriate in vocabulary and general readability for the grade level tested.

  • The college reportable entrance exam will be administered on only one day throughout the state and administered in roughly the same time period on that day.

  • All test booklets/materials shall be formatted in a manner assuring ease of readability and have physical characteristics that research has indicated to facilitate reading.

  • Vendor must provide all test booklets in Braille and large print formats for students who notify them of the need.


3.2.3.1.3.1 Universal Design Principles

The implementation of the Universal Design Principles is a critical feature of the NCLB legislation. The vendor must document the procedures used to assure that Universal Design Principles have been used in development of test items and must describe in detail how principles of Universal Design are implemented in the creation of items, forms, and all other student assessment materials. Vendor will provide samples of the Universal Design protocols for committee review.


3.2.3.1.4 Bias Reviews

The vendor will be responsible for providing tests that are fair and free of bias for all students, including students who are visually impaired or deaf or hard of hearing. The vendor must provide examples of the company’s protocols or procedures for conducting Bias Reviews.


3.2.3.1.5 Test Development Process

3.2.3.1.5.1 Field Test Results

The vendor will describe the characteristics of field tests providing calibration results, scoring procedures, and other data from previous administrations. The documentation should also address the reliability of scores and dimensionality of the proposed operational tests.


3.2.3.1.5.2 Operational Form Development

The requirements for constructing operational tests include creating test construction specifications for building the forms. Vendors must detail their specifications and form building processes.


The vendor shall be responsible for developing and producing test forms, answer documents, manuals, and ancillary materials for the operational assessments. Braille and large print versions are required. The vendor must provide examples of sample test forms, answer documents, manuals and ancillary material along with any other protocols or procedures they wish to be evaluated by committee.
3.2.3.1.6 Ancillary Product Development

The vendor will develop and print all publications, materials, and forms in compliance with Agency printing specifications for all materials that are not part of the standard materials used for the “off-the-shelf” assessments for grades 8, 10, 11 and 12. All ancillary product development is subject to Agency final approval.


The vendor shall also print any additional materials needed to implement the project, such as transmittal memoranda, labels for packing, and packing lists. The vendor shall be responsible for all aspects of production for publishing printed products, including formatting, graphics, and key entry. For any customized publications, the vendor shall submit for approval, printing plans that identify type size and style, ink and paper color, paper quality, and layout. Agency desires attractive, high resolution quality printed materials.
3.2.3.1.7 Art and Production

Interesting design is required for all test products developed by the vendor. These designs include the organization, format, page layout, and covers required for te st books, reports of assessment results, information publications, and other printed materials. The vendor must provide examples of test products created for other programs as part of the proposal. All information will be held confidential.


3.2.3.1.8 Accommodations

Braille and large print test booklets and answer documents will be offered as accommodations for identified students with disabilities. All products created for use by students with visual impairments will have Braille and large print versions for both the college predictive test and for the college entrance exam. Established publishers of Braille and large print materials must produce the large print and Braille versions of the test books, answer documents, and other documents at the vendor’s expense. The large print materials will meet or exceed the font requirements suggested by the American Printing House for the Blind. All other vendor accommodations must be provided in hard copy for committee review.


3.2.3.1.8.1 Braille

Regular test booklets shall be produced in Braille at each grade level. Approximately 10 booklets per grade are required. The vendor is responsible for production of camera-ready formats and files to be provided to the Braille subcontractor. Any manipulatives will also be brailled.


The vendor should designate a person in the proposal as knowledgeable with regard to brailling, who will give the Agency assurance all tests produced are modified correctly, and will assume final responsibility for the accuracy of the Braille test instruments.
Test administrator notes and scripts to accompany Braille test versions must be available. Supplemental directions for transferring of responses must be provided as needed to the test administrators. APHB guidelines (Refer to http://www.aph.org/) for brailling need to be honored. The vendor must provide examples of Braille and large print tests for committee review.
3.2.3.1.8.2 Large Print

Large print test booklets in 18 font and answer documents will be needed. Approximately 35 test booklets and other pertinent material per exam will be required. The test booklets and answer documents will be made available for review and approval by Agency prior to administration.


3.2.3.1.9 Psychometric Information

Of particular significance is how all of the psychometric data will be collected, analyzed, verified, and stored for future use. The vendor must identify what information will be included, how missing parameters or statistics will be acquired or computed, any transformations needed of parameters using historical data files, and any limiting issues anticipated, along with proposed solutions.


The vendor will conduct and provide results from a comparability study of WESTEST 8, 10, and 11 and the grades 8 and 10 college predictive exam benchmarks and the grade 11 college entrance exam benchmarks. A Technical Report of the results of this study will be bound and presented to the Agency.
3.2.3.1.10 Copyright Issues

The vendor is responsible for maintaining copyright agreements obtained by any means and for securing agreements with copyright holders over the life of the contract, for a variety of potential purposes including the following:



  • publication in the grades 8, 10, 11, and 12 tests,

  • publication in interpretive or public usage products in the form of electronic media,

  • use in interpretive or public usage products in the form of electronic media distributed to districts or other parties, and

  • use in the form of electronic media for Internet or any future electronic access.

West Virginia will be held harmless for any unsecured copyrighted material in this proposal and/or in future deliverables of the vendor.
3.2.3.1.11 Examples of Tests

The Vendors must provide examples of multiple choice tests for each content area for each of the proposed grade level tests. The vendor must provide an example of one prompt/passage for the college entrance exam writing component.


3.2.3.1.12 Examples of Page Layouts

The Vendors must provide page layouts for each grade level tests for each content area of vendor’s choice.


3.2.3.2 Item Alignment to the CSOs

The vendor will provide a comparison of the exams with the corresponding content of the 21st Century West Virginia Content Standards and Objectives (CSOs).




  • The West Virginia Higher Education Policy Commission must recognize the college entrance exam score and subscores as acceptable college admission scores for all West Virginia colleges and universities.

  • The college predictive scores for Grades 8 and 10 must provide score predictions for the college entrance exam component for grade 11 and the make-up and optional re-test for eligible students in grade12.


3.2.3.3 Psychometric Research and Technical Services Introduction

Maintaining test validity, reliability, and the equivalence of tests and score scales across years is a fundamental priority. The vendor will be responsible for psychometric services related to the development of valid, reliable, generalizable, equitable (by race, ethnicity, gender, and all other applicable criteria), bias-free, and legally defensible assessments.


All test design activities must provide evidence that the activities were conducted according to acceptable industry standards. The successful vendor must provide copies of their technical report for each operational test administration.
3.2.3.4 Psychometric Support

A high level of communication is absolutely essential in the area of psychometric support. While not every element of this communication can be defined here, the Agency expects the utmost in consultation and availability of the vendor’s psychometric staff.


This section of the proposal must specify plans for the vendor to describe procedures to provide measurement consultation services to Agency staff (and others) and describe procedures for providing measurement consultation in a collaborative manner. These support services may include problem-solving discussions, consultation with outside experts, participation in content reviews, review of field test and operational data, exploratory analyses, or more formal studies. Key measurement and other technical issues are anticipated to arise. The successful vendor must be prepared to provide psychometric support to address these issues by working closely with the Agency to resolve all issues. Attendance of selected psychometric staff at Agency Technical Advisory Committee (TAC) meetings is mandatory.
Under Section 3.2.4.6.5 Other Staff Members, vendors should provide strong evidence and identification of experienced key psychometricians, research and measurement specialists, data analysts, legal experts, and other key technical staff assigned to the program.
3.2.3.5 Analyses Reports

The vendor will be responsible for providing a hard copy and an electronic copy of all data analyses to the Agency. The vendor will also provide the data files and a record layout to the Agency. The electronic format and data layout will be mutually agreed upon by the vendor and the Agency at the first technical meeting.


3.2.3.6 Materials Production

The vendor shall be responsible for developing and producing all test materials (e.g., test forms, answer documents, manuals, and header sheets, etc.) for the administration of the Grades 8 and 10 college predictive assessments and the Grades 11 and 12 college entrance exam. Development and production must include, but is not limited to writing, copyediting, proofreading, graphic design, layout, and print or electronic publishing of documents.


The vendor will also print any additional materials needed to implement the project, such as transmittal memoranda, labels for packing and packing lists. The vendor will be responsible for all aspects of production for publishing printed products, including formatting, graphics, and key entry.
For each customized publication, the vendor will submit for approval printing plans that identify type size and style, ink and paper color, paper quality, and layout. Printing examples that show type size and style will be included. The vendor should provide examples of print materials for committee review.
3.2.3.6.1 Test Booklets

Vendors shall include a description of the shelf test booklets and answer documents.


The vendor will store used paper answer documents for the duration of the contract and used test booklets as per their existing procedures and timelines. The vendor will outline these procedures and timelines for the Agency.
The test booklets have the following general characteristics:

  • Different item types may be interspersed throughout the test books.

  • Tests for grades 11 and 12 will include a college entrance exam writing component currently used by the vendor that is recognized by the West Virginia Higher Education Policy Commission as acceptable for partially meeting the admission requirements for West Virginia colleges and universities.

  • Test booklets for grades 8 and 10 college predictive exams must be reusable over multiple years.


3.2.3.6.1.1 Test Booklet Covers

The Agency recognizes that the assessments for grades 8, 10, 11, and 12 are “off-the-shelf” products and thus the test booklet covers will be designed by the vendor.



3.2.3.6.2 Answer Documents

Answer documents will not be customized for the college predictive assessments in grades 8 and 10 or for the college entrance exams in grades 11 and 12. Agency will use the “off-the-shelf” answer documents developed by the vendor.
The scannable answer document design may vary according to exam. The Agency requires each answer document to be on quality paper, and include certain student information across all grade levels and subject areas. This includes such items as name, WVEIS identification number, gender, race/ethnicity, date of birth, classification as to exceptional education services or limited English proficiency, etc.



Download 5.9 Mb.

Share with your friends:
1   ...   13   14   15   16   17   18   19   20   ...   38




The database is protected by copyright ©sckool.org 2020
send message

    Main page