Request for Proposal (rfp)



Download 5.9 Mb.
Page10/38
Date09.08.2018
Size5.9 Mb.
TypeRequest
1   ...   6   7   8   9   10   11   12   13   ...   38

3.2.2.2.34 Data Management


Agency requires the vendor’s data management system to interface with the Agency’s data management system. A general information file of individual student performance must be submitted to the Executive Director, Office of Technology, immediately upon completion of scoring not to exceed 3 weeks (for multiple choice tests) or 6 weeks (for multiple choice and constructed response tests).
Agency requires vendors to provide software solutions to the data management and disaggregation of data at the class, grade, school, county and state levels. In addition, disaggregated group reports must be available by the following classifications:

  • Limited English Proficient

  • race/ethnicity

  • gender

  • free/reduced lunch

  • migrant

  • special needs

  • other groups as specified over the life of the contract

School, county, and state level reports to be received by will be provided by the vendor through a secure FTP site in the same form as the specified reports. The vendor will work in conjunction with the Agency to finalize the data and data layout. The Agency will have final approval of variable names and the formatting layout. All optional items for purchase will follow the established format.


Any proprietary software required (along with all software support) to read the data must be included for the Agency and updated throughout the contract. Vendors are to describe this software in full. If additional copies will be required at the county level, pricing for this must be included for the life of the contract.
3.2.2.2.35 Transition Plan

At the conclusion of the program, either through the successful completion of the contract period or through termination, it is expected the vendor will directly and fully participate in the transfer of the program to another vendor. Several steps will be involved, and the vendor of the program associated with this RFP is expected to take a leadership role. Vendors are expected to provide a plan describing, at a minimum:



  • Duration of transition

  • Transition Meetings

  • Team members designated

  • FTE required by person

  • Determine core transition team

  • Establishment of common definitions for terms

  • Documentation of all meetings and agreements in detail

  • Determine contractor responsibility for each aspect of important deliverables

  • Documentation of all transferables as sent to new vendor and new vendor’s inspection of all materials

  • Documentation/identification of all psychometric files and like documents to be transferred

  • Monitoring timely delivery of agreed transferables

  • Plan appropriate replication studies involving Agency Technical Advisory Committee - Equating and handscoring studies are a minimum

  • Transfer of all student essays supplied by West Virginia students.

  • All student essays supplied by the Agency shall be the property of the State of West Virginia

Vendors’ proposals should address completing a full plan for transition prior to the conclusion of 2008. This plan will be updated annually to reflect new information.


Table 21: Transition Meeting Plan


Meeting

Number

Meeting Purpose

1

Discuss Data Collection, Scanning, and Processing

2

Discuss Analysis and Reporting

3

Discuss Master Schedule

4

Discuss Psychometrics, including replication study

5

Discuss Handscoring


3.2.2.2.35.1 Transition Psychometric Issues

Vendors should define in their proposals a set of key psychometric issues to be considered at the time of transition. At an absolute minimum, equating and handscoring replication studies will be included. An exploration should be conducted (and defined here) to determine whether change (positive/negative) in test results is solely related to student performance and not change(s) in testing methodology/material from vendor to vendor. The final transition plan will include a literature review detailing past efforts in transitioning programs with a series of recommendations.


The vendor’s subject area lead scoring director and the grade scoring directors will participate in meetings with up to eight West Virginia educators and a Agency staff member, who will serve as chair, to review scoring criteria and rangefinder papers used to score the tasks during field test scoring. Separate meetings will be conducted for each grade combination. The vendor will prepare necessary materials for the meetings.



Modified Assessment Grades 3-8 and 11

(Contingent upon funding, the Agency reserves the right to activate this section by grade level at any time during the life of the contract)

3.2.2.3 Modified Assessment Grades 3-8 and 11 Introduction

The Modified Assessment will be created for defined special education population in order to meet the requirements of the NCLB legislation. The vendor will work in conjunction with Agency to develop the Modified Assessment forms of WESTEST in grades 3-8 and 11 for Spring 2008 field test and Spring 2009 – 2014 operational forms. Two forms will be created for Mathematics, Reading/Language Arts, Science, and Social Studies at each grade level.


Modified Assessment measures skills that will be associated with West Virginia’s Modified Content Standards in Reading/Language Arts, Mathematics, Science, and Social Studies. The Modified Content Standards are on grade level and have been developed from the general education content standards for each grade, grades 3-8 and 11. These standards were written to be directly aligned and linked to the on-grade 21st Century West Virginia Content Standards and Objectives and are further described via performance level descriptors:

  • Modified Assessment grades 3-8 and 11 tests must have CRT customized Reading/Language Arts, Mathematics and Science multiple choice (MC) and constructed response (CR) items.


3.2.2.3.1 Item/Form Development Introduction

WV requires the vendor provide a detailed plan of how they propose to meet all development requirements. The following sections outline required development tasks. Vendors should elaborate on these where appropriate and provide new ideas regarding the development process when possible.


All elements of the development process including procedures, processes, and products used by the vendor to complete contract work are subject to final approval by Agency. Agency is to participate fully in all form construction. Any items developed specifically for Agency will be owned by the Agency. The vendor will work closely with the Agency and groups of West Virginia educators to complete all work tasks. Throughout the contract period, the vendor will confer with the Agency on a continuing and consistent basis and will be involved in frequent face-to-face meetings with the Agency as needed.

3.2.2.3.1.1 Test Construction/Design Specifics


The specifics for test construction are for the Modified Assessment for grades 3-8 and 11 in Reading/Language Arts, Mathematics, Science, and Social Studies. The vendor and Agency will jointly develop or select all items. Agency will retain all copyrights to the items developed along with the associated graphics and artwork. The specifics for test construction for the Modified Assessment in grades 3-8 and 11 in Reading/Language Arts, Mathematics, Science, and Social Studies must include;

  • Evidence that test forms align to the WESTEST and Modified blueprints (content, standards, objectives, thinking skill levels)

  • Test form format that follows principles of Universal Design for assessments

  • Schedule of second copy reviews

  • Schedule of camera copy reviews

  • Schedule of blueline reviews

  • Description of the process to only allow the use of items within a .2 –.8 p-value range

The alignment evaluation will be determined using the Mathematics and Reading/Language Arts Extended Standards; however, the Social Studies and the Science will be developed and those assessments must be built into the proposals.


3.2.2.3.1.2 Blueprint/Item Specifications

The vendor will work with the Agency to plan a system for accomplishing the task of blueprint and item specification development. The vendor will develop in conjunction with Agency a blueprint (standards, objectives, and thinking skill level) for the Grades 3-8 and 11 Modified Assessment. The vendor must describe their process for developing test blueprints and item specifications. Agency has the final approval on all test blueprints and item specifications.


The proposed blueprint will describe the relative emphasis of the extended standards, objectives, and thinking skill levels to be assessed. The blueprint will include the approximate number (provided below) of items to be included on each form of the assessment. The test design will also describe the measurement model and procedures implemented for Modified Assessment. This blueprint document will form the basis for developing the first draft of the item specifications and the scoring specification documents. The vendor should provide in the proposal their process for creating test blueprints.
Table 22: Grades 3-8 and 11 Field Test Design – 2007-2008






# Required Reading/

Language Arts Items

# Required Mathematics Items

# Required Science Items

# Required Social Studies Items

Grade

MC

CR

MC

CR

MC

CR

MC

CR

Grade 3

47

6

66

6

56

6

56

6

Grade 4

47

74

66

6

56

6

56

6

Grade 5

56

6

66

6

56

6

56

6

Grade 6

47

6

74

6

56

6

56

6

Grade 7

47

6

83

6

56

6

56

6

Grade 8

56

6

66

6

56

6

56

6

Grade 11

47

6

56

6

56

6

56

6



Table 23: Grades 3-8 and 11 Operational Test Design – 2008-2009



# Required Reading/

Language Arts Items

# Required Mathematics Items

# Required Science Items

# Required Social Studies Items

Grade

MC

CR

MC

CR

MC

CR

MC

CR

Grade 3

35

3

47

3

41

3

41

3

Grade 4

35

3

54

3

41

3

41

3

Grade 5

41

3

47

3

41

3

41

3

Grade 6

35

3

54

3

41

3

41

3

Grade 7

35

3

59

3

41

3

41

3

Grade 8

41

3

47

3

41

3

41

3

Grade 11

35

3

41

3

41

3

41

3


3.2.2.3.1.3 Item Specifications

The vendor must provide a detailed description and provide examples of how item specification work was completed in other states or with other contracts. The successful vendor will conduct a review of the Modified Assessment Test Item Specifications, the CSOs, and the distribution of thinking skill levels. The review will be attended by Agency staff and may involve West Virginia educators. The vendor will follow the Agency specifications for all test item development. The Agency will expect the vendor to align items to the final DOK to each standard and objective per grade level and per content. All DOK determinations will use the Webb alignment model and definitions of DOKs. The successful vendors will be provided with the DOK distribution charts for the 21st Century West Virginia Content Standards and Objectives by grade level per content area. The newly developed assessments will require a large number of higher thinking skill levels; therefore, all proposals must demonstrate higher levels of thinking. Based on the review, the vendor will provide and distribute an updated version of the specifications for all grades 3-8 and 11.


It will be the vendor’s responsibility to electronically maintain an updated electronic version of the item specifications to be shared with Agency. These specifications will be used throughout the development cycle by item writers and reviewers and at item review meetings with West Virginia educators and Agency staff.
3.2.2.3.1.4 Item Development/Selection Process

The vendor’s proposal will describe and specify the quality of their development procedures in terms of their alignment to the validity and reliability of testing principles as per the APA Standards. The development procedures must result in items and passages created or selected to generate reliable and valid performance at the four performance levels for Modified Assessment. During development, the consideration of additional technical factors should include, but are not limited to, test bias and fairness of items to all students, readability of test directions and items, and item analysis. (See Sections 3.2.2.3.10 and 3.2.2.3.11 for submittal of test items, passages for the committee’s review)


The vendor shall describe in detail the procedures to be used in detecting item bias and other technical quality principles to be used. The determination of whether or not each field tested item goes into the operational item pool to become eligible for use in operational tests is to be a joint decision between the vendor and the Agency with Agency having the final approval. Agency plans to participate fully in the item selection process and form construction.
The grades 3-8 and 11 Modified Assessment items will measure the knowledge and skills identified in the Modified Academic Achievement Standards for West Virginia Schools. All items and passages must be field tested prior to actual administration.
The test format, directions for administering, and item development shall use the following guidelines:

  • Test booklets are to be formatted in compliance with specifications for large print. Test booklets must be available in Braille.

  • Items must not elicit any information that would be considered in the realm of “values and/or morals” and so the student should not be led to discuss religious or political values or any issue that would invade personal or family privacy.

  • Verbal information in the item should be appropriate in vocabulary and general readability for the grade level tested.

  • The operational exam will be administered during a two-week window. Two forms will be constructed and will be rotated annually. The form not used will be the breach form.

After the field test, the vendor is to review appropriate item statistics with select Agency staff. In the proposal, the vendors will describe, in writing for the Agency and for later committee reference, the type of item statistics to be used in determining appropriateness of student performance data for use in final test forms. In the final Technical Report, the vendor shall provide assurances that students with disabilities, limited English proficiency, and each of the West Virginia NCLB race/ethnic subgroups were appropriately included.


3.2.2.3.2 Universal Design Principles

The implementation of the Universal Design Principles is a critical feature of the NCLB legislation. The vendor must document the procedures used to assure that Universal Design Principles have been used in development of test items and must describe in detail how principles of Universal Design are implemented in the creation of items, forms, and all other student assessment materials. Vendor will provide samples of the Universal Design protocols for committee review.


3.2.2.3.3 Reviews

The vendor shall be responsible for coordinating and conducting the Content Review Committee meetings and the Bias Review Committee meetings in West Virginia. The Face-to-Face Review will be performed at the vendor’s site, or at a location mutually agreeable to both West Virginia and the vendor. The vendor must include a schedule of activities, time lines, anticipated number of days for completion, and number of participants required for conducting these meetings. The vendor shall have appropriate staff on-site to successfully conduct all review meetings. Agency retains the right to involve West Virginia teachers in all reviews.


The vendor will be responsible for all costs and arrangements related to the review meetings for vendor personnel and meeting materials. The Agency will assume the cost for the facility, refreshments, and lunch, as well as travel reimbursements (hotel, mileage, meals) for West Virginia participants. The vendor will submit written protocol of content/bias/face-to-face reviews for the committee’s review.
3.2.2.3.3.1 Content Review Meetings

The vendor will outline a plan for a review of test passages, items, and their scoring rubrics by the West Virginia Content Review Committees. The plan will include content reviews of all items with particular emphasis on the congruency of all items with test specifications, readability requirements, Universal Design, technical quality, content match, and continuity and articulation of skills across the assessed grade levels. Procedures and materials for orienting reviewers must be described. Content Review Committees will be established to ensure the test passages and items accurately reflect item specifications and test blueprints. The vendor shall provide a minimum of one qualified item writer to facilitate the content reviews for each grade/content area. The vendor shall be responsible for preparing all meeting minutes and providing monthly reports for the Agency. All proposals must provide examples of vendor protocols and procedures for committee review.


3.2.2.3.3.2 Bias Reviews

The vendor will be responsible for providing a plan for a bias/sensitivity review of test items. The Bias Review Committee will review all of the items to ensure test items are fair and free of bias for all students, including students who are visually impaired, or deaf, or hard of hearing. The Bias Review Committee will meet prior to field testing. There will be one additional meeting of the Bias Review Committee for the purpose of reviewing reading passages, which takes place prior to initial item development.


The vendor shall provide a minimum of one qualified item writer for each grade/content area to facilitate the Bias reviews. The Agency may conduct a separate Bias/Sensitivity Review, with the results being provided to the vendor. Agency personnel will be in attendance during the review sessions. All proposals must provide examples of vendor protocols and procedures for committee review.
3.2.2.3.3.3 Face-to-Face Review

Following the field test, the vendor will provide a plan for Face-to-Face Item Review meetings with the Agency staff at the vendor’s site, or at a location that is mutually acceptable for each content area. The vendor must have at least one item writer per content area/grade level with related content expertise available to successfully conduct Face-to-Face Review meetings. The Agency expects each vendor to provide recommendations concerning the timeline and number of participants for this activity. At least two weeks prior to the Face-to-Face Item Review, the vendor will provide the Agency a written report including revisions of the items, as well as comments and suggestions on the content and editorial issues, based on the content and equity reviews.


3.2.2.3.4 Form Development Process

3.2.2.3.4.1 Field Test Development

The vendor shall be responsible for developing and producing field test forms, answer documents, manuals, and ancillary materials that resemble, to the extent possible, the operational assessment materials. For the field test, test forms will be formatted to large print specifications; additionally, Braille forms are required for the field test.


It is the responsibility of the vendor to group the newly developed items into item sets to be included in the field test forms. Using items that have been revised, as determined after the Item Review meetings, the vendor will develop a plan that indicates which of the items should be field-tested. Agency will review the vendor’s plan and approve or request revisions. The determination of whether or not each item goes into the item pool for use in operational tests is to be a joint decision between the vendor and the Agency with the Agency having final approval. Agency is committed to reviewing products submitted by the vendor as efficiently as possible.
The vendor will design the field test to provide information to evaluate test items and to evaluate the overall test design. The vendor will describe the characteristics of the field test providing calibration results, scoring procedures, and other data from the 2008 field test. The documentation should also address the reliability of scores and dimensionality of the proposed operational tests, and include multiple methods for estimating the underlying dimensionality of test data as determined in consultation with and approved by the Agency. A written report of the results of this investigation will be presented to the Agency prior to October 2008. The vendor will make recommendations for reliable scaling approaches in the event of appreciable multidimensionality. All technical data requested in this section must be included in the Field Test Technical Report. All proposals must provide examples of vendor protocols and procedures for committee review.
3.2.2.3.4.2 Operational Form Development

The requirements for constructing operational tests include creating test construction specifications for building the forms. Vendors must detail their specifications and form building processes to at a minimum to include alignment to CSOs, p-values and point bi-serial information. In addition, test construction must be supported by sophisticated computer software that will generate test characteristics.


The vendor will design and produce any camera-ready art for the test forms. Two operational test forms measuring the Modified Academic Achievement Standards for West Virginia will be selected (along with ancillaries) and moved through the vendor’s form development process to camera copy for each operational administration.
Agency elects to participate in the Modified Assessment operational forms development. Agency may choose to modify the design of the test books and ancillaries prior to any test administration, within the constraints of the specifications for test books and answer documents provided, and reserves the right to change this configuration, if necessary, through change orders or appropriate contract amendments.
3.2.2.3.5 Ancillary Product Development

The vendor will develop for Agency’s approval and print all publications, materials, and forms in compliance with Agency printing specifications. All print color proposed by the vendor must be read by populations with visual impairments, therefore West Virginia must have final approval on all ink colors to assure this test is easily readable for students with visual impairments. The specifications and quantities for the major products to be printed are found in Tables 24 and 25.


Table 24: Modified Assessment Grades 3-11 West Virginia Materials Production Specifications 2008 Field Test of Modified Items


Item #

Admin.

Grade

Item Description

Color Specs

Packing Quantities

Pages/

Form

Number of Forms

1

Spring 2008 FT

3

Scannable Consumable Test Booklet

2c/2c

10s

100

2

2

Spring 2008 FT

4

Scannable Consumable Test Booklet

2c/2c

10s

100

2

3

Spring 2008 FT

5

Scannable Consumable Test Booklet

2c/2c

10s

100

2

4

Spring 2008 FT

6

Scannable Consumable Test Booklet

2c/2c

10s

100

2

5

Spring 2008 FT

7

Scannable Consumable Test Booklet

2c/2c

10s

100

2

6

Spring 2008 FT

8

Scannable Consumable Test Booklet

2c/2c

10s

100

2

7

Spring 2008 FT

11

Scannable Consumable Test Booklet

2c/2c

10s

100

2

8

Spring 2008 FT

3-11

Braille Tests

1c




As needed

2

9

Spring 2008 FT

3-11

Large Print Tests

2c/2c




As needed

2

10

Spring 2008 FT

3-11

County Test Coordinator’s Manual

2c/1c




50

1

11

Spring 2008 FT

3-11

County Header

3c




2

1

12

Spring 2008 FT

3-1

School Header

3c




2

1

13

Spring 2008 FT

3-11

Answer Book Envelopes










1

15

Spring 2008 FT

3-11

Shipping Labels

Color coded



3c







1

16

Spring 2008 FT

3-11

Materials Checklist

1c




2

1

17

Spring 2008 FT

3-11

Memos







As needed

1

18

Spring 2008-FT

3-11

Manipulatives

4c

25s




1

19

Spring 2008-FT

3-11

Security Checklist

1c




Triplicate Forms






Table 25: Modified Assessment Grades 3-11 West Virginia Materials Production Specifications 2008 Operational Test


Item #

Admin.

Grade

Item Description

Color Specs

Packing Quantities

Pages/
Form

Number of Forms

1

Spring 2009-2014 OP

3

Scannable Consumable Test Booklet

2c/2c

10s

80

2

2

Spring 2009-2014 OP

4

Scannable Consumable Test Booklet

2c/2c

10s

80

2

3

Spring 2009-2014 OP

5

Scannable Consumable Test Booklet

2c/2c

10s

80

2

4

Spring 2009-2014 OP

6

Scannable Consumable Test Booklet

2c/2c

10s

80

2

5

Spring 2009-2014 OP

7

Scannable Consumable Test Booklet

2c/2c

10s

80

2

6

Spring 2009-2014 OP

8

Scannable Consumable Test Booklet

2c/2c

10s

80

2

7

Spring 2009-2014 OP

11

Scannable Consumable Test Booklet

2c/2c

10s

80

2

8

Spring 2009-2014 OP

3-11

Braille Tests

1c




As needed

2

9

Spring 2009-2014 OP

3-11

Large Print Tests

2c/2c




As needed

2

10

Spring 2009 OP

3-11

Test Examiner’s Manual

2c/1c




50

1

11

Spring 2009 OP

3-11

County Test Coordinator’s Manual

2c/1c




50

1

12

Spring 2009-2014 OP

3-11

District Header

3c




2

1

13

Spring 2009-2014 OP

3-11

School Header

3c




2

1

14

Spring 2009-2014 OP

3-11

Answer Book Envelopes










1


Table 25: Modified Assessment Grades 3-11 West Virginia Materials Production Specifications 2008 Operational Test (Continued)

Item #

Admin.

Grade

Item Description

Color Specs

Packing Quantities

Pages/
Form

Number of Forms

15

Spring 2009-2014 OP

3-11

Shipping Labels

3c







1

16

Spring 2009-2014 OP

3-11

Materials Checklist

1c




2

1

17

Spring 2009-2014 OP

3-11

Memos







As needed

1

18

Spring 2009-2014 OP

3-11

Manipulatives

4c

25s







19

Spring 2009-2014

3-11

Security Checklist

1c




Triplicate Forms



The vendor shall also print any additional materials proposed in the implementation of the project, such as transmittal memoranda, labels for packing, and packing lists. The vendor shall be responsible for all aspects of production for publishing printed products, including formatting, graphics, and key entry. For each publication in Tables 24 and 25, the vendor shall submit for approval, printing plans that identify type size and style, ink and paper color, paper quality, and layout. West Virginia requires attractive, high resolution quality printed materials at reasonable cost.


3.2.2.3.6 Art and Production

Interesting, attractive design is required for all test products developed by the vendor. These designs include the organization, format, page layout, and covers required for test books, reports of assessment results, information publications, and other printed materials. The vendor will produce all graphics, charts, and illustrations for the products for which it is responsible. All graphics, charts, and illustrations will be age appropriate and approved by Agency. The vendor must provide examples of test products created for other programs as part of the proposal. Vendor will provide art examples for committee’s review.


3.2.2.3.7 Accommodations

Braille and Large Print booklets and answer documents will be offered as accommodations for students with disabilities. All products created for use by students will have Braille and large print versions at each grade level. Established publishers of Braille approved by the Agency will produce Braille versions of the test books. These documents will be produced so they will be delivered to counties in the same shipment with the regular format versions of the products.


3.2.2.3.7.1 Braille and Large Print

Regular test booklets shall be produced in Braille and Large Print at each grade level. Approximately 10 booklets per grade are required for Braille and 200 booklets may be needed for Large Print. The vendor is responsible for production of camera-ready formats and files to be provided to the Braille subcontractor. Any manipulatives will also be brailled.


The vendor should designate a person in the proposal as knowledgeable with regard to brailling, will give the Agency assurance all tests produced are modified correctly, and will assume final responsibility for the accuracy of the Braille test instruments. The Braille test booklets/answer documents will be made available for review and approval by the Agency prior to reproduction. The Agency may employ the services of Braille proofreaders. In addition, the vendor is responsible for having the Braille materials proofed by an independent party.
Test administrator notes and scripts to accompany Braille test versions will also be developed. Supplemental directions for transferring of responses must be provided as needed to the test administrators. APHB (Refer to http://www.aph.org/) guidelines for brailling need to be honored.
3.2.2.3.8 Content/Form Management System

3.2.2.3.8.1 Introduction

The vendor shall have the ability to acquire and manage items, statistical data, item codes, test booklet formats, and other pertinent information from their own (and subcontractors’) item development, scoring, and psychometric systems. The Agency will have continuous access to these systems on a twenty-four hour seven days per week basis.


The data (the items and related information) will be placed into a secure content management system that the Agency can access using either web-based technology or other methods proposed by the successful vendor. The proposal must contain a detailed plan to continually acquire and inventory all West Virginia content and item/form data across the life of the contract. At an absolute minimum, it will be the responsibility of the vendor to update and maintain the actual data in the management system after each field test, or operational administration. Item information will be updated each time an item is used either as a field test or operational item. If applicable, transferring existing information between vendors’ management systems will be the responsibility of the primary vendor as designated by the Agency. The plan must also provide details of how the items and related material will be converted to the vendor’s content management system, if applicable. Past experience suggests that transfer of items, their accompanying graphics, and their statistical information from one system to another is problematic and time consuming. Historical information will be retained for all items, including any developed prior to this project. Vendors must build this consideration into their proposals.
The vendor shall provide assurances/solutions/examples in the proposal to ensure this level of availability. The resolution images of items, item graphics, and reading passages in the system must be considered. Vendors must document the features and capability of the content/form management system through screen captures or access to demonstration materials. Vendors must provide adequate copies of their quality control documents on the accuracy of the system and the data for the committee’s review. Quality control checks of the data in the content management system are mandatory and vendors must specify the procedures to be utilized for this purpose.
The content management system must be in a form meeting the requirements of this section and shall maintain the ability to use historical information about all items. Items and passages developed and utilized during the course of the project described herein will be added to the system according to the schedule approved by the Agency, and these items will become the property of West Virginia.
3.2.2.3.8.2 Descriptive Information for Content/Form Management System

Descriptive information associated with each item to be included in the Content/Form Management System should include:



  • content area

  • grade

  • reporting category

  • content standard and grade level objective alignment

  • thinking skills level

  • item identification and classification codes

  • item type

  • test forms

  • position in the test book(s)

  • number of answer choices

  • answer key (list of correct responses and scoring rubrics)

  • passage/stimulus name

  • images of items

  • item graphics and other art or stimuli associated with items

  • administration date

  • status (field-test or operational)

  • history of use including test form(s), page, and item number

  • history/tracking of changes/edits made to items during reviews

  • identify the person who requested or authorized changes/edits


3.2.2.3.8.3 Psychometric Information Descriptive Information for Content/Form Management System

Of particular significance is how all of the psychometric data will be collected, analyzed, verified, and stored for future use in building test forms. The content/form management system must be capable of retrieving and utilizing both traditional and IRT item parameter values for use in test item selection and test construction procedures. The plan must include what information will be included, how missing parameters or statistics will be acquired or computed, any transformations needed of parameters using historical data files, and any limiting issues anticipated, along with proposed solutions.


The content management system must be able to incorporate item data, when applicable, must include:

  • maximum item information

  • location of maximum information

  • IRT parameters and statistical values

  • fit index, chi-square values

  • difficulty values

  • classical item analysis for distractors

  • Differential Item Function (DIF) statistics, if appropriate to and depending on population, include contrast values for Mantel-Haenszel comparisons.

Vendor may suggest other psychometric data that need to be collected. Agency requires a Technical Report to include the above mentioned data for the field test as well as for the operational administrations.


3.2.2.3.8.4 Software/Hardware Concerns for Content/Form Management System

Security of the content/form management system is of utmost importance to the Agency. The Agency must have twenty-four hour, seven days per week continuous access to this secure site. All update costs for software/hardware must be defined in the proposal and the costs must be honored throughout the life of the contract. The costs will be evaluated in the cost proposal.


3.2.2.3.8.5 Optional Content/Form Management System Tasks

Vendors may propose additional content/form management system tasks, or activities, if they will substantially improve the results of the project. The additional tasks or activities must be described in detail within this section, yet separated from the required items in the cost proposal.


3.2.2.3.9 Copyright Issues

The item development plan should include a schedule for acquiring copyright permission as needed for new development and uses, as well as for future uses of any existing passages. Vendors should fully describe their experience with obtaining and retaining copyrights, the processes used to perform this, and a description of the specific personnel to be assigned to this task and their experience.


The vendor is responsible for maintaining copyright agreements obtained by any means and for securing agreements with copyright holders for continuing use of passages for a period of 10 years, for a variety of potential purposes as follows:

  • publication in the Modified Assessment Grades 3-8 and 11 tests over the life of the contract

  • publication in any interpretive or public usage products

  • use in interpretive or public usage products in the form of electronic media distributed to counties or other parties

  • use in the form of electronic media for Internet or any future electronic access.

The vendor is liable for assuring all copyright permission acquisitions The vendor’s proposal should explicitly make assurance that the Agency will be held harmless should a vendor fail to acquire copyright permissions.


3.2.2.3.10 Examples of Items

The vendor must provide examples of multiple choice items, constructed response items and any other item formats that the vendor may deem appropriate at a grade level of the vendor’s choice for each content area. All examples must show thinking skill levels and alignment to the Extended Content Standards.


3.2.2.3.11 Examples of Page Layouts

Vendor must provide a two page layout for a single grade of vendor’s choice for each content area. Layout should demonstrate ink coloration, font size, white space, paper quality, passage layout to item, presentation of answer options, art work, method of communicating stop and go on option to students, numeration, and solution for addressing blank pages within test booklet.


3.2.2.3.12 Item Alignment to the Modified Academic Achievement Standards (MAAS) for West Virginia Schools

Items and passages must be high quality and must align with the MAAS Reading/Language Arts, Mathematics, Science and Social Studies. Vendors must submit their processes to ensure that the development of all passages and items will be in direct alignment with MAAS using the Webb Alignment Model. (See Section 3.1.2) The alignment criteria may need to be adjusted with any future state or federal requirements. The Agency will contract with an independent vendor to conduct an external alignment study.


Vendors should be aware that future development schedules might be adjusted, depending on the revision schedule for the MAAS. In this event, the existing blueprints and item specifications will need to be updated to reflect any changes in the standards, skills or the classification system for the objectives.
3.2.2.3.13 Psychometric Research and Technical Services Introduction

Maintaining test validity, reliability, and the equivalence of tests and score scales across years is a fundamental priority for the Modified Assessment Grades 3-8 and 11 program. The vendor will be responsible for psychometric services related to the development of valid, reliable, equitable (by race, ethnicity, gender, and all other applicable criteria), bias-free, and legally defensible assessments.


These services shall include: planning and coordination of data design and analysis; producing descriptive statistics; scoring model; standard setting; providing measurement consultation; and conducting any additional special studies as needed to document the technical adequacy of the tests. Additional studies may be proposed by the vendor for consideration.
The scoring model used for the field test data must be the same as that used for the operational tests. Any other formats suggested by the vendor must meet these same specifications and the vendor must provide the scoring model for the additional item types.
Vendors should note the technical specifications outlined in this section reflect minimum requirements. Vendors must provide a detailed plan of all proposed psychometric research and technical services necessary to deliver quality products. All test design activities must be conducted according to the most recently published version of the Standards for Educational and Psychological Testing (AERA, APA, and NCME). The successful vendor must provide a Technical Report for the field test and each operational test administration. A copy of a technical report should be submitted as per this RFP.
3.2.2.3.13.1 Descriptive Statistics

Following each field test, or operational administration, the vendor will provide the Agency with a detailed item analysis. The vendor will provide classical statistics for all field test items:



  • For multiple-choice items, the analysis will divide the student population into the performance levels to be determined, using total performance on the test, and show the relationship between answer choices and level of performance.

  • For constructed response/performance tasks, the analysis will demonstrate the proportion of students in total test performance category achieving each score point.

  • For any other suggested item format the vendor will provide the required relational data.

For all items, the analysis will indicate the correlation between the item and performance on the relevant subscore category and the total test. The vendor will provide, at a minimum:



  • difficulty estimates, p-values and point biserials for every item

  • alpha reliability estimates for each test form and standard

  • inter-rater agreement indices

Values for items resulting from these (and all other pertinent) analyses will be included in the vendor’s content/form management system. Other descriptive analyses will include, but not be limited to, means and standard deviations for the population and by each demographic subcategory (i.e., gender, ethnicity, disability status, and English-speaking status), item total correlations, and frequency distributions. Vendors are encouraged to propose additional analyses, based on their experience and emerging statistical theory.


3.2.2.3.13.2 Validity

In alignment with the Agency’s requirements for technical excellence, the vendor will be responsible for establishing and documenting various evidences of validity. This will include, but not necessarily be limited to, the following:



  • evidence of content validity: the alignment to WV CSOs, test blueprints, item specifications, and test items (See Section 3.1.2)

  • evidence that test item formats measure the intended content

  • evidence of the interrelationship among standards

  • evidence items were chosen based on test specifications

  • evidence that alternate forms cover the same content

Vendors must provide a plan for demonstrating the validity evidences mentioned above, along with any others, based on their experience and emerging statistical theory.


3.2.2.3.13.3 Reliability

The vendor will be responsible for establishing and documenting methods to collect evidence of the reliability of test scores and if included the scoring of constructed response items. This evidence of test score reliability will include, but not necessarily be limited to:



  • classical measures of reliability and standard errors for totals and subscores

  • evidence of the reliability of the constructed response scores

  • analyses of the reliability of classification decisions in relation to achievement levels

  • inter-rater reliability

  • frequency distributions of student scores on constructed response items

  • internal consistency of total scores

  • reliability and standard errors of group mean scores

  • longitudinal analyses of reliability and standard errors of student group and sub-group mean scores as well as school, county and state levels

Vendors must provide a plan for demonstrating the reliability evidences mentioned above. In addition, the vendor must provide reliability evidence regarding any applicable surveys, technical studies, contrasting groups studies, descriptions of norm groups, test form equating, test level equating, and field tests.


3.2.2.3.13.4 Scale Scores - Calibration Scaling and Equating Procedures

The processing and scoring of the Modified Assessment grades 3-8 and 11 answer documents requires the calibration, scaling, and equating of student responses. Vendors must provide a detailed plan that articulates the scaling procedures used for assessments.


The first operational year, 2008-2009, will serve as the base-year scale for items and scores. The second operational test form for each grade will be equated to the base-year scales.
3.2.2.3.13.4.1 Statistical Software

The vendor may utilize proprietary software for calibration, scaling, and equating, but should provide a fully licensed copy (or a copy of implemented “shelf” software) to the Agency for this confidential review. This software must be transferable to an Agency subcontractor, if required. Vendors must describe hardware prerequisites and the provided training associated with this software within their proposal. In their proposal, vendors are to provide the name, historical usage, and an overview of the software to be used to complete this task. Additionally, vendors are to provide a description of the technical support for the software and transition plans should the software need to be upgraded, or replaced. No cost technical support must be provided for the length of the contract.


3.2.2.3.14 Vertical Scaling

Student achievement on Modified Assessment is reported using scale scores and a vertical scale is required for the Modified Assessment to include all test items. Vendors must create a vertical scale providing for reporting growth continuously from Grades 3-8 and 11. All vendors must include a reliable vertical scale proposal.


The collection of vertical scaling data must embed items as part of the Spring 2008 field test. In addition to this method, vendors may propose alternative strategies, if desired. The vendor will describe the procedures and requirements for conducting vertical scaling prior to the selection of items and construction of forms. Any alternate procedures must be approved by the Agency.
3.2.2.3.15 Standard Setting

Vendors must provide a detailed plan regarding the procedure recommended for the Agency to determine cut scores, as well as cut score ranges for performance standards for each grade level 3-8 and 11. The Agency will have final approval of all cut scores and will have the final approval on the methodology used to derive such scores.


The vendor is responsible for facilitating the Agency’s process to establish achievement level standards in consultation with West Virginia educators and citizens. The performance descriptors define the student performance levels to be determined. With the ultimate goal of “learning for all,” these descriptors allow the teacher, students and parents to judge the level of student proficiency in each 21st Century Learning Standard.
Performance standard setting methods must include both test-based methods and student-based methods and must include the use of impact data. The vendor will be responsible developing these specifications for the achievement level standard setting process for the RFP committee review. The specifications will address the nature of the proficiency level standards, methods for determining the standards, and procedures for validating and analyzing the quality of information reported using the achievement levels. Vendor’s proposed timeline for completing proposed services must respect the Spring 2008 field test administration and the Spring 2009 test operational dates. Standard Setting must be completed in fall of 2008.
The vendor will be responsible for organizing and implementing the standard setting process, based on the achievement level specifications, and for conducting the standard setting meetings (in conjunction with the Agency). The standard setting process will involve meetings of standard setting committees, consisting of West Virginia teachers and instructional leaders.
The vendor will also be responsible for developing a Standard Setting Technical Report that outlines the processes, procedures, materials, etc. used in the standard setting. The Standard Setting Technical Report should include but not limited to:

  • Table of Contents

  • Copyright

  • Executive Summary

  • Standard Setting Overview

  • Standard Setting Agenda

  • Training Materials

  • Standards

  • Test Development

  • Pilot Sample

  • Test Administration

  • Scoring

  • Analyses and Results

  • Summary of Results – Reliability and Validity

  • Median Results and Impact Data

  • Graphical Presentations by Content and Grade

  • Standard Error Tables

  • Interpolated Cut Scores by Grade (Including performance levels)

  • Evidence for Procedural Validity

  • Special Studies

  • References

Samples of Technical Reports should be provided for committee review.


3.2.2.3.16 Statistical Analyses for Special Populations and Other Studies

Vendors may propose other appropriate studies in the event the RFP has not covered every eventuality. In addition, the Agency may, over the life of the contract, request additional studies to meet a particular need, in consultation with the vendor. Vendors must provide examples for committee’s review.


3.2.2.3.17 External Quality Control

The Agency and the vendor shall operate separate quality control operations. In so doing, the Agency may utilize the services of one or more vendors to assist in verification of the quality and accuracy of the statewide assessment results. If implemented, these vendors will work under the direction of the Agency and will perform data verification checks at times and places so designated. The vendor will be obligated to provide data, information, explanations, and workspace, if necessary, for Agency staff and data verification vendors working with the Agency. The objective of the quality control processes is to triangulate analyses and to verify that the data being reported are correct. The Agency will review all quality control findings and will provide timely permission for the vendor to score and distribute test results. Vendor must provide copies of their quality control protocols and procedures for the committee’s review.


3.2.2.3.18 Psychometric Support

A high level of communication is absolutely essential in the area of psychometric support. While not every element of this communication can be defined here, the Agency expects the utmost in consultation and availability of the vendor’s psychometric staff.


The vendor proposal must describe procedures to provide measurement consultation services to Agency staff (and others) and describe procedures for providing measurement consultation in a collaborative manner. These support services may include problem-solving discussions, consultation with outside experts, participation in content/bias reviews, review of field test and operational data, exploratory analyses, unexpected measurement or technical issues or more formal studies. The successful vendor must be prepared to provide psychometric support to address these issues by working closely with the Agency to resolve all issues. Attendance of selected psychometric staff at Agency TAC meetings is mandatory.
Under Section 3.2.4.6.5, vendors should provide strong evidence and identification of experienced key psychometricians, research and measurement specialists, data analysts, legal experts, and other key technical staff assigned to the program.
3.2.2.3.19 Technical Reporting

3.2.2.3.19.1 Analyses Reports

The vendor will be responsible for providing a hard copy and an electronic copy of all data analyses/reports to the Agency. The vendor will also provide the data files and a record layout to the Agency. The electronic format and data layout will be mutually agreed upon by the vendor and the Agency at the first technical meeting.


3.2.2.3.19.2 Final Technical Reports/Documents

The vendor will be responsible for designing, writing, and producing Technical Reports to provide documentation of all technical work associated with each field and operational test. The content of the reports shall include detailed narrative descriptions of content and bias reviews, item review/selection, validity and reliability studies, scaling, and item statistics. These reports will provide sufficient information to allow for an independent evaluation of the quality of the assessments.


Following each field and operational administration, the vendor will produce both draft and final technical documents (incorporating recommended revisions) based on an overall analysis of the administration. The draft technical document will be reviewed by the Agency and by the Technical Advisory Committee (TAC) prior to completion of the final copy of the report. Recommendations and changes will be incorporated into the final draft.
Technical manuals must include the following sections. Others may be proposed, as the vendor deems necessary (and when applicable):

  • Purpose

  • Background

  • Overview of test design

  • Test development procedures employed to construct the test forms

  • Evidence of fairness in development of test

  • Test administration

  • Test scoring

  • Handscoring reliability and validity

  • Online reliability and validity

  • Rater effects

  • Description and analysis of sampling procedure

  • Description and analysis of calibration and equating

  • Model fit; local dependence

  • Scaling and equating procedures

  • Reliability and validity of individual and group scores

  • Reliability of classification decisions, including SEMs

  • Reliability of year-to-year changes in school means

  • Generalizability for all relevant sources, such as variability of groups and internal consistency of item responses

  • Description and analysis of equating procedure

  • Vertical scaling procedure

  • A comparison of the characteristics of the current test administration to previous administrations

  • Sample of Reports

  • Performance Standards

  • Quality Control Procedures

  • Glossary Of Terms

The Agency will work with the vendor to determine the final contents of the Technical Report and any supplements. The vendor will provide ten printed copies of the final report and an electronic version in the format determined by the Agency. The report will include tabular and graphic displays of data to illustrate the characteristics and quality of test scores. Hard copy reports must be professionally bound and labeled.


Agency will work with the vendor on the delivery date for the technical manuals and documents, but in no case shall it be longer than 60 working days from the dissemination of the written student reports to the counties.
The proposal requires the vendor to submit a sample technical document for the committee’s review.
3.2.2.3.20 Materials Production

The vendor shall be responsible for developing and producing all test materials (e.g., scannable consumable test booklets, manuals, etc.) for the administration of the Agency’s annual tests in compliance with Agency printing specifications. Development and production should include, but is not limited to: writing, copy editing, proofreading, graphic design, layout, and print or electronic publishing of documents.


The vendor will also print any additional materials needed to implement the project, such as transmittal memoranda, labels for packing and packing lists. The vendor will be responsible for all aspects of production for publishing printed products, including formatting, graphics, and key entry.

For each publication, the vendor will submit for approval printing plans that identify type size and style, ink and paper color, paper quality, and layout. Printing examples that show type size and style will be included. The Agency desires attractive, high quality printed materials at reasonable cost


See Section 3.2.2.3.5. - Tables 24 and 25 for information regarding additional material specifications. A description of the test documents comprising the field test in Spring 2008 and the first operational test in Spring 2009 is provided. The vendor can expect the configurations in 2010 and beyond will be similar to those shown for 2009, however, the Agency reserves the right to change this configuration, as needed.
3.2.2.3.20.1 Scannable Consumable Test Booklets

Two Modified Assessment forms for grades 3-8 and 11 measuring the MAAS in Reading/Language Arts, Mathematics, Science and Social Studies are to be constructed. The forms will alternate and the form not being used will serve as the breach form. Vendors shall propose a defensible plan for the final test form construction process. This must include a description of the final documents. The Agency may choose to modify the design of the test booklets and answer documents prior to any test administration within the constraints of the materials specifications. A scannable consumable test booklet will be used for the Modified Assessment.


The Agency must receive a minimum of 10 copies of each test form before distribution to local counties. After distribution to counties is completed, all overage is property of Agency. Each year, the vendor will solicit the specific quantities to be shipped to Agency and quantities for disposition. The vendor will store used paper answer documents for the duration of the contract, and used test booklets for one calendar year from the date of retrieval.
The test booklets will have the following general characteristics:

  • Different item types may be interspersed throughout the test books.

  • If required, blocks of embedded field test or anchor items may be located in various locations in test books.

  • Multiple field test forms will be produced and packaged for spiraled distribution in Spring 2008.

  • Tests will include a combination of up to 60 multiple choice and 3-6 constructed response

  • Reading passages, if used, are to be published works for which copyrights are to be obtained.

  • Reading passages may be reproduced with graphics and pictures.

  • Mathematics, Science and Social Studies items may also utilize graphics and pictures. Graphics may also be included in constructed response answer documents.

  • Universal Design Principles should be considered when selecting graphics and pictures.


3.2.2.3.20.2 Custom Covers

Test booklets shall have customized covers to include the Agency logo. The cover must dovetail with a selected color scheme and have a similar look to test material from other WV assessment programs. The vendor may provide options to customize covers for grades 3-8 and 11 for committee’s review.



3.2.2.3.20.3 Production of Scannable Consumable Test Booklets

The Agency requires each test booklet for Modified Assessment grades 3-8 and 11 to be on quality paper, and include certain student information across all grade levels and subject areas. This includes such items as name, identification number, gender, race/ethnicity, date of birth, classification as to exceptional education services or limited English proficiency, primary and secondary disability, etc.
The test booklet shall also include appropriate fields for Agency and county use. Vendors should be aware that, in accordance with policy changes from the federal level, the Agency may alter the current design of the race/ethnicity fields to be in proper compliance with federal guidelines.
The program requires a test booklet for each student attempting the test. The end result will be production of a data file containing all students who were properly assessed and their results and invalid test scores. The Agency and the counties/schools will be provided with a report containing the results for their students. Vendor shall provide copies of answer documents for committee’s review.



Download 5.9 Mb.

Share with your friends:
1   ...   6   7   8   9   10   11   12   13   ...   38




The database is protected by copyright ©sckool.org 2020
send message

    Main page