On-Line Student Assessment Richard Hill Center for Assessment Nov. 5, 2001 Speaking Points



Download 7.92 Kb.
Date07.01.2017
Size7.92 Kb.
#6260

On-Line Student Assessment

  • Richard Hill
  • Center for Assessment
  • Nov. 5, 2001

Speaking Points

  • Current paper-and-pencil-based assessments
  • Image Scoring
  • Computer Administration
  • Computer Scoring

Typical Current Paper-and-Pencil Based Statewide Assessment

  • 3 grades
  • Reading, writing, math, science, social studies
  • 30 MC and 6 OE questions for four areas, one essay for writing
  • 50,000 students per grade

Materials Processed

  • 150,000 28-page test booklets
    • 2 millions sheets of paper
    • 10 tons of paper, a stack 700 feet high
  • 150,000 20-page answer documents
    • 1.5 million sheets of special paper
    • 7.5 tons
    • 600 boxes to store (per year)

Process

  • Materials shipped to schools
  • Materials shipped back to contractor
  • Materials logged in

Process for Receiving Materials

  • Separate answer booklets from test booklets
    • Test booklets placed in temporary storage in original boxes, then destroyed after reporting complete
    • Answer sheets guillotined
      • MC answer sheets scanned
      • OE sheets packaged by scoring

Processing of OE Sheets

  • Separate by content area
  • Sorted by form, randomized across schools
  • Scanned to capture ID numbers
  • Scoring headers prepared, then merged with answer sheets

Scoring

  • Hire, train, qualify
  • Score
  • On-going evaluation of quality of scoring
  • Determine papers that need adjudication, then rescore as necessary
  • Scan scoring headers
  • Merge MC, OE and writing scores

Scoring Time

  • 20 seconds per OE question
  • 5 minutes per essay (2 scorings plus adjudication, if necessary)
  • 13 minutes per student
    • 32,500 hours
    • 1000 person-weeks, plus training, qualifying, quality control and equating

Equating to Previous Year

  • MC
  • OE
    • Difficulty of items
    • Changes in scoring

Count, Count, Count

  • Initial log-in counts
  • After packaging
  • Every time a box is opened or closed
  • Count boxes, too

Final Steps

  • Ship reports back to schools
  • Resolve problems
    • Missing or misplaced students
    • Challenges to scoring (requires finding answer sheets—perhaps all for one student)
  • Destroy test materials
  • Long-term storage for answer documents

Solution # 1—Image Scoring

  • High-speed scanners capture images of documents
  • All processing is done on CRTs by looking at electronic image of original paper

Advantages

  • Control
    • Scoring
      • Blind read-behinds
      • Real-time tracking of accuracy of every scorer
      • Multiple sites
    • Equating
      • Blind rescores from previous year

Advantages (cont’d)

  • Scoring speed
    • Next response is ready to be scored when first is done
    • Scoring stops when rates decline
    • No fumbling for papers
    • Up to 1/3 faster

Advantages (cont’d)

  • Tracking
    • No need for counting
    • Nothing is lost
    • Nothing is damaged
    • Records automatically linked
    • Special-request papers easy to obtain
      • Prep for next year’s scoring
      • Challenged papers
      • Adjudication

Advantages (cont’d)

Disadvantages

  • Hardware and software costs
    • Costs have dropped dramatically ($150,000 server two years ago now selling for $16,000)
  • Need to prove that scoring is the same
    • Writing vs. OE
  • Connectivity
  • Power outages

Computer Administered Tests

  • Web-based vs. CD
  • Comparability
    • Standards—especially writing
      • Students that write on paper and then just type in
    • Full use of computer capabilities
    • Underestimation of (some) students’ abilities

Georgia’s Proposed System

  • Huge item bank, three levels
  • Teachers can create tests
  • Capacity concerns for Level III tests

Advantages

  • Elimination of paper
  • Accommodations
  • Adaptive testing
    • Shorter tests
    • Diagnostic tests
    • Lower frustation levels
  • Real-time scoring

Issues

  • Administration time
    • All schools have some computers, but how many?
  • Transition
    • Recommendation is to test all schools the same way
      • Comparability
      • Logistics of operating two programs at same time

Computer Scoring

  • Major vendors
    • NCME Session N1, April 12, 2001
    • ETS Technologies—E-rater (Princeton, NJ)
    • Vantage Learning—Intellimetric (Yardley, PA)
    • TruJudge—Project Essay Grade (PEG) (Purdue)
    • Knowledge Analysis Technologies—Intelligent Essay Assessor (Boulder, CO)

Advantages

  • Time
  • Cost
  • Objective (or at least impersonal)

Issues

  • Accuracy rates
    • PA study—computers vs. humans
      • Computer more accurate than one human
      • Computer less accurate than two humans
      • Bias vs. random error
  • Beating the system (“Stakes changes everything”)
  • Capacity of contractors to deliver logistics

Alternate Testing Modes

  • Listening
  • Special education adaptations—see Tindel
  • Virtual reality


Download 7.92 Kb.

Share with your friends:




The database is protected by copyright ©sckool.org 2023
send message

    Main page