IntelliMetric® Assessment Tools and Online Essay Grading

Download 5.24 Kb.
Size5.24 Kb.
  • IntelliMetric® Assessment Tools and Online Essay Grading
  • IntelliMetric® is an intelligent automated essay scoring system that:
    • Provides immediate holistic and analytic feedback for extended or short answers
    • May be used as an instructional tool or for high stakes assessment
    • Is based on more than a decade of research and development with continual innovations and advancements
    • Was first commercially used in 1998 for use in college course placement assessments
  • What is IntelliMetric®?
  • Who’s Using IntelliMetric®?
  • Association of American Medical Colleges (AAMC) for the writing section of the Medical College Admission Test (MCAT).
  • Graduate Management Admission Council (Analytical Writing Assessment (AWA) portion of the GMAT).
  • The College Board (ACCUPLACER® WritePlacer and SAT Prep).
  • ACT, Inc.  (COMPASS e-Write™, COMPASS ESL, ACT Prep, WorkKeys ®).
  • Medical College Admission Test (MCAT)
  • State Departments of Education and School Districts throughout the country.
  • Creates a unique scoring solution (model) for each prompt
  • Learns the characteristics of the score scale through exposure to examples of previously-scored responses
  • Uses Artificial Intelligence to learn how to score essays
  • Trained with a set of known papers (previously scored by experts)
  • Modeled after traditional expert scoring process
  • How does IntelliMetric® work?
  • Prompt specific training mirroring the human scorer calibration process:
    • Training set of approximately 300 representative papers previously scored by expert scorers
      • Must have definition at tails in order to avoid truncating the scale
      • Must be consistently scored- garbage in, garbage out
      • Roughly normal distributions provide strongest models
    • Validation set kept blind
  • How is IntelliMetric® trained?
    • IntelliMetric® infers rubric and learns the characteristics of the score scale from training set
      • IntelliMetric® is inductive rather than rule based
    • Model is created and used to score validation set
    • Performance data is reviewed (comparing human score and IM score for validation set)
    • Agreement Rates
      • Pearson correlation (measure of linear relationship)
  • Continued……How is IntelliMetric® trained?
  • How is an IntelliMetric® model created?
  • Hundreds of studies provide support that IntelliMetric®:
    • Agrees with expert scoring more often than two skilled human raters will agree with each other.
    • Accurately scores open-ended responses across a variety of grade levels, subject areas and contexts
    • IM is perfectly internally consistent. It will consistently score the same essay the same way every time. This is not true of human scorers.
    • Shows stable results across samples.
  • What does the data say about the quality of the scoring?
  • Uses over 400 semantic, syntactic, and discourse-level features
  • Neurosynthetic approach
  • Learning engine
  • Systemic
  • Inductive
  • Uses Multiple “Judges”
  • Other automated essay scoring systems use a more narrow approach.
  • What makes IntelliMetric® unique?
  • Demo
  • Let’s take a look at how IntelliMetric® works.

Download 5.24 Kb.

Share with your friends:

The database is protected by copyright © 2022
send message

    Main page