Discuss the advantages and disadvantages of multiple choice questions in student-evaluation process



Download 22.96 Kb.
Date08.05.2017
Size22.96 Kb.
  • Rita John, DNP, EdD, CPNP, PMHS

Discuss the advantages and disadvantages of multiple choice questions in student-evaluation process.

  • Discuss the advantages and disadvantages of multiple choice questions in student-evaluation process.
  • Develop a knowledge of test construction procedures for multiple choice tests
  • Integrate principles of multiple choice construction by evaluating improving selected questions.

Versatile

  • Versatile
    • Easy to mark and score
    • Can be set at different cognitive levels
  • Comprehend concepts
    • Can help you understand whether student mastered a large amount of material
  • Statistical information on performance can be obtained
  • Better coverage of content
  • Instructor sets the agenda
  • Student cannot focus on superficial aspects (essay questions)

Construction is difficult

  • Construction is difficult
  • Recall questions are easier to write and favored
  • Creativity cannot be tested unlike essay questions
  • Does not test originality or the student’s thought processes
  • Can allow student to guess
  • Wording ambiguity

Decide on what you are trying to test

  • Decide on what you are trying to test
    • Level 1: memory
      • Recognition or recall
    • Level 2: Higher order
      • Does the question ask the student for higher level objectives—
        • Comprehension and application
        • Problem solving

How many calories/oz. is in standard baby formula?

  • How many calories/oz. is in standard baby formula?
  • 20
  • 28
  • 30
  • 32
  • How can we turn this into a higher level question and still test the concept?

Objective: To interpret a p value

  • Objective: To interpret a p value
  • A study of 281obese 8 year old showed that after a session of motivational interviewing, there was a reduction in BMI-z scores P=.01 from the control group of similar composition. How would you interpret this study?
  • There were no significant differences between groups
  • There is a significant difference between the control and experimental group
  • The sample is too small to make any conclusions

Which of the following factors served as a stimulus to the initiation of employer based health insurance?

  • Which of the following factors served as a stimulus to the initiation of employer based health insurance?
      • Union management discussions
      • Movement towards social reform
      • Wage freezes during World War II
      • Supreme Court Rulings on Socialized medicine

Map out the content

  • Map out the content
  • Did you map out the composition of your test according to your material?
  • Is the knowledge that you are testing core for practice?

Should relate to the instructional objectives

  • Should relate to the instructional objectives
    • Measure learning objectives
    • Immeasurable learning objective
      • Understand the appearance of a pneumonia on chest X-ray

What material to you plan to cover

  • What material to you plan to cover
  • How many items for the entire test
  • Weight the content
  • Consider test item weight
    • Assign lower point values to items that assess lower-order skills (knowledge, comprehension)
    • Higher point values to items that assess higher-order skills (synthesis, evaluation).
  • Content Area
  • Common Problem of Toddler -15
  • Autism-5
  • Common Problem of Preschooler-15
  • Dermatology-15
  • Lower order question-Memory
  • 2
  • 3
  • 2
  • Higher order questions
  • 13
  • 5
  • 12
  • 13
  • 15
  • 5
  • 15
  • 15
  • Number of items: 50 items
  • Conventional Multiple choice
  • Stem & 3-5 distractors
  • Alternative choice
  • Stem and 2 options
  • Matching
  • 3-12 options preceding a group of stems
  • Multiple true-false
  • Declarative statement evaluated in terms of its truthfulness
  • Context dependent items
  • A stimulus following by one or more items
  • Unique MC
  • Complex
  • A stem followed by choices that are grouped into sets for test takers to choose
  • Haladyna, Downing, & Rodriquez, 2002

Which of the following would most effectively slow down the process of respiration in plants?

  • Which of the following would most effectively slow down the process of respiration in plants?
    • Cold weather
    • Stormy weather

Matching

  • Matching
  • Multiple True –false
    • You are a farmer who knows the secrets of growing strong healthy plants. Which of the following would describe the best farming practices (mark A if true, B if false
      • List of options numbered with possible true or false
  • Haladyna, Downing, & Rodriquez, 2002

Scenario

  • Scenario
  • Several questions follow about the scenario

Stem

  • Stem
    • Phrase as succinctly as possible
    • Get rid of excess information
        • Distractor
          • All should be plausible
        • Distractor
        • Distractor
        • Answer

A 6 week old (age) presents with a 12 hour history of non bilious vomiting (History). There is a 12 ounce weight loss. The exam is otherwise unremarkable (Physical exam results).

  • A 6 week old (age) presents with a 12 hour history of non bilious vomiting (History). There is a 12 ounce weight loss. The exam is otherwise unremarkable (Physical exam results).

Short and to the point

  • Short and to the point
  • What is the desired cognitive task for the examinee?
  • Avoid excessive verbiage
  • Include the central idea in the stem, not the choices
  • Word the stem in a positive way, avoid use of NOT or EXCEPT in stem

Which of the following is true?

  • Which of the following is true?
  • All of the following causes neuropathy except?
  • Which of the following is NOT a cause of tonsillitis?

A 6 week old (age) presents with a 12 hour history of non bilious vomiting (History). There is a 12 ounce weight loss. The exam is otherwise unremarkable (Physical exam results). What is the diagnostic study of choice?

  • A 6 week old (age) presents with a 12 hour history of non bilious vomiting (History). There is a 12 ounce weight loss. The exam is otherwise unremarkable (Physical exam results). What is the diagnostic study of choice?
    • Options would be listed

What is the most likely diagnosis?

  • What is the most likely diagnosis?
  • What is the most likely result to be obtained
  • Which of the following tests should be ordered.
  • What is most appropriate next step?
  • What of the following is the most likely cause?
  • Which of the following medications would be the most appropriate?

Which of the following is the most common side effect?

  • Which of the following is the most common side effect?
  • What is the most common organism?
  • What should be the first course of action?
  • What is the most common complication of this procedure, drug, or problem?
  • What additional information is needed for a proper diagnosis?

Edit and proof items

  • Edit and proof items
  • Use correct grammar, punctuation, capitalizations
  • Minimize the amount of reading in each items
  • Haladyna, Downing, & Rodriquez, 2002.

Does the question fit your content?

  • Does the question fit your content?
  • Do the items involve the application of Knowledge or a recall of facts?
  • Is decision making required by this question?
  • Avoid repeating words from the stem in the correct option
  • Try not to write the entire test in a day
  • Do not take phrases directly from text or lecture

Phrase repeated in each option

  • Phrase repeated in each option
  • Item with all relevant information in the stem
  • Which of the following actions would decrease radiation dose by one half?
    • Decreasing the mA by one quarter
    • Decreasing the mA by one third
    • Decreasing the mA by one half
    • Decreasing the mA by three quarters
  • By what fraction would mA need to be decreased to lower the radiation dose by one half?
    • One quarter
    • One third
    • One half
    • Three quarters

Develop as many effective choices as you can, three is adequate (Vyas & Supe, 2008; Haladyna et al)

  • Develop as many effective choices as you can, three is adequate (Vyas & Supe, 2008; Haladyna et al)
  • Only one choice should be the right answer
  • Vary the location of the right answer
  • Place choice in logical or numerical order
  • Choices should be independent and not overlapping
  • All of the above should be avoided
  • None of the above should be used carefully if at all.

Item should reflect specific, key content

  • Item should reflect specific, key content
  • Avoid repeating content questions
  • Avoid opinion based items
  • Avoid tricky items
  • Vocabulary
    • Consistent
    • Rephrase textbook slide content

Use the typical errors of students to write the distractors

  • Use the typical errors of students to write the distractors
  • Limit the number of answer options
    • 3 distractors is actually enough
  • Do not try to trick students—attractive and plausible
  • Longest answer is usually correct one
    • When writing, make sure all options are equally constructed
      • Item writers spend more time on the correct answer

Use words that are familiar to students to make the distractor more plausible

  • Use words that are familiar to students to make the distractor more plausible
  • If a recognizable key word appears in correct answer, it should be in other distractors also
  • Incorrect option or distracters must be plausible but definitely incorrect
  • Do not make the options too easy to discriminate
  • Distractors should be attractive and plausible but not tricky

Avoid “All of the above”

  • Avoid “All of the above”
    • Recognition of one wrong option eliminate the all of above option
  • Watch the length, grammatical structure, and the level of specificity
    • The more specific, the more likely the answer
  • Similar option items increase the item difficulty (Ascalon et al. 2007)
  • Avoid imprecise terms: seldom, rarely, occasionally sometimes, few or many (Collins, 2006

Overlapping distractors

  • Overlapping distractors
    • If you are asking when to give a particular drug
      • Before breakfast
      • On a full stomach
      • With meals
      • Before going to bed
    • Three options have to do with eating and two with time of day. Only one relates to both. Students can pick this one out.

Order the options to facilitate the tester finding the correct answer

  • Order the options to facilitate the tester finding the correct answer
  • If you use numerical values, list them from high to low or low to high
  • Options should not overlap or be too specific
    • Example
      • Less than 25%
      • 30-50%
      • Exactly 24%
      • 50% or great

Avoid the complex multiple choice format

  • Avoid the complex multiple choice format
    • Which of the following are fruits
      • Tomatoes
      • Tomatillos
      • Habaneras peppers
        • 1 & 2
        • 2 & 3
        • 1 & 3
        • 1, 2, & 3
  • Haladyna, Downing, & Rodriquez, 2002

Option C does not follow grammatically from the stem

  • Option C does not follow grammatically from the stem
  • All options follow grammatically from the stem
  • A chest radiographic findings of left upper lobe collapse is
    • Hyperlucency of the upper and lower left hemithorax
    • Elevation of the left diaphragm
    • When the mediastinum shifts to the right
    • Posterior displacement of the minor fissure
  • What is a chest radiographic finding of left upper lobe collapse?
    • Hyperlucency of the upper and lower left hemithorax
    • Elevation of the left diaphragm
    • Mediastinal shift to the right
    • Posterior displacement of the minor fissure
  • Collins, 2006)

Guide to terms that will guide you assessing the cognitive level of the item

  • Guide to terms that will guide you assessing the cognitive level of the item
  • Reasoning/application items
    • Apply
    • Analyze
    • Anticipate
    • Assess
    • Compare
    • Contrast
    • Decide
    • Develop
    • Differentiate
    • Evaluate
    • Justify
    • Prioritize
    • Memory
      • Define
      • Describe
      • Identify
      • Recall

What is the primary sign of an intestinal obstruction?

  • What is the primary sign of an intestinal obstruction?
      • Coffee colored emesis
      • Bile like emesis
      • Jaundice
      • Projectile vomiting

A 12 week old presents with bile colored emesis. What is the most likely diagnosis?

  • A 12 week old presents with bile colored emesis. What is the most likely diagnosis?
    • Pyloric stenosis
    • Gastroesophageal reflux
    • Volvulus
    • Overfeeding
  • Editing your work

How do your exam questions match your content map

  • How do your exam questions match your content map
  • Have you tested key points
  • Run a grammar and spell check
    • Stedman’s medical dictionary
  • Review your options
    • Is there only one correct answer
    • Have another person review the test
    • Do a scantron to make sure your numbering is correct.
  • Response Item #1
  • Correct
  • P (diff)
  • Difficulty
  • 1 – P (diff)
  • Point biserial correlation
  • A
  • Correct
  • 0.72
  • 0.28
  • 0.40
  • B
  • 0.09
  • C
  • 0.10
  • D
  • 0.09
  • Response Item #2
  • A
  • Correct
  • 0.70
  • 0.30
  • 0.35
  • B
  • 0.00
  • C
  • 0.15
  • D
  • 0.15


Download 22.96 Kb.

Share with your friends:




The database is protected by copyright ©sckool.org 2020
send message

    Main page