Enhancing noncognitive skills to boost academic achievement Educational Testing in America: State Assessments, Achievement Gaps, National Policy and Innovations Session III: Innovations in Testing



Download 11,45 Kb.
Date conversion29.10.2017
Size11,45 Kb.

Enhancing noncognitive skills to boost academic achievement Educational Testing in America: State Assessments, Achievement Gaps, National Policy and Innovations Session III: Innovations in Testing Patrick C. Kyllonen Educational Testing Service Princeton, NJ

  • Kyllonen, P.C. (September, 2008). Enhancing noncognitive skills to boost academic achievement. In Educational Testing in America: State Assessments, Achievement Gaps, National Policy and Innovations (Session III: Innovations in Testing). Willard Hotel, Washington, DC.

Noncognitive Skills

  • What are they?
  • Are they important for achievement?
  • How can we measure them?
  • Can they be improved?
  • How can we improve them?

Noncognitive Skills

  • What are they?
  • Are they important for achievement?
  • How can we measure them?
  • Can they be improved?
  • How can we improve them?
  • Cognitive Skills = Knowledge,
  • Ability,
  • Intelligence,
  • “Smarts”
  • Noncognitive Skills = Personality,
  • “Soft Skills” Attitudes,
  • “Personal Skills” Values,
  • Beliefs

What Are the Noncognitive Skills?

  • Professionalism
  • Work ethic
  • Teamwork
  • Collaboration
  • Oral communication
  • Critical thinking
  • Agreeableness
  • Ethics
  • Self-esteem
  • Diversity
  • Leadership
  • Creativity
  • Innovation
  • Lifelong learning
  • Enthusiasm
  • Values
  • Character
  • Breadth
  • Open-mindedness
  • Persistence
  • Collegiality
  • Independence
  • Motivation
  • Planning
  • Organization
  • Self-efficacy
  • Anxiety
  • Self-concept
  • Leading
  • Deciding
  • Supporting
  • Cooperating
  • Interacting
  • Presenting
  • Adapting
  • Coping
  • Enterprising
  • Performing
  • Extroversion
  • Emotional stability
  • Conscientiousness
  • Openness

A Simplified Framework

  • Personality (“Noncognitive Skills”)
    • Conscientiousness
      • Dependability, responsibility
      • Aspiration, achievement striving
      • Ethics, integrity, honesty
    • Emotional Stability
      • Resilience (response to feedback; working under pressure)
      • “Core self evaluation” (self-efficacy, locus of control)
    • Openness
      • Engagement, interest, enthusiasm
    • Agreeableness
    • Extroversion
      • Leadership
  • Attitudes
    • Subject-specific interest/self-efficacy (e.g., math, reading, science)
    • School (e.g., identification with)

Noncognitive Skills

  • What are they?
  • Are they important for achievement?
  • How can we measure them?
  • Can they be improved?
  • How can we improve them?

Noncognitive factors

  • Yes
  • Why do we think they are important?
    • K-12 large-scale assessments
    • Industry incremental validity studies
    • Industry & Higher education interview studies

Predictive validity in K-12 Achievement tests

  • NAEP Math
  • Demographics (income, gender, race, etc.)
  • Home possessions (books, computer, etc.)
  • Noncognitive (personality, attitudes, values, etc.)
  • PISA Math
  • PISA Reading
  • PISA Science
  • ECLS Math
  • ECLS Reading
  • ECLS Science
  • Multiple R
  • Note. PISA didn’t measure income or race/ethnicity (demographics), but it did measure lateness to class (study time), and student aspirations (noncognitive).
  • Source: Lee, J. (2007). Noncognitive factors in education. Paper presented at the Annual Meeting of the American Psychological Association, San Francisco, CA..

Meta-analyses have shown that noncognitive skills add to cognitive skills in predicting workforce performance

  • % variance accounted for
  • Job performance
  • Training time
  • }
  • noncognitive
  • cognitive
  • }
  • Source: Schmidt, F.L., & Hunter, J.E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin. 124(2), 262-274.
  • Example items:
  • “I arrive on time”
  • “I work hard”
  • Example tests
  • Problem-solving
  • Reading

Interviews with Industry

  • Conference Board et al. (2006)
  • 400+ employers interviewed
  • “What skills are most important for workforce?”
  • “How well prepared are graduates?”
  • “applied skills” (mostly noncognitive) ranked higher than content skills
  • Examples:
  • The Conference Board, Partnership for 21st Century Skills, Corporate Voices for Working Families, & Society for Human Resources Management (2006). Are they really ready to work? Employers perspectives on the basic knowledge and applied skills of new entrants to the 21st century U.S. workforce. New York: The Conference Board.

How Important are the Noncognitive Skills?

  • Prof./Work ethic (86%)
  • Teamwork/collaboration (84%)
  • Oral communication (83%)
  • Critical thinking/ problem solving (74%)
  • Ethics (73%)
  • Written communication (72%)
  • Information tech. (68%)
  • Diversity (60%) 4-yr only
  • Leadership (82%)
  • Creativity/Innovation (81%)
  • Lifelong learning (78%)
  • Rated “Very Important”
  • N = 347 ~ 413
  • Reading comp (74%)
  • English (73%)
  • Writing (77%)
  • Math (64%)
  • (Mean of HS, CC, 4-yr)

Noncognitive Skills

  • What are they?
  • Are they important for achievement?
  • How can we measure them?
  • Can they be improved?
  • How can we improve them?

How do we assess noncognitive skills?

  • Self Assessments
  • Teacher (and others’) Ratings
  • Situational Judgment Tests

Situational Judgment

  • (Teamwork: Resolve conflict and negotiate)
  • You have recently formed a study group with several of your classmates in order to prepare for a difficult final exam. Unfortunately, the various members of the group have very different schedules, so you all meet after class one day to try to work out a final schedule for your group review sessions.
  • Which of the following is the most important factor to consider in weighing any proposed suggestions?
  • (A) Making sure that the schedule will allow the smartest students to attend, so that the study group will cover more material.
  • (B) Making sure the proposed meeting times do not conflict with your own course schedule.
  • (C) Yielding to the majority of the group even if it means some members will not be able to participate.
  • (D) Breaking the group down into sub-groups based on compatible schedules. *

Noncognitive Skills

  • What are they?
  • Are they important for achievement?
  • How can we measure them?
  • Can they be improved?
  • How can we improve them?

Noncognitive Skills

  • What are they?
  • Are they important for achievement?
  • How can we measure them?
  • Can they be improved?
  • How can we improve them?

Noncognitive Skills

  • What are they?
  • Are they important for achievement?
  • How can we measure them?
  • Can they be improved?
  • How can we improve them?
    • Analysis of questionnaire data (to get scales)
    • Expert panels to get feedback & action plans
    • Randomized control trial to determine whether interventions lead to achievement (and other) gains

Expert Panels to Develop Feedback & Action Plans

Summary

  • Noncognitive skills are important
    • Educators believe
    • Workforce community believes
    • Validity evidence to support the belief
  • Noncognitive skills change over the lifespan
  • We are currently evaluating efforts to improve noncognitive skills (e.g., time management, test anxiety)
    • & evaluate the effects of such improvements on achievement

Questions? Comments?



The database is protected by copyright ©sckool.org 2016
send message

    Main page