Deriving and measuring group knowledge structure via computer-based analysis of essay questions: The effects of controlling anaphoric reference



Download 226 Kb.
Date19.05.2018
Size226 Kb.

Deriving and measuring group knowledge structure via computer-based analysis of essay questions: The effects of controlling anaphoric reference

  • Roy B. Clariana
  • Pennsylvania State University, USA
  • RClariana@psu.edu
  • Veronica M. Godshalk
  • University of South Carolina, USA
  • Godshalk@gwm.sc.edu
  • Patricia E. Wallace
  • The College of New Jersey, USA
  • PWallace@tcnj.edu
  • Available at: http://www.personal.psu.edu/rbc4/CELDA.ppt

How do pronouns influence computer-based text analysis?

  • Participants in an undergraduate business course (N = 49, 4 missing data, final N = 45) completed an essay as part of the course final examination and investigators manually edited every occurrence of pronouns in these essays to their antecedents.
  • The original unedited essays (with pronouns) and the edited essays (no pronouns) were processed by ALA-Reader software using linear aggregate and sentence aggregate methods. These data were then analyzed using a Pathfinder network (PFNET) approach.

ALA-Reader text analysis

  • ALA-Reader – free software, converts text to a network representation (open with CMAP tools from IHMC) and also outputs relationship data in the prx format that can be analyzed by KNOT Pathfinder network software.
  • ( http://www.personal.psu.edu/rbc4)

ALA-Reader text analysis

  • Example sentence from a student’s essay
  • Key words are underlined
  • Pronouns are shown in red
  • ALA-Reader has two approaches,
  • either linear aggregate or sentence aggregate

ALA-Reader text analysis

  • Humanists believed that job satisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.
  • humanist
  • job
  • satisfaction
  • productivity
  • employee
  • empowered
  • Here is the sentence aggregate run shown in black.
  • Historical basis…an excel spreadsheet to do sentence aggregates
  • job
  • productivity

ALA-Reader text analysis

  • Humanists believed that job satisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.
  • humanist
  • job
  • satisfaction
  • productivity
  • employee
  • empowered
  • Here is the linear aggregate run shown in black,
  • so far so good,
  • but what happens on the 2nd occurrence of ‘job’

ALA-Reader text analysis

  • Humanists believed that job satisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.
  • humanist
  • job
  • satisfaction
  • .
  • empowered
  • .
  • productivity
  • employee

ALA-Reader text analysis

  • Humanists believed that job satisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.
  • humanist
  • job
  • satisfaction
  • productivity
  • employee
  • .
  • empowered

ALA-Reader text analysis

  • Humanists believed that job satisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.
  • The force-directed
  • graph begins to
  • fold back on itself

ALA-Reader text analysis

  • Humanists believed that job satisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.
  • Unedited essay – 7 links
  • (pronouns present but invisible)
  • Edited essay – 8 links
  • (pronouns converted to referents)

ALA-Reader text analysis

  • When pronouns are changed to their referent, the term Humanist (right figure) takes on central importance in the network (has the most links).
  • Unedited essay – 7 links
  • (pronouns present but invisible)
  • Edited essay – 8 links
  • (pronouns converted to referents)

ALA-Reader data output

  • ½ array = prx file
  • 29 terms used here

ALA-Reader text analysis

  • Humanists believed that job satisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.
  • humanist
  • job
  • satisfaction
  • productivity
  • employee
  • empowered
  • Here is the sentence aggregate run shown in black.
  • Sparse sentences under specified and dense sentences over specified, pronouns always under specify representations in the sentence aggregate approach.
  • job
  • productivity
  • Sentence aggregate
  • prx output
  • Humanists believed that job satisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.
  • Linear aggregate
  • prx output
  • ALA-Reader Sentence and linear aggregate output

The essay writing prompt and 29 key terms

  • Describe and contrast in an essay of 300 words or less the following management theories: Classical/Scientific Management, Humanistic/Human Resources, Contingency, and Total Quality Management. Please use the terms below in your essay: administrative principles, benchmarking, bureaucratic organizations, contingency, continuous improvement, customers, customer focus, efficiency, employee, empowerment, feelings, Hawthorne studies, human relations, humanistic, leadership, management (i.e., bosses), Management by Objectives, motivate, needs, organization (i.e., corporation), plan, product, quality, relationship, scientific management (classical), service, situation (or environment), TQM, and work (or job, task).”

Descriptive data regarding the essays

  • The 45 student essays averaged 301 words per essay (range from 170 to 479 words, standard deviation of 70.1).
  • The fifteen most common words account for 31% of all of the text and include in order the (6.2% of all words), and (3.3%), of (3.3%), to (3.1%), management (2.4%), a (1.9%), is (1.8%), in (1.6%), that (1.4%), employees (1.4%), this (1.1%), on (1.1%), are (0.9%), as (0.9%), and their (0.9%).
  • The three most common pronouns are their (ranked 15th. 0.9% of all words), it (18th, 0.8%), and they (19th, 0.8%). Students use of ‘their’ and ‘they’ in their essays indicates that they were writing from the perspective of either a manager or of an employee and not always as an independent observer.

prx output for student essays b07 and b10 (29 key terms)

Example: prx average of student essays b07 and b10

  • It is possible (even easy) to use KNOT software to average together any number of individual student prx (essay) files in order to represent a team or group composite network representation; and then compare group-to-group and individual-to-group, etc. We compared top (n=15) and bottom (n=15) students.

Pathfinder KNOT

  • KNOT analyzes prx data
  • Based on graph theory, KNOT uses an algorithm to determine a least-weighted path that links all of the terms. The resulting PFNET (force-directed graph) is based on a data reduction approach that is purported to represent the most salient relationships in the raw proximity data.

Linear aggregate PFNET group average representation of the top 15 student essays

  • Unedited essay – 31 links
  • (pronouns are not used)
  • Edited essay – 34 links
  • (pronouns converted to referents)
  • 29 common links
  • Unedited essay
  • Edited essay
  • It is fascinating to us to compare the PFNET visual depictions of the averaged essays. For example, the terms ‘management’, ‘employee’, ‘organization’, ‘customer’, and ‘product’ are well connected and so are central terms in both representations. Three of the four super-ordinate management theory categories in the essay writing prompt, ‘scientific management’, ‘contingency’, and ‘TQM’, were all associated with ‘management’ while the fourth category, ‘humanistic’, was associated with both ‘management’ and ‘employee’. Emotion-related terms such as ‘feelings’, ‘needs’, ‘relationship’, and ‘motivation’ were all associated with ‘employee’ in both PFNET representations as were the action words ‘benchmarking’, ‘work’, and ‘product’.

Comparing unedited and edited group representations

  • Pearson correlations of the averaged proximity raw data (above the diagonal) and PFNET similarity data (below the diagonal in red) for the top group (n = 15) and the bottom group (n = 15) for both unedited essays and edited essays.

Individual essay scores

  • To calculate students’ essay scores, we used KNOT to compare the students PFNET to an expert PFNET. Student’s score consisted of the number of links in common with the expert.
  • A “universal” essay score for each student was established for comparison purposes using the SPSS factor score combination of the three human essay scores plus the ALA-Reader linear aggregate unedited essay score

Individual essay scores

  • Correlations with ALA-Reader individual essay scores are shown in red. The length-adjusted ALA-Reader score correlation with the universal score (r = .84) is almost as good as that of the human raters.

Summary/Conclusions

  • The ALA-Reader sentence aggregate approach was not discussed much in this presentation, it didn’t perform well, but it is described in the manuscript
  • For the ALA-Reader linear aggregate approach, there was little difference between the PFNETs obtained for edited and unedited essays. Apparently the linear approach is relatively robust to pronoun interference and is generally superior to the sentence aggregate method for the narrow purposes of average group knowledge representation and for scoring individual essays.


Download 226 Kb.

Share with your friends:




The database is protected by copyright ©sckool.org 2020
send message

    Main page