Jane Seale 4 Introduction

Download 334.5 Kb.
Size334.5 Kb.

Using CAA to support student learning Jane Seale




omputer-assisted Assessment, or CAA, refers to the use of computers in assessment. Computers can be used to deliver, mark and analyse student assignments or examinations. CAA can be delivered using stand-alone computers or online, using an Intranet or Internet. There are two main forms of CAA. The first is where students input their answers via the computer. The second is where students input their answers onto a pre-printed paper test. This is then fed into an Optical Mark Reader (OMR), which rapidly processes the paper forms by scanning for marks or crosses. CAA tests can be supervised or non-supervised, formative or summative. There is also a diagnostic role for CAA in determining students’ prior knowledge of a subject. The growing interest in online learning within tertiary education is leading to a recognition that any plans to implement online learning should also include a consideration of CAA

.How can CAA support learning and teaching?

What kind of questions can be used in CAA?

When most people think of CAA, they think of multiple choice questions. There are however, a number of other kinds of questions (see Table 1) and it is generally recognised that in order to increase student participation and motivation, especially in formative applications of CAA, it is better to use a range of questions types.

The most common kind of question that is used in CAA tests is objective test questions where answers are selected from or compared to a limited set of pre-defined responses to a question.

Computerised marking is considered appropriate or useful with such questions because no judgement has to be made about the correctness of the answer.

Objective test questions might require students to:

  • Select an answer from a set of choices, eg Multiple Choice Questions (MCQs) and True-False questions

  • Select an area on the screen (graphical hotspot)

  • Supply short numeric or text responses (text input).

  • Assertion-Reason questions

  • Build up images

  • Crossword puzzles

  • Case Studies

  • Completion questions

  • Drag and drop

Drawing questions

Essay questions

Field simulation

Graphical hotspot questions

Image identification questions

Justification questions

Labelling and building questions

Matching questions

Matrix questions

Multiple choice questions

Multiple response questions

Ranking questions

Sequencing questions

Short answer questions

Sore finger questions

Text/Numerical questions

True/False questions

Yes/No questions

  • Table 1: The range of question types that can be used in CAA

  • These types of questions lend themselves easily to the creation of question banks. Questions can be randomly chosen from a bank allowing a unique set of questions to be selected for each assessment.

  • According to the CAA Centre “it is worth remembering that an objective test is only as objective as the test designer makes it” 1. The Centre provides a detailed guide to designing objective questions including looking at the phrasing of questions, writing alternative answers and advice on the time and effort required.

Objective test questions are useful for assessing knowledge and comprehension, but there is a common assumption that they cannot assess higher order critical thinking skills. Bull and Collins2 argue, however, that this perception often limits the application of objective tests and restricts the potential for creative and sophisticated question and test design.

Although objective test questions are the most frequently used questions in CAA, subjective testing does exist. For example computer programs can be written that assess student projects for content, style, originality and layout. (See CEILIDH3 project for an example).

Computerized assessment of student essays (free text responses) is a continuing research area. Current work is focusing on principles of computational linguistics. While this research has yet to be translated into wide spread commercial products, there are ways of using the computer to handle assessed essays. For example, TELRI contains tools that allow students to submit their essays online and for their peers to evaluate these essays using online “white space” 4. Phil Davies5 describes how he has used computerised peer assessment as a method for assessing essays of students on a final year module of an undergraduate course in computing studies in the University of Glamorgan. Students are required to electronically submit their essay, and undertake a process of computerised self-assessment on the essay. This is followed by a process of peer assessment.

What subject disciplines are using CAA?

In 1999 the CAA Centre conducted a survey of CAA usage across different subject areas. The results revealed that most academic disciplines deployed CAA in some way. Subject areas where CAA tests most commonly included computing/IT; biomedical science, mathematics, engineering and modern languages1.

An example of the use of CAA in engineering is the e3an project, funded by HEFCE to develop a network of expertise in assessment issues within electrical and electronic engineering. A major focus of this project is the development of a test bank of questions for use in both formative and summative assessments6.

Although CAA has been used predominantly in science and technology subjects, there are increasing examples of the use of CAA in other disciplines. For example, Chris Hopkins7 describes how he has used QuestionMark to introduce CAA into a first year module called “Modernity and Modernisation” within the Cultural Studies department at Sheffield Hallam University. Evaluation of student responses to the CAA tests led Hopkins to conclude that CAA could be perceived as liberating under the right circumstances (see Resources section).

What CAA tools are available?

A number of CAA tools are available. Some are available from commercial producers (e.g. QuestionMark Perception8- a web based tool), others are the results of institutional projects and are freely available to staff in higher education. Examples of Institutional developers include:

CASTLE (Computer Assisted Teaching and Learning) is a JISC/JTAP funded project, which provides an authoring shell for the delivery of web-based MCQs. It is particularly useful for tutors who have minimal IT skills and do not have any prior knowledge of HTML or CGI scripting9.

TRIADS (The University of Derby/Tripartite Interactive Assessment Delivery System) is a toolkit for users of Authorware Professional to develop CAA using a wide range of question types10.

In addition to these specialised CAA tools Virtual Learning Environments such as WebCT and Blackboard also contain assessment tools that facilitate electronic submission of assignments and self-test exercises. Many institutions are moving towards campus-wide implementation of Virtual or Learning Environments. This has led to new projects that are looking at how to integrate commercial CAA packages into Managed Learning Environments (MLEs). For example, the JISC Web CAA Project at Loughborough University aims to incorporate the QuestionMark Perception package into their Management Information System11 (see the LTSN starter guide number 2 on VLEs).

Advantages and disadvantages

Administrative advantages

  • Computerised marking is not prone to human error

  • Saves staff time in terms of supervising and marking (including double marking) assessments

  • Reduction of printing costs, particularly when tests are updated or changed.

  • Administrative disadvantages

  • Implementing a CAA system can be costly and time-consuming (especially when trying to integrate with an institution’s MLE)

  • Staff who design and invigilate CAA need training in assessment principles and design, IT skills and examinations management

  • A high level of collaboration between all those involved in designing and implementing CAA is required

  • Some systems cannot implement anonymous marking

Hardware and software used to deliver CAA needs to be robust in order to avoid failure at crucial times such as examinations

.Pedagogic advantages

  • Tutors can incorporate hints into test questions.

  • Tutors can monitor the progress of students through frequent use of assessments

  • Students can monitor their own progress and revise and rehearse at their own pace

  • Detailed and specific feedback can be given to students during and immediately after a test

  • Tutors can assign different learning activities to students based on their test results

  • It can be linked to other web-based/computerised learning materials and systems (e.g MLEs)

  • Can provide tutors with feedback for evaluation of modules/courses/programmes.

  • Pedagogic disadvantages

  • Unsupervised CAA sessions present a risk of plagiarism ( it can be difficult to authenticate the identity of students)

  • Students need to have sufficient IT skills and experience of the requirements of CAA

  • Staff have a tendency to just use MCQs which can be tedious and demotivating for students, and it has been argued that MCQs focus on testing superficial levels of student learning.

Getting started

Individual practitioners who are thinking about using CAA may find it useful to start off using simple MCQs with students as part of a formative assessment exercise. In cases where practitioners wish to use CAA as part of the summative assessment process it is likely that they will need to consult departmental and institutional teaching and learning committees.

Any decisions that individual lecturers or institutions make regarding how CAA is going to be implemented will probably need to consider the associated advantages and disadvantages. The guide, “A Blue Print for Computer-Assisted Assessment”, by Joanna Bull and Colleen McKenna, offers assistance with the weighing-up process12, and provides a number of useful check-lists and frameworks for practitioners who are introducing CAA into their courses or institutions.

Ricketts et al13 describe how they implemented CAA across six sites at the University of Plymouth. They argue that a number of key decisions were made that led to acceptance of the CAA system by both staff and students. These included:

  • Establishing a steering group which involved staff from all affected departments and a student representative

  • Providing staff training programmes

  • Giving the students opportunity to familiarise themselves with the system before it was used for summative assessment.

  • Finally, before launching into CAA it is important to address fundamental issues of assessment and its role and purpose in learning and teaching. The aim of any assessment is to be valid, reliable, fair and useful to the student.

References and resources

1 caacentre.ac.uk/

2 Bull, J. and Collins, C. (2001), Ways of using technology to assess higher order skills: a comparison of subjective and objective assessment techniques and the implications for teaching and learning. Paper presented at
ALT-C 2001

3 CEILIDH Project: www.cs.nott.ac.uk/~ceilidh/

4 Technology Enhanced Learning in Research Institutions Project: www.telri.ac.uk/

5 Davies, P. (2001), Computerised self/peer support, learning and assessment of final year graduates. Paper presented at ALT-C 2001

6 Electrical and Electronic Engineering Assessment Network: www.e3an.ac.uk/

7 Hopkins, C. (1998), “Web+Qmark+Humanities=?”, Computers and Texts, 16/17. info.ox.ac.uk/ctitext/publish/comtxt/ct16-17/

8 QuestionMark commercial software:

9 CASTLE: www.le.ac.uk/cc/ltg/castle/tools/intro.html

10 www.derby.ac.uk/assess/newdemo/mainmenu.

11 JISC Web CAA: www.lboro.ac.uk/service/ltd/flicaa/jisc/index.html

12 Bull, J. and McKenna, C. (2001), Blue Print for Computer-assisted Assessment
Available from http://caacentre.ac.uk/

13 Ricketts, C., White, S. and Bouch, G. (2001), The distributed university rolling out computer assessment. Paper presented at ALT-C 2001.

Case studies/examples of CAA

Discussion about students taking a module in building structures and their use of CAA to assess their own and obtain feedback: ctiweb.cf.ac.uk/HABITAT/HABITAT2/tanner.html

The role of CAA for ESL: www.aitech.ac.jp/%7Eiteslj/Articles/


Case studies in the arts: info.ox.ac.uk/ctitext/durham/caa.html


The author acknowledges the constructive feedback from Susi Peacock at QMUC; and wishes to thank the ELICIT project for permission to reproduce Table 1.

Dr Jane Seale is a Lecturer in Assistive Technology at Kings College, London.

Design by jamietshaw@hotmail.com

Download 334.5 Kb.

Share with your friends:

The database is protected by copyright ©sckool.org 2022
send message

    Main page