Universities as Responsive Learning Organizations Through Competency-Based Assessment with Electronic Portfolios

Download 179.8 Kb.
Size179.8 Kb.
Universities as Responsive Learning Organizations Through Competency-Based Assessment with Electronic Portfolios
Darren Cambridge

Assistant Professor of Internet Studies and Information Literacy

George Mason University, New Century College

4400 University Drive MS 5D3

Fairfax, VA 22030


Electronic Portfolios, Standards and Competencies in General Education

Electronic portfolio are increasingly being used by colleges and universities to track progress towards general education outcomes. Advocates of this approach see portfolios as both more flexible than standardized testing and more easily comparable across a program or institution than the results of authentic assessments at the course level, such as papers, projects, and exams. Portfolios have the potential to provide multi-dimensional assessment data while remaining firmly grounded in the diversity of learning activities and their products with which faculty and students engage in the classroom. Pursuing such assessment with psychometric rigor present considerable challenges {Wilkerson, 2003 #547}. However, there are several successful examples of systematic portfolio assessment with portfolios around, both print and electronic, such as work with writing across the curriculum portfolios at Washington State University, the Learning Record system, and Alverno College’s Digital Diagnostic Portfolio {Haswell, 2001 #544; Hallam, 2000 #707}. [What should I cite for Alverno?] Many other institutions are building on these pioneering models. Within the membership of the Inter/National Coalition for Electronic Portfolio Research, for example, in addition to Alverno and Washington State, Indiana University Purdue University at Indianapolis, Portland State University, Arizona State University Polytechnic, Thomas College, Bowling Green State University, and multiple campuses of the California State University and Minnesota State Colleges and Universities systems are using electronic portfolios to track student learning in connection with a set of general education outcomes {Cambridge, In press #740}.
Most of these projects are similar in two respects. First, from an institutional perspective a primary goal of the assessment is quality assurance. The results of the portfolio assessment enable the institution to determine the extent to which student learning is meeting the goals the institution has set for general education. Determined by faculty and administrators, these goals dictate the structure of the portfolios and the way they will be evaluated. While in many cases helping students become more reflective, intentional learners who see connections between knowledge and action in multiple domains is also an essential result, the lack of data on which to judge institutional performance towards a set of standards set by the institution would mean a failure.
Second, these standards are communicated through a conceptual framework that breaks down the complexity of learning in general education into a set of valued categories that can be used to examine it. These categories take a variety of names, such as student learning outcomes, abilities, standards, principles of learning, and competencies. In most cases, because the focus of assessment is to see if the evidence of student learning for each category matches a description of a desired level of performance defined by the institution, standards or outcomes may be the most accurate terms. Reducing the complexity of general education through a logical and comprehensive system of classification allows for each dimension to be assessed outside of the confines of individual disciplines or programs. Institutions can determine towards which outcomes students are making excellent progress and focus attention on those where they are not.
This form of assessment is undoubtably valuable, and institutions embracing it are becoming more effective and accountable because of it. However, the experience of New Century College (NCC) at George Mason University suggests that portfolios may be profitably be used in conjunction with a shared conceptual framework for classifying general education for a different organizational purpose. The categories, which in NCC are calls competencies, may serve as something other than standards or outcomes. A different flavor of competency-based assessment may enable sharing power with students. Sharing power is both a central value of NCC and a priority for many other educators committed to building learning communities {Eby, In press #741; Smith, 2001 #742; Shor, 1992 #743}. It also may serve as a model for relationships between institutions and individuals in the world beyond the academy.
Higher Education Institutions as Models for the Responsive Learning Organization

Higher education increasingly encourages reflective learning as a path towards living a full, integral life and being able to continue to learn in a rapidly-changing workplace that demands flexibility and adaptation. However, as authors such as Richard Sennett suggest, this increased call for mobility and flexibility erodes the commitment employers once had to their workers and frustrates the desire for mastery, making it difficult to craft coherent, long-term life narratives {Sennett, 1998 #719}. Thus, the humanist goal of self-actualization through lifelong learning and the expectation of lifelong learning in the contemporary workplace may come into conflict, despite a significant body of writing in management and education that attempts to see them as one and the same {Elmholdt, 2006 #739}. The discourse of the “learning organization” may show how individual reflective learning supports organizational success, but it rarely insists upon organizational success being defined in a way that takes into the account the values and objectives of workers. The equation of individual lifelong learning with ongoing organizational development usually only solves in one direction.

However, escape is not an option. Human identity forms in relationship to institutions, and power flows through them. Even in a highly networked world, lifelong learners are unlikely to meet their goals and to develop in ways that enable them to be productive members of society—as defined from either an individual or institutional standpoint—unless they develop strong, long-term relationships with businesses, governments, and other institutions. Rather than avoiding dependence on institutions, our goal should be fostering interdependence of individuals and institutions, each shaping the other over time in a manner that balances their interests without glossing over their differences.
Institutions of higher education can serve as models for this more ethical relationship between individual learners and learning organizations. Universities should give students the experience of learning reflectively where they are both shaped by the educational community of which they are a part and have genuine agency to change the way that community works. With this background, students will be prepared both to succeed in a workplace that requires them to be flexible and to expect and demand that employers continuously hold themselves accountable for promoting their employees’ interests. If we want an ethical workplace for knowledge workers, we have to give students a voice in charting the course of the University.
Doing so requires a means for capturing and connecting both individual and organizational values and performance. In a higher education context, this means documenting what students think is important and the evidence they have to support these beliefs, putting theses voices into conversation with those of other stakeholders, such as faculty and alumni. By so doing the organization discovers what it believes, as reflected in its actual practice of teaching and learning. This understanding can then be used to better align how the institution functions with what it values. At the same time, in order to avoid suppressing difference, space also is needed for individual identities to be represented in a form differentiated from the collective that expresses the integrity of individual life stories.

A New Role for Competency-based Assessment

Competency-based assessment can be this means, can be a process that makes this balanced relationship a reality. However, it requires rethinking some assumptions commonly associated with the use of competencies in higher education in recent years, particularly in relationship with portfolios, several of which were introduced above.
First, competency-based assessment can no longer be simply a matter of measuring how well students’ learning matches a set of pre-defined, measurable standards of performance. Rather, it must be focused on capturing a picture of the standards all stakeholders value as enacted in practice and analyzing how not only student learning but also the curricular and programmatic structure of the University align with these values.
Second, competencies need to be seen not simply as a means for articulating the institution’s expectations to students, but more centrally as a means for connecting individual reflection with organizational learning where each is accountable to the other. Competencies should function as what Susan Leigh Star has termed boundary objects, conceptual technologies that join local and global actions together, providing the opportunity to be very specific at the local level while maintaining a sense of collective coherence when multiple local activities are woven together {Star, 1989 #257}. Competencies serve as a shared framework that enable students to tell their highly-individual stories while enabling the larger learning community within the institution to put these stories into conversation and to collectively reflect on what it values and how well it’s doing.
Finally, competencies must be used within the context of what I term a reflective philosophy of assessment {Cambridge, In Press #744}. Peter Gray identifies two primary philosophical approaches to assessment in the history of US higher education {Gray, 2002 #705}. The objectivist and utilitarian perspective argues that educational outcomes can be precisely described before the fact in terms of observable behaviors that can be measured in a manner independent of who is doing the measuring. The subjectivist and intuitionist perspective, in contrast, holds that learning is more complex and situated. The value of outcomes cannot be determined until learning has occurred and must be judged through the expertise of professional authorities. Gray notes that, while the former may be attacked as reductive, the latter is open to charges of elitism because it restricts conversation about what counts of genuine and valuable to an exclusive class and experts and institutional leaders.
The reflective philosophy shares with the later of these two traditions an investment in the situated judgments of those most directly involved with teaching and learning. However, it avoids elitism by valuing the reflective analyses of individual learners alongside those of those who have traditionally had exclusive control over judgment. In the words of Kathleen Yancey, from the perspective of reflective assessment, students are authoritative informants about their own learning {Yancey, 1998 #520}.
Competencies, in reflective assessment, can provide a common language that helps connect the critical inquires about learning of all members of the learning community. They serve as a heuristic for framing individual knowledge about learning in a form that can connect to the larger community conversation and affect institutional change.

Reflective Assessment with Portfolios at New Century College

At New Century College (NCC) of George Mason University, this philosophy of reflective assessment underlies the use of electronic portfolios throughout the curriculum. Portfolios serve both as an opportunity for students to tell their individual stories, taking ownership of their learning, and as a bridge connecting inquiry into individual and organizational practice. NCC’s nine competencies serve as a shared conceptual framework that enables this connection. Student portfolios become an important part of a community conversation about what it means to be an educated person in the twenty-first century, and this conversation shapes the way programmatic and curricular decisions are made in the work of the College.
Students in NCC compose portfolios in multiple iterations over the course of their undergraduate careers. In the first year program, students compose portfolios for each of the four quarter-long intensive learning communities that make up the first-year program—Community of Learners, The Natural World, The Social World, and Self as Citizen—culminating in an ePortfolio that chronicles their learning experiences, both within and beyond the classroom, throughout the year. In most other courses students take at NCC, especially in team-taught, upper-division learning communities, they are asked to develop a portfolio that represents their learning during the course. In their final year, drawing on their portfolio work throughout their time at Mason, students compose an extensive graduation portfolio {New Century College, 2001 #745}. Students complete this final iteration of their portfolio in the collaborative context of a senior capstone course, sharing their evidence-based understanding of the meaning of their learning with other students and the larger community.
Each of the portfolios students create generally share a common structure. Students are asked to organize their interpretations and evidence of their learning and performance in relationship to several of the nine NCC competencies: communication, critical thinking, strategic problem solving, valuing, group interaction, global understanding, effective citizenship, aesthetic awareness, and information technology {New Century College, 2005 #746}. In the final first-year and graduation portfolios, this takes the form of a series of short essays linked to diverse artifacts that capture learning activities. In addition to analyzing their learning in terms of competencies, students present an integrative picture of their learning that synthesizes the competencies and contextualizes them within the students’ own stories of themselves as learners. In the first-year and graduation portfolios, this integrative process is realized through one or more integrative essays that draw both on the competency material and additional evidence of the students’ own definition.
Unlike in the typical applications of electronic portfolios to general education assessment discussed earlier, the graduation portfolio is not intended to measure whether the student has met a pre-defined set of minimum standards of performance in relationship to each competency. The portfolio assessment is not a quality assurance mechanism. There is, for example, no pre-defined description of what a candidate for graduation must show they are able to do to be considered a competent critical thinker. Through their courses and in experiential learning experiences such as service learning and internships, students are exposed to a variety of different ways of understanding and evaluating each of the competencies. The curriculum embraces disciplinary and professional differences, with the conviction that competencies like aesthetic awareness may be conceived quite differently across a range of specific contexts, such examining a linear algebra proof, planning a fundraiser for an NGO, or describing changes to the countryside in the wake of urban sprawl.
Rather than traditional majors, NCC students have concentrations in multi-disciplinary areas of focus, some pre-defined by the faculty, such as Internet and New Media and Community Studies, and others crafted by the students themselves. Even within pre-defined concentrations, students have a great deal of choice as to which NCC learning communities and courses from across the University they take and which experiential learning experiences, such as internships, service learning projects, and study abroad, they engage. How and what students learn varies considerable from individual to individual, and NCC tends to attract students who benefit from this flexibility. Given this diversity of experience, the NCC faculty believe it would be both extremely difficult and ultimately counter-productive to prescriptively define rubrics for evaluating levels of development for each competency, even if such descriptions were indexed by concentration.
Much of the responsibility for making coherence across this diversity is undertaken by students. Virtually every NCC course includes reflective activities linked to the competencies that are designed to prepare students to examine and make effective choices about their education as a whole. Applying insights from this reflection is further facilitated by skilled group of academic advisors. In upper division learning communities, students explore the connections between the competencies and theories, concepts and methods they identify as important to their area of concentration and future goals. This competency-supported reflective practice culminates in students’ graduation portfolios, which they compose in a required capstone course. As with reflection in general, students have regular opportunities to compose portfolios throughout their time at NCC that prepare them for the final synthesis.
In the final first-year and graduation portfolios, students synthesize these varied perspectives on the competencies. Early in their treatment of each competency, students are asked not to show how they conform to the official NCC definition (which is brief and broad) but to redefine the competency for themselves, showing how their definition is appropriate to their own unique experiences and goals. They combine insights from the perspectives they’ve explored through course work, experiential, and informal learning over the course of their educational experience to show how their reflections and evidence of learning add up. Through this process of redefinition, students take ownership of the competency. Rather than simply internalizing NCC’s standards, they leave us with their own. These individualized competencies prove far more useful to graduates than any generalized set NCC could hope to develop.
Perhaps even more importantly, they demonstrate the ability to learn intentionally, integrating strategies and objectives from a wide range of sources independently to make smart choices about learning in new environments, while still remaining true to their own long-term goals and commitments. Indeed, the primary outcome the portfolios measure is not the ability, say, to think critically or be an effective citizen—although the portfolios provide rich and convincing evidence of these things—but the ability to learn. As Meeus, Petegem, and Looy argue, these learning competencies may in fact be the outcome of higher education portfolios are most powerfully able to capture {Meeus, 2006 #714}.
The rigor and persuasiveness of these demonstrations hinges on students use of evidence. Successful graduation portfolios are composed through careful examination and integration of evidence connected to ideas and theories students’ see as key to their concentrations. Students closely read and analyze their work from multiple contexts, using the theories and ideas to show how this work is situated within academic and professional communities of practice. Strong portfolios include both sophisticated descriptions, analysis, and judgment of the students’ work and contextualization of that work in terms of theories and techniques valued in both the communities within which they have learned and those they hope to enter. Because students within a concentration may see different sets of theories and ideas as central, depending on which communities of practice they’ve engaged, the competencies help students connect the content of their concentrations-as-experienced with a common way of thinking about the whole of undergraduate learning.
NCC’s success in preparing students to compelling integrate intellectual engagement with concentration-related content and reflection on their own work and experiences is uneven. Their ability to do this well seems largely to depend on the practice they get upper division learning communities, and some provide better practice than others. Those that succeed make the connections between the competencies and multiple perspectives on the area of concentration central to the curriculum. For example, in an upper-division course on migration, members of the learning community look at the process of Problem Solving (one of the NCC competencies) across four disciplinary approaches to migration and conservation, ranging from conservation biology to cultural studies. In another course on focused on family relationships, students use a variety of theories of Valuing (another NCC competency) within the family to reflect on and compare their experiences working with a variety of social service organizations in the Northern Virginia community. Ensuring that all upper division learning communities offer similar opportunities to students is a current focus of the NCC faculty’s work on curriculum development.
Throughout the curriculum, as well as in the culminating portfolio experiences, NCC asks students to continuously examine and develop their knowledge, skills, strategies, and values as they encounter new sources of information and function in diverse and rapidly changing environments. In order to live up to its promise as an ethical learning organization, NCC must not only ask students to engage in reflective practice but must also do so itself. Furthermore, putting the reflective philosophy of assessment into practice, this collective reflection must incorporate and value the evidence of learning and perspectives about what is valuable about a University education that students develop through their individual inquiries.
As boundary objects, the competencies connect the individual perspective, as represented in portfolios, with a community conversation about what it means to be an educated person. The competencies are left open enough to support the specificity of local practices needed to account for complexity of individual student experience and professional and disciplinary difference while provide enough common structure to enable these perspectives to be mutually intelligible, compared, put into dialog, and integrated. Shared understanding of the competencies and NCC’s effectiveness in helping students develop towards them is built through discussions in the capstone course taken by all students, brown-bag lunches open to all members of the community, and more formal examinations of specific competencies on reading days and at retreats. Student portfolios serve as the shared texts for discussions of the competencies, and students are invited to participate directly in all of these settings. The results of these conversations are highly valued by the NCC leadership and are primary drivers of changes to NCC’s curriculum and programs.
[but could be more systematic ]

Responsive, reflective assessment through the digital, networked medium

Portfolios were part of the founding vision of New Century College, prior to the widespread popularity of electronic portfolios, and NCC’s long experience with them undoubtedly contributes to its success in integrating independent, individual reflective learning within a learning organizational structure that is responsive to student values and experiences. While NCC students’ portfolios now mostly take electronic form, what has been described so far could apply equally well to both print and electronic portfolios. As NCC continues to build on its tradition, it is taking more extensive advantage of the distinctive affordances of digital and networked media for achieving its goals.
NCC’s print portfolio presented two primary challenges, related to synthesis and communication. First, in print, student portfolios at NCC have traditionally taken the form of a series of discrete essays with associated samples of work. While the individual essays made connections between the competencies, theories and ideas from the curriculum, and student work and experiences, students had limited opportunities to express their sense of the portfolio as an integral, synthesize whole, something that adds up to more than the sum of its parts. The portfolio itself was more of a collection than a composition in its own right. Even within the reflective essays, the formal challenge of writing within the form of a traditional essay sometimes distracts students’ energy away from making the connections that ought to be the heart of the portfolio. Offering other options for organization and articulation of connection has proved difficult in print.
Second, the print medium makes it more challenging to discuss and analyze both portfolios and the competencies in general. By default, print portfolios are private, shared only with one reader at a time. Prior to the introduction of the capstone course, the only readers for most graduation portfolios were the faculty members evaluating them. Keeping an archive of portfolios for subsequent analysis also is impractical. In capstone, portfolios can be shared within a community of learners to discuss the competencies. However, these conversations are only accessible to this particular group at a particular place and time, necessarily excluding many within the larger NCC community who ought to active in shaping how the competencies are defined and used. Even in more public venues, such a brown bag lunch discussions of the competencies, participation is limited by who can physically attend, making it very difficult for stakeholders such as alumni and the leaders of community organizations with which students work to have a voice.
Throughout its curriculum and programs, NCC is working to use the digital medium both the new possibilities it offers for expression and as a new space of building relationships. This push will likely result on all of NCC’s portfolio work moving online, which will result in a rich archive of information about student learning that could serve not only for reflective assessment, but also for the kinds of quality assurance around which many implementations of electronic portfolios in general education are centred. This article concludes by briefly describing two specific examples of how NCC is using digital media now to address the challenges it has encountered in its print portfolio practice, the first focused on individual integrative reflection and the second on bringing the community conversation online.

Digital Synthesis: Contextualizing competencies through concept maps

In NCLC 249, a mid-level class about doing research on how people use the Internet, instead of the traditional integrative essay students compose concept maps as the integrative interfaces to their portfolios. Student choose at least three of the NCC competencies, three of the more specific goals of the course goals, and a parallel number of key concepts they found particularly powerful in their individual experience of the course. In their concept maps, they create visual representations of how these concepts and competencies combine, linked with examples of their work from the course, both formal and informal, to form holistic representations of their learning over the course of the semester. This approach is especially appropriate for the newest generation of university students, many of whom are highly visual learners {Oblinger, 2003 #747}.
In addition to the overarching visual structure, students annotated the short descriptions of each link with a reflective paragraph providing a more detailed analysis of the relationship between the two components of the portfolio. (See Figure 1.) When the reader of the portfolio rolls the cursor over the label, the annotation pops up, provided both the global, integrated visual representation and the rich textual analysis simultaneously. This effect would be impossible to duplicate in print.
Rather than spending their time trying to represent their learning within the confines of a traditional expository essay, students concentrate on reflective writing focused on the relationships between the items within the portfolio that synthesize them into a whole greater than the sum of the parts. This process of relationship building in central to integrative practice {Repko, 2005 #721; Klein, 1996 #284}. The relationships themselves, not simply the collected evidence or the activity of reflection, is what makes a portfolio a portfolio {Cambridge, 2006 #685; Cambridge, 2005 #679}.
The presence of the NCC competencies within this highly individualized, student-owned picture of learning enables it to connect to the competency-driven community conversation about what matters in higher education. However, the competencies themselves do not significantly constrain the ways in which students may choose to tell their individual stories. Concept maps are also being used as alternate methods of integration in the capstone course and may be included in graduation portfolios.

Figure 1: Screen shot of a concept map portfolio for NCLC 249

Digital communication: Using portfolios to capture capstone conversations

The senior capstone course is also a key site for collective reflection on the competencies. Over a six week period, while composing and sharing the competency sections of their portfolios, groups of capstone students facilitate panel discussions about each of the nine competencies. Each panel includes current students, NCC faculty members, NCC alumni, and guests from around the University and local community, such the director of institutional assessment or the chief information officer of a local technology firm. Students previous reflections on the competencies and evidence they have collected to include in their graduation portfolios help guide the discussions, along with a set of shared texts and students’ research. Students document the conversations with through recording audio and/or video and produce a multimedia synthesis of the discussion that captures both emergent consensus and distinctive perspectives.
While in the past this integrative work was largely confined within the context of capstone, it is in the process of being made more widely accessible through a collaborative Capstone Conversations portfolio (14). The conversation results are linked to selections from graduation portfolios focused on each competency, making the collective process of reflection more visible and accessible to the larger NCC community.
The Capstone Conversations portfolio is a first step towards using networked spaces to facilitate organizational reflective learning. In future iterations of the capstone experience, NCC plans to conduct the capstone conversations in virtual environments, using newly available Web conferencing software, and to build more powerful mechanisms for dialog and feedback within the portfolio space itself. By continuing its tradition of innovative portfolio practice while tapping the power of digital and networked communication technologies, New Century College models the responsive relationship between reflective learners and learning organizations we hope our students help enact in the larger world.


Download 179.8 Kb.

Share with your friends:

The database is protected by copyright ©sckool.org 2020
send message

    Main page