| Collegiate Seminar has institutional general education assessment responsibilities in addition to program-level learning outcome assessment and student-learning-level assessment needs. We would like to use qualitative analysis (emergent thematic coding) of a reflective essay assigned in all sections of the new seminar program to generate data for these levels of assessment. In addition, Seminar has to work closely with the Composition Program to align the writing goals across both programs to foster a developmental writing process. Finally, the Seminar Program has some historical weaknesses that Jose Feito, the current director, would like to address that fall outside of the current learning goals. Therefore the Collegiate Seminar Assessment Plan has five axes, which may entail some tweaking as each axis progresses. This plan primarily addresses the assessment of the learning goals and outcomes put into place to assess student learning, but it will outline all four aspects, and will refer to the other axes as needed.
6. Learning Outcomes. List the learning outcomes that have been developed for the program. If the outcomes will be further developed, please describe your plan for refinement.
Learning Outcomes. List the learning outcomes that have been developed for the program. If the outcomes will be further developed, please describe your plan for refinement.
Collegiate Seminar Learning Outcomes
Outcomes for Critical Thinking
Distinguish the multiple senses of a text (literal and beyond the literal).
Identify and understand assumptions, theses, and arguments that exist in the work of authors.
Evaluate and synthesize evidence in order to draw conclusions consistent with the text. Seek and identify confirming and opposing evidence relevant to original and existing theses.
Ask meaningful questions and originate plausible theses.
Critique and question the authority of texts, and explore the implications of those texts.
Outcomes for Shared Inquiry
Advance probing questions about a common text or other objects of study.
Pursue new and enriched understandings of the texts through sustained collaborative inquiry.
Reevaluate initial hypotheses in light of evidence and collaborative discussion with the goal of making considered judgments.
Engage in reflective listening and inclusive, respectful conversation.
Outcomes for Written and Oral Communication
Recognize and compose readable prose, as characterized by clear and careful organization, coherent paragraphs and well-constructed sentences that employ the conventions of Standard Written English and appropriate diction.
Recognize and formulate effective written and oral communication, giving appropriate consideration to audience, context, format, and textual evidence.
Analyze arguments so as to construct ones that are well supported (with appropriate use of textual evidence), are well reasoned, and are controlled by a thesis or exploratory question.
Use discussion and the process of writing to enhance intellectual discovery and unravel complexities of thought.
The outcomes above are set until assessed by our process of assessment below. There is a set of outcomes currently entitled “seminar-specific outcomes” that the UEPC rejected because some of them are not assessable. There is an on-going discussion about how much the Seminar reading list should reflect the intellectual history of the Catholic Church, and one of the outcomes in this group comes out of this sense. Others below are not assessable, and some are. I include them below because we may try to refine them to make them assessable, and would then add them into the assessment process at a later date. Outcome 2 about students making connections back and forth through the chronology of readings is especially important. The fourth goal below is also directly assessable in the reflective essay assignment that we will use to assess the learning goals. And we believe the first is evident from the reading list.
Seminar Specific Learning Outcomes: As a result of their participation in the Collegiate Seminar Program, students will grow in their ability to:
1. Understand, analyze, and evaluate challenging texts from different genres and periods.
2. Comprehend the intellectual threads that connect works both backward and forward through history.
3. Relate the works studied to their own experience and to notions of authentic humanity.
4. Reflect on prior knowledge and assess one's own process of learning.
Assessment Methods. Describe how at least one learning outcome will be assessed, including the selection of assessment method(s). At least one assessment method must be a direct measure of student learning (i.e., the direct review of student work).
We are planning to have a signature assignment for all 4 seminar courses. The draft assignment is a reflective essay that asks students to think about their progress and the role they take on in seminar: the assignment may see some slight revisions as more stakeholders read the draft.
Signature Draft Reflective Essay Assignment:
Throughout the four years of Seminar, we will be encouraging students to reflect upon and assess their own process of learning. Toward the end of Seminar 1, students will write a self-reflection essay that assesses their progress in the main learning outcomes of 1) Shared Inquiry, 2) Critical Thinking and 3) Written & Oral Communication. This self-assessment should involve some form of explicit dialogue between the student and the professor, either in conferences or in writing. The final version of the essay will be placed in the student’s electronic portfolio and be available to their future Seminar instructors.
All essays should:
be 1000-1200 words in length
explicitly address each of the three learning goals in general and at least one learning outcome in particular from each. Instructors are encouraged to allow the students themselves to decide which particular outcomes they would like to reflect upon under each of the three goals.
use specific evidence from informal and formal writing assignments, actual class discussions, experiences of reading, and reading notes.
explore how the student’s learning has impacted their behavior in seminar discussions and their role if the group’s shared inquiry process.
There is one feature though, that while it seems to be best for student learning, may prove unworkable for the assessment process. Right now, the essay allows students to choose which learning outcomes they reflect on under each goal. That is, they must reflect on one outcome from each of the learning goals, but may choose any outcome under each goal. The benefit of this is that students reflections will emerge from their own sense of what they are working on. But it means that we will have to choose what outcome to assess based on what the students choose to write about.
Alternatively, we could let them choose in the essay, but have a preparatory assignment that focuses on the one outcome that we wish to assess: in this case the first would be Shared Inquiry Goal #1: advance probing questions…. Imagine a one paragraph assignment that mirrors the essay assignment but asks for reflection on question formation with evidence deployed in the same ways as the essay. (We may also find that students gravitate to one outcome again and again. And plan for the upper division to have students reflect on outcomes they haven’t done before.) It may also turn out that some outcomes are relatively straightforward, and in that case we would modify the assignment to address the straightforward outcomes AND one of the others.
Once we have the essay results, we will do emergent thematic coding of the paragraphs on our given learning outcome: this will involve separating paragraphs of the essay into individual learning outcomes; read the set we want to assess; read through again and come up with preliminary codes; code the set with two people coding each paragraph, and a third if the first two coders disagree; we will probably want to compare the resulting themes to simple demographic data that tracks year, course grade, and when relevant in a year, previous grades in seminar. We will do a pilot study of these essays this December with a pool of about 44 students who took the pilot sections of seminar in fall 2012. The benefit of doing thematic coding is that we can see how students understand the outcomes’ functioning in their own learning process. After the pilot, we will have a rubric that we can use to track levels of this understanding. That rubric will show what kinds of development are happening over the four or two years of Seminar, and it will speak to the institutional general education core goals.
8. Use of Assessment Results. Describe your plan for faculty review and use of assessment results for the improvement of student learning.
At the Level of Student Learning: Once we see what the range is from this batch, we would like to develop rubrics that measure where students are and ought to be in terms of these outcomes for the purposes of assessing student learning.
At the Level of Program Assessment: We will also consider whether we need to make the teaching modules that we use in seminar more specific or more or less challenging; each outcome can be addressed through a few teaching modules, so we have multiple opportunities for fine-tuning both our learning outcomes and our teaching modules if it turns out that students are not meeting the benchmarks we have set.
At the Level of the Institution: We can use our direct measure of student learning, our reflective essay to see where our students begin and where they come out at the end of four years or two years for transfer students.
Our codes will be useful at all three levels of assessment: student learning, programmatic and institutional to track how students understand the learning goals in relation to their own learning.
9. Faculty Involvement. Describe how faculty will be involved in the assessment process. List the names and titles of faculty members who will be involved in the project.
Faculty drawn from the experienced instructors in the program, those on the Seminar Governing Board, and those who have been involved in the planning process for the new seminar courses. They will spend one day coding the paragraphs on the chosen learning outcome at the end of each semester. We may also invest in a software program of the CAQDAS variety to help with coding. For the pilot assessment, and assessment for the first year, we will code the paragraphs by hand. Once we have the codes, we can develop a rubric to track benchmark, cornerstone and capstone skill for each goal.
Sandra Grayson, full professor English
Rashaan Meneses, adjunct faculty, Seminar and Civic and Liberal Studies
Ellen Rigsby, associate professor, Communication and Seminar
Jose Feito, full Psychology and Director of Seminar
Jen Heung, associate professor, Anthropology and Seminar
One governing board member, yet to be determined
10. Sustainability. Describe how the assessment project's activities will become an ongoing process. Send a letter indicating a commitment to continuing the assessment process after the grant ends from your department chair/program director to Chris Procello, Academic Affairs.
We will receive and collate new data for the assessment every semester. Depending on the amount of essays needed to cull a reasonably sized dataset, we will assess our chosen goal (currently the questioning goal from shared inquiry) at the end of every semester, or just at the end of the spring if we need a larger sample of essays. We plan on assessing one learning outcome per year. This should be a doable cycle, especially if we can begin to use one of the qualitative software suites to do at least preliminary coding.
In addition Seminar must carry out the on-going projects concerning alignment with Composition; make use of the institutionally-gathered Critical Thinking data and the Writing data gathered by Composition. Once these process are routinized, the co-ordination of them can fall back to the program director, but in the meantime, a specialist is needed, hence this funding request.
It is conceivable that Seminar can continue to pay the stipends for faculty who help with the assessment at the end of the term out of its budget, once we know what kind of time is entailed and we can plan for it.
11. Project Timeline. Provide a detailed timeline indicating when various activities will take place and who will conduct them.
Collection of the essays with comments and collation of the relevant paragraphs will be done by the seminar administrative assistants (December 14 and May 24; assessment will occur the following weeks, December 17 and May 27 over the course of about 6 hours, 9am-3pm) with the pilot group of 45.
Depending on whether we mandate a reflection on a specific learning outcome, or we use the emergent model to let students choose what they reflect on, we will have a base pool of 38 sections to cull paragraphs from at the end of the spring.
Ellen Rigsby will coordinate the gathering of the raw data through moodle and turn-it-in; Jose Feito will assign administrative staff to cull the relevant paragraphs, and we will schedule a reading and norming session for the week of May 27. We will read the needed number of paragraphs twice over and a third time if there are disagreements.
If there is time that day, we will then compare the resulting codes to our modules and see how and whether students refer to them, what the strengths are of the modules and the students, and what the deficiencies are for students and modules relative to the goal. There is an extent that we see the teaching modules of seminar as standards for the outcomes, and the most important assessment task is to see if they are sufficient standards or not.
Ellen and Jose will then meet to create preliminary rubrics based on the data and the MIC committees will review the modules as needed. Please find below a copy of the modules.
The modules fall into the following three categories depending upon when and how often they should be addressed during the course.
Instructors should address these modules in the first 4 weeks of the term:
Pre-Reading – how do I prepare to engage with a new text?
Annotation – how do I track my responses to a text?
Finding a Voice – how do I manage to enter a discussion?
Listening – what are people saying and how am I responding to it?
Instructors should address these modules at least two times during the semester, building upon the previous work done in an iterative manner:
Close Reading – how do we focus in on the details of a text?
Questioning – how do I form meaningful questions about a text or during a discussion?
Genre – how does it change the way we read and talk about a text?
Instructors should address these at least once during Weeks 5-15 of the semester:
Standpoint – what particular perspective do I bring to the text?
Authority and Legitimacy – who gets to have an opinion? Why is their opinion important? Can it be challenged?
Disagreement – why do we do it and how do we do it skillfully?
Collaboration – how do we work together during a discussion? How do we track and integrate multiple voices in a text or a discussion?
12. Budget. Prepare a budget showing expenses in two categories: (1) Project director stipend and (2) other costs (e.g., resource materials, survey development).
$3000 director stipend—To create, plan and implement the learning outcome assessment plan for seminar to looks at student learning and programmatic assessment, and to coordinate the necessary institutional-level assessments with the Composition Program and the potential Critical Thinking Test with seminar outcomes assessment.
$500.00 resources materials—purchase and training for an inexpensive CAQDAS program such as MVSP or Nvivo8—the cheapest programs available.
$1500.00 for $250-500 stipends--for each member of the faculty that helps with the end of the year coding for MIC1/MIC2/Collegiate Seminar Governing Board members to assess learning outcomes at the end of each semester. We imagine 3-4 per semester after the initial pilot is carried out in December.
13. Total amount of funding requested: