Estrella Mountain Community College Student Academic Assessment Committee Composition / Writing Assessment Results Spring 2013 Background



Download 362 Kb.
Page1/3
Date20.01.2019
Size362 Kb.
#77426
  1   2   3
Estrella Mountain Community College

Student Academic Assessment Committee

Composition / Writing Assessment Results

Spring 2013

Background:
Compositional writing is one of seven Estrella Mountain Community College (EMCC) General Educational abilities. Starting in the fall 2010, EMCC faculty adopted a curriculum-integrated assessment approach to assess student abilities instead of standardized, national instruments. The spring 2013 semesters represents the first time the Compositional Writing ability has been assessed using the integrated assessment approach. Prior to the spring 2013 semester, the College Basic Academic Subjects Examination (CBASE) was used to assess student’s writing ability during the spring semesters of 2008, 2009, and 2010. The results on CBASE were less than desired. EMCC faculty collaborated to develop a writing-across-the-curriculum rubric that could be used by all faculty. A version of this rubric was employed for the spring 2013 assessment curriculum integrated assessment.
EMCC's Definition of Composition/Writing
The Composition/Writing assessment was developed to demonstrate student’s ability to understand the subject matter by creating a focused purpose statement followed by the appropriate support to sustain the document’s focus. Students are challenged to demonstrate clarity in the organization of thoughts and use of language through the proper utilization of mechanics and the ability to cite sources.
Purpose and Methodology:
The primary objectives of the assessment are to:


  • Establish initial data for EMCC’s writing outcome using curriculum-integrated assessment practices

  • Identify gaps in performance between the freshman and sophomore cohorts

  • Inform the faculty of our student body’s strengths and areas of needed improvement in writing, in order to help focus faculty members’ pedagogical improvements

  • Develop best practices for SAAC and the administration for future composition/writing assessment

During spring 2013, SAAC updated the existing writing rubric in preparation for the 2013 writing assessment, no longer using the CBASE assessment instrument. The spring 2013 assessment had faculty participants collect artifacts of students' writing ability and provide them to EMCC's Writing Center. Trained writing tutors used the EMCC Writing rubric (see Appendix A) to assess students' work on the basis of content, clarity and editing. The writing tutors critiqued content for subject matter, focus, and support issues. Clarity was evaluated on organization and language use issues. The writing tutors evaluated editing on the basis of format, citation, and mechanics issues. Instructor grades on the submitted documents were independent to the writing tutor scoring of the same documents.


Instructors were initially invited (see Appendix B) during fall 2012 to participate in the spring 2013 writing assessment. Guidelines for participation were as follows:


  • Faculty choose/create a writing assignment that meets the criteria

  • Students submit their assignment to their instructors electronically so that the assignments can be forwarded to the Online Writing Center (OWC)

  • Instructors shall grade/assess the papers for the purposes of their course, but keep an unmarked digital copy of the essay for submission to the OWC

  • Instructors create an electronic folder to store the digital essays

  • Faculty submit the folder electronically to the OWC

Primary participation criteria consider a writing assignment or prompt of sufficient breadth and depth and assessable using the Writing Rubric. The length of the assignment is sufficient to allow assessment of each component. Additional requirements for assessment standardization include:




  • 12 point Times New Roman font

  • Double Space with1” margins

  • Student’s MEID in title of document

Student papers were evaluated on both an ordinal scale and qualitative comments. The ordinal scale indicated the degree a writing sample addressed the necessary issues. Qualitative comments indicate why a score may have been assigned. The ordinal scale evaluated student writing for specific issues relating to content, clarity and editing. The ordinal scale rating:

4 = "The reader is easily able to understand and/or appreciate"


3 = "The reader is generally able to understand and/or appreciate"
2 = "The reader is often challenged to understand and/or appreciate"
1 = "The reader is seriously challenged to understand and/or appreciate"

The spring 2013 assessment was conducted across 10 classes (AJS123, ECN211, ECN212, EDU222, ENG081, ENG091, ENG101, ENH254, MAT217, SOC212), 15 sections, 8 instructors, and included 214 unique students. There were a total of 235 documents submitted for consideration in the composition / writing assessment. Some students had multiple submissions.


Results included within this report compare overall results of new freshman (0-7 cumulative earned hours at beginning of term) to sophomore students (30+ cumulative earned hours at beginning of term).
KEY FINDINGS:
Overall Findings: The EMCC Writing rubric was designed to assess students' work on the basis of content, clarity and editing. Clarity was the strongest of the three major categories with a mean score of 3.52, followed with a content mean score of 3.49. Editing was the weakest of the three major assessment categories, with a mean score of 3.21. Overall, sophomores outperformed freshmen with the exceptions of organization and formatting under the clarity component of the rubric. However, these differences were statistically significant for Editing. Both Citation and Mechanics resulted in statistically high mean score for sophomores vs. freshmen at p<.05. Table 1 reports on the mean scores.




Table 1:

Overall

New Freshmen

Sophomores

Content: (composite score)

3.49

3.46

3.56

Subject Matter

3.61

3.58

3.69

Focus

3.53

3.53

3.59

Support

3.33

3.28

3.42

Clarity: (composite score)

3.52

3.49

3.51

Organization

3.69

3.72

3.69

Language

3.34

3.27

3.40

Editing: (composite score)

3.21

2.93

3.18

Format

3.48

3.49

3.45

Citation *

2.99

2.83

3.12

Mechanics *

3.15

3.03

3.23

* Statistically Significant @ P<.05 (Citation at 0.033, Mechanics at 0.04)


Key Findings Regarding Content:
The overall mean score of (3.49 out 4.0) suggests a high level of student performance in the category of Content. Content was assessed in terms of subject matter issues, focus issues, and support issues. Student performance was highest for subject matter (3.61) and focus (3.53). The one area of content that suggest a need for improvement is support for the documents focus (3.33). Each of the content areas are explored in more detail.
Subject Matter Issues: The subject matter of the document should be appropriate for the assignment, adequate to complete the assignment, and suitable for the level of the assignment. Overall, the mean respondent score was 3.61; the new freshmen average was lower than sophomores (3.58 vs. 3.69) but the difference not statistically significant at p<.05. Table 2 also includes a frequency breakdown for the subject matter using the rubric categories. Of note is that more than 2 out of 3 (69%) sophomore students received the highest rubric rating.

Table 2 indicates the writing evaluator scoring regarding subject matter issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the subject matter.




Table 2: Subject Matter 

 

Student Level

 

 

New Freshmen

Sophomores

The reader is easily able to understand and/or appreciate

Count

50

58

% within Student Level

67.6%

69.9%

The reader is generally able to understand and/or appreciate

Count

19

24

% within Student Level

25.7%

28.9%

The reader is often challenged to understand and/or appreciate

Count

3

1

% within Student Level

4.1%

1.2%

The reader is seriously challenged to understand and/or appreciate

Count

2

0

% within Student Level

2.7%

.0%

Totals:

Count

74

83

 

% within Student Level

100.0%

100.0%


Additional Subject Matter Issues Feedback from the Writing Evaluators: The evaluators noted that the subject matter was NOT adequate to complete the assignment for 24.7% of all respondents (n=58). Sophomores recorded a marginally better percentage (20.5%, n=17) than new freshmen (23.0%, n=17).
Focus Issues: The focus (or main idea or thesis) should be apparent to the reader; presented so as to interest the reader, and targeted to achieve the purpose of the assignment. Overall, the average score for all respondents was 3.53; the new freshmen average was lower than sophomores (3.53 vs. 3.59) with the difference shy of statistical significance. Table 3 also includes a frequency breakdown for the subject matter using the rubric categories. Nearly 2 out of 3 (64%) sophomore students received the highest rubric rating.
Table 3 indicates the writing evaluator scoring regarding focus issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the focus.


Table 3: Focus  

 

Student Level

 

 

New Freshmen

Sophomores

The reader is easily able to understand and/or appreciate

Count

44

53

% within Student Level

59.5%

63.9%

The reader is generally able to understand and/or appreciate

Count

25

26

% within Student Level

33.8%

31.3%

The reader is often challenged to understand and/or appreciate

Count

5

4

% within Student Level

6.8%

4.8%

Totals:

Count

74

83

 

% within Student Level

100.0%

100.0%

Additional Focus Issues Feedback from the Writing Evaluators: According to writing tutors observances the focus (main idea or thesis) was NOT considered targeted enough to achieve the purpose of the assignment for 33.6% of all respondents (n=79). A similar percentage of sophomores students received the same comment (31.1%, n=26) marginally better percentage results (31.3%, n=23) compared to new freshmen.
Support Issues: The support should be appropriate to sustain the document’s focus, carefully chosen to meet the needs of the assignment and of sufficient quantity to carry out the development of the document. Overall, the average score for all respondents was 3.33; the new freshmen average was lower than sophomores
(3.28 vs. 3.42) but the difference is not statistically significant. Table 4 also includes a frequency breakdown for the subject matter using the rubric categories. Nearly 2 out of 3 (64%) sophomore students received the highest rubric rating.
Table 4 indicates the writing evaluator scoring regarding support issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the support.


 Table 4: Support

 

Student Level

 

 

New Freshmen

Sophomores

The reader is easily able to understand and/or appreciate

Count

32

40

% within Student Level

43.2%

48.2%

The reader is generally able to understand and/or appreciate

Count

33

38

% within Student Level

44.6%

45.8%

The reader is often challenged to understand and/or appreciate

Count

7

5

% within Student Level

9.5%

6.0%

The reader is seriously challenged to understand and/or appreciate

Count

2

0

% within Student Level

2.7%

.0%

Totals:

Count

74

83

 

% within Student Level

100.0%

100.0%


Additional Support Issues Feedback from the Writing Evaluators: The writing tutors found that the support is NOT of sufficient quantity to carry out the development of the document for 27.7% (n=65) and was NOT carefully chosen to meet the needs of the assignment for 23.4% of all respondents (n=55). Sophomores had better percentage scores (26.5%, n=22 and 21.7%, n=18) compared to new freshmen (32.4%, n=24 and 20.3%, n=15).
Key Findings Regarding Clarity:
Student content clarity was assessed in terms of document organization and language use. Student responses averaged between “generally” and “easily” able to understand or appreciate the clarity of the assessment. Students were more able to address organizational issues compared to language issues.
Organization Issues: The document should be developed in a way that shows how all material relates to the focus, shows how all development relates to other parts of the document, and provides adequate transitions to guide the reader and illustrate these relationships. The average score for all respondents was 3.69; with new freshmen averaging slightly higher than sophomores (3.72 vs. 3.69), however this is not statistically significant.
Table 5 indicates the writing evaluator scoring regarding clarity issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored marginally lower in being easily or generally able to understand and/or appreciate the clarity.


Table 5: Clarity

 

Student Level

 

 

New Freshmen

Sophomores

The reader is easily able to understand and/or appreciate

Count

54

59

% within Student Level

73.0%

71.1%

The reader is generally able to understand and/or appreciate

Count

19

22

% within Student Level

25.7%

26.5%

The reader is often challenged to understand and/or appreciate

Count

1

2

% within Student Level

1.4%

2.4%

Totals:

Count

74

83

 

% within Student Level

100.0%

100.0%


Additional Organizational Issues Feedback from the Writing Evaluators: Writing tutors indicated the document is NOT currently developed in a way that provides adequate transitions to guide the reader and illustrate these relationships for 14.0% of all respondents (n=33). Sophomores had marginally better percentage scores (10.8%, n=9) compared to new freshmen (16.2%, n=12).
Language Use Issues: The language of the document should aid the reader’s comprehension and appreciation by using organized paragraphing, clear and correct sentences, and appropriate word choice. The overall average score for respondents was 3.34; with new freshmen average slightly lower than sophomores (3.27 vs. 3.40). The difference is not considered statistically significant.
Table 6 indicates the writing evaluator scoring regarding language use issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored marginally higher in being easily or generally able to understand and/or appreciate the language use.


Table 6: Language Use

 

Student Level

 

 

New Freshmen

Sophomores

The reader is easily able to understand and/or appreciate

Count

27

40

% within Student Level

36.5%

48.2%

The reader is generally able to understand and/or appreciate

Count

40

36

% within Student Level

54.1%

43.4%

The reader is often challenged to understand and/or appreciate

Count

7

7

% within Student Level

9.5%

8.4%

Totals:

Count

74

83

 

% within Student Level

100.0%

100.0%


Additional Language Use Issues Feedback from the Writing Evaluators: The writing tutors noted that the language of the document DOES NOT aid the reader’s comprehension and appreciation by using appropriate word choice for 43.8% of all respondents (n=103). Sophomores had better percentage result (33.7%, n=28) compared to new freshmen (45.9%, n=34).


Key Findings Regarding Editing:
Student format and citation usage content was also assessed. Student responses averaged between “generally” and “easily” able to understand or appreciate the editing of the assessment. Students performed well with formatting issues but were less successful with mechanics and citation issues.
Format Issues: The format of the document should aid comprehension by being clear, consistent, and appropriate for the assignment. The average overall score for all respondents was 3.48; with new freshmen averaging slightly higher than sophomores (3.49 vs. 3.45); this is not statistically significant.
Table 7 indicates the writing evaluator scoring regarding format issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the format.


Table 7: Format

 

Student Level

 

 

New Freshmen

Sophomores

The reader is easily able to understand and/or appreciate

Count

43

41

% within Student Level

58.9%

50.0%

The reader is generally able to understand and/or appreciate

Count

24

37

% within Student Level

32.9%

45.1%

The reader is often challenged to understand and/or appreciate

Count

5

4

% within Student Level

6.8%

4.9%

The reader is seriously challenged to understand and/or appreciate

Count

1

0

% within Student Level

1.4%

.0%

Totals:

Count

73

82

 

% within Student Level

100.0%

100.0%


Additional Format Feedback from the Writing Evaluators: Writing tutors found that the document format is NOT assignment-appropriate for 37.0% of all respondents (n=87). New freshmen scored a higher percentage (43.4%, n=36) compared to sophomores (31.1%, n=23).
Citation Issues: The citation in the document should be complete, correct, and appropriate for the assignment. The average overall score for respondents was 2.99; the new freshmen average score was lower than sophomores (2.83 vs. 3.12), although not statistically significant. Note, however, that 25 freshman papers and 13 sophomore papers did not have a score for editing citation. This may be a result of the assignment submitted did not have a citation requirement.
Table 8 indicates the writing evaluator scoring regarding citation issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the citation.


Table 8: Citation 

 

Student Level

 

 

New Freshmen

Sophomores

The reader is easily able to understand and/or appreciate

Count

7

20

% within Student Level

14.6%

29.0%

The reader is generally able to understand and/or appreciate

Count

27

38

% within Student Level

56.3%

55.1%

The reader is often challenged to understand and/or appreciate

Count

13

10

% within Student Level

27.1%

14.5%

The reader is seriously challenged to understand and/or appreciate

Count

1

1

% within Student Level

2.1%

1.4%

Totals:

Count

48

69

 

% within Student Level




100.0%


Additional Citation Feedback from the Writing Evaluators: Writing tutors indicated citation in the document is neither complete for 32.3% (n=76) nor correct for 15.7% of all respondents (n=37). Sophomores performed better (32.5%, n=27) than new freshmen (36.5%, n=27) in terms of not being appropriate for the assignment. Sophomores scored poorer on the citation component (18.1%, n=15) compared to new freshmen (14.9%, n=11).
Mechanics Issues: The document should be comprehensively edited to ensure that the spelling is correct, that punctuation is properly used, and that grammar is correct. The overall average score for all respondents was 3.14; with new freshmen scoring lower than sophomores (3.03 vs. 3.23), this is not statistically significant.
Table 4 indicates the writing evaluator scoring regarding mechanics issues comparing new freshmen to sophomore responses, in terms of the count and percentage of total responses for the question. Sophomores scored higher in being easily or generally able to understand and/or appreciate the mechanics.


Table 9: Mechanics 

 

Student Level

 

 

New Freshmen

Sophomores

The reader is easily able to understand and/or appreciate

Count

15

27

% within Student Level

20.3%

32.9%

The reader is generally able to understand and/or appreciate

Count

46

47

% within Student Level

62.2%

57.3%

The reader is often challenged to understand and/or appreciate

Count

13

8

% within Student Level

17.6%

9.8%

Totals:

Count

74

82

% within Student Level

100.0%

100.0%


Additional Mechanics Feedback from the Writing Evaluators: The document is NOT considered comprehensively edited to ensure that that grammar is correct for 62.2% of all respondents (n=148), according to the writing tutors.
Next Steps:
The areas with the greatest opportunities for improvement are citation and writing mechanics. SAAC may wish to encourage instructors’ reinforcement of these areas. SAAC may also consider reinforcing that assignments to be submitted contain a citation requirement to ensure consistency of the writing samples. The next writing assessment is scheduled for spring 2016.

Appendix A: EMCC Writing Rubric


EMCC Writing Rubric Scale:

4: The reader is easily able to understand and/or appreciate

3: The reader is generally able to understand and/or appreciate

2: The reader is often challenged to understand and/or appreciate

1: The reader is seriously challenged to understand and/or appreciate





Scale

Points

Comments


Content










Subject Matter Issues:

The subject matter of the document should be appropriate for the assignment, adequate to complete the assignment, and suitable for the level of the assignment.



4

3

2



1

N.A.








Focus Issues:

The focus (or main idea or thesis) should be apparent to the reader, presented so as to interest the reader, and targeted to achieve the purpose of the assignment.




4

3

2



1

N.A








Support Issues:

The support should be appropriate to sustain the document’s focus, carefully chosen to meet the needs of the assignment and of sufficient quantity to carry out the development of the document.



4

3

2



1

N.A









Clarity










Organization Issues:

The document should be developed in a way that shows how all material relates to the focus, shows how all development relates to other parts of the document, and provides adequate transitions to guide the reader and illustrate these relationships.



4

3

2



1

N.A








Language Use Issues:

The language of the document should aid the reader’s comprehension and appreciation by using organized paragraphing, clear and correct sentences, and appropriate word choice.



4

3

2



1

N.A









Editing










Format Issues:

The format of the document should aid comprehension by being clear, consistent, and appropriate for the assignment.



4

3

2



1

N.A








Citation Issues:

The citation in the document should be complete, correct, and appropriate for the assignment.



4

3

2



1

N.A








Mechanics Issues:

The document should be comprehensively edited to ensure that the spelling is correct, that punctuation is properly used, and that grammar is correct.



4

3

2



1

N.A







Appendix B: Writing Participants Letter Spring 2013



Dear EMCC Faculty Member,

Thank you for signing up to have your students participate in EMCC’s across-the-curriculum writing assessment! We simply wouldn’t be able to conduct it without supporters like you. Below is some helpful information to make this experience as clear and easy as possible.


How to Participate

As you may know, we will submit our students’ writing samples to the Online Writing Center (OWC). To do this, please follow these steps:



  • Choose/create a writing assignment that meets the criteria (listed below).

  • Have students submit their assignment to you electronically, so that they can be forwarded to the OWC per their dictates.

  • Grade/assess the papers yourself however you want to for the purposes of your course, but keep an unmarked digital copy of the essay for submission to the OWC.

  • Create an electronic folder where you will keep the digital essays.

  • Submit the folder electronically to the OWC.


Criteria for Participation

The basic criterion for participation is that the writing assignment or prompt is of sufficient breadth and depth to be assessable using the Writing Rubric (see attached). That means that the length of the assignment is long enough to allow assessment of each component. Beyond that, we have some basic requirements for each essay so they are standardized for our assessors.



  • Font: Times New Roman

  • 12 point font

  • 1” margins

  • Double Space

  • Student’s MEID in the title of the document


Questions?

If you have any questions that spring up, please do not hesitate to contact Erik (erik.huntsinger@estrellamountain.edu; 5-8137) or Pete (peter.turner@estrellamountain.edu; 5-8454).


Sincerely, and with deep appreciation,

Pete & Erik






Download 362 Kb.

Share with your friends:
  1   2   3




The database is protected by copyright ©sckool.org 2023
send message

    Main page