Independent Evaluation of the California High School Exit Examination (CAHSEE):
2005 Evaluation Report
Lauress L. Wise
D. E. (Sunny) Becker
Carolyn DeMeyer Harris
Leslie R. Taylor
Christopher J. Johnstone [National Center for Educational Outcomes]
Nicole A. Miller [National Center for Educational Outcomes]
Sandra J. Thompson [National Center for Educational Outcomes]
Felicia L. Butler
Lisa E. Koger
Sally Gensberg [Professional Management and Evaluation Services, Inc.]
Robert A. Hilton
Nelson L. Kelley
Prepared for: California Department of Education
Contract Number: 00-07
September 30, 2005
Independent Evaluation of the California High School Exit Exam
In 1999, the California legislature established the requirement that students pass a graduation exam in English-language arts (ELA) and mathematics beginning with the Class of 2004. Some modifications to the requirement for the California High School Exit Examination (CAHSEE) were passed in 2002. (For more details on the bills establishing this test and the basis for continuing evaluations and reports, including this one, see Chapter 1 of this report.) In July 2003, after the completion of the 2002–03 CAHSEE testing, the State Board of Education (Board) voted to defer the CAHSEE requirement until 2006.
Over the six years since the CAHSEE was established by law, a wide range of information has been gathered, analyzed, and reported by the Human Resources Research Organization (HumRRO) and others. This report focuses on HumRRO’s information and analysis process during Year 6 of the CAHSEE evaluation. The findings have implications for most aspects of the CAHSEE, from the development of the test itself to how it is used and its impact on specific groups of students. Year 6 evaluation activities are reported under the following topics, each of which is summarized briefly here:
Review of the CAHSEE
Results from the 2004–05 CAHSEE Administrations
2005 Instruction Study
The trends in Educational Achievement and Persistence During the Era of CAHSEE
Study of Options for Students Receiving Special Education Services
The final chapter of the Year 6 report includes both a number of general policy recommendations and specific technical recommendations for further improving the CAHSEE and its use. These are presented below.
Review of the CAHSEE Examination
Review of the CAHSEE Test Questions
HumRRO conducted reviews of CAHSEE test questions in 2000, before the first form was developed, and again in 2002 after the first administration of CAHSEE to 10th graders. We conducted a third review of CAHSEE test questions during 2005. The new review addressed two key questions:
Do new forms of the CAHSEE, after revisions were introduced in 2004, still cover the targeted content standards completely and in sufficient depth?
Is the CAHSEE fair and accessible to English learners (EL) and students receiving special education services?
The review assessed: (a) the alignment of an intact operational test to the content standards (using Webb’s alignment method) and (b) how well the test questions conform to emerging principles of universal test design. (See Chapter 2 for a discussion of these principles for designing test questions that are fair and appropriate for all students.)
Key findings with respect to alignment.
Some issues were noted with the depth of knowledge of questions on the ELA test although the overall results showed acceptable alignment.
Reviewers wanted to use the essay responses to measure additional or different content standards beyond those in Writing Applications.
The depth of knowledge of the math questions matched the test content standards well; the test was not inappropriately easy or difficult.
Reviewers had difficulty matching test questions to the mathematical reasoning standards, which was not surprising since all of these questions also assessed content standards in other areas.
Key findings with respect to universal test design. In examining the perceived appropriateness of the CAHSEE questions for English learners and students receiving special education services, reviewers had some queries and comments about specific test questions. These were forwarded to CDE and the test developers for their consideration and review. Overall, the current item review process was judged to yield acceptable results.
Recommendations. Several recommendations for continued improvement of the CAHSEE item development process with respect to principles of universal test design include the following:
Ensure the CAHSEE is designed to optimize access by all groups of students.
Extend item-level analyses to include indicators of different problems for students receiving special education services.
Make changes to future CAHSEE tests at the whole-test level first.
Revisit regularly issues related to alignment between the tests and the California Content Standards.
Review of Psychometric Properties of the Exam
HumRRO conducted independent psychometric analyses of the February 2005 test results as a check on the processes used by the operational test contractor, Educational Testing Service (ETS). We used different software and programming, but reached the same results with respect to both item statistics and overall equating of the test scores.
We also examined the consistency with which the essays were scored in each of the 2004–05 administrations. We found the consistency to be equivalent to, or slightly better than the consistency in scoring essays from prior administrations and concluded that scoring accuracy was sufficient.
Results from the 2004–05 CAHSEE Administrations
All 10th grade students in the Class of 2007 were required to take the CAHSEE for the first time in February, March, or May of 2005. In addition, 11th graders from the Class of 2006 who had not yet passed both parts of the exam were given up to two opportunities to take the CAHSEE in any of the five administrations from September 2004 through May 2005. Detailed analyses of these results are presented in Chapter 3. Key findings are summarized here.
Consistency of Results
The results for 10th graders in the Class of 2007 were very similar to results for 10th graders in the Class of 2006. Passing rates improved slightly for the ELA exam and were about the same for the mathematics exam. Passing rates for different demographic groups were also largely unchanged. Students receiving special education services continued to have considerably more difficulty in passing the CAHSEE than all other groups of students.
Rates of Improvement/Failure
Students in the Class of 2006 who retested as 11th graders showed some improvement in their scores. About half of those testing each part had passed that part by the end of the 11th grade. Conversely, about half of those retested members of the Class of 2006 still have not passed. In addition, some unknown, but possibly large number of students who did not pass in 2004 appears not to have retested in 2005.
The Need for Consistent Statewide Identifiers
Due to the absence of a statewide system of unique student identifiers there were considerable difficulties in estimating the number of students in the Class of 2006 who have now passed both parts of the CAHSEE. Our best estimate is that 78 percent have passed both parts, although the true value could be one or two percentage points higher or lower. The estimated percentage is based on all students in the Class of 2006 who either passed in 2004 or who were still trying to pass during the 2004–05 school year.1 It excludes students who did not pass in 2004 and were retained in 10th grade, dropped out of school altogether, or did not attempt to retake the exam for some other reason.
Demographic Group Disparities in Passing Rates
There continue to be large disparities in passing rates for specific groups of students. Only 20 percent of 10th graders receiving special education services, 31 percent of English learners, 46 percent of African American students, and 51 percent of Hispanic students passed both parts of the CAHSEE, compared to 65 percent for all students. Estimates of cumulative passing rates through 11th grade for students in the Class of 2006 were 35 percent for students receiving special education services, 51 percent for English learners, 63 percent for African American students, and 68 percent for Hispanic students, compared to 78 percent overall.
Concentration of Lower Passing Rates in Certain Schools
We also examined differences among schools in CAHSEE score levels and passing rates. Minority and disadvantaged students in schools where there were high concentrations of such students had lower passing rates than their counterparts at other schools. We also began to examine models of how student, school, and district level characteristics jointly relate to student scores on the CAHSEE. Additional analyses of these models will be included in our February 2006 biennial report.
The 2005 Instruction Study
In 2003, we conducted a study as required by AB 1609 to determine whether standards-based instruction was sufficient to support the CAHSEE graduation requirement. We conducted a similar study in 2005 to provide updated information on the impact of instruction in preparing students to take the CAHSEE, and on the impact the CAHSEE requirement has had on instruction. The study involved surveys of district and school personnel, district executive summaries of instructional efforts related to the CAHSEE, and more than 500 interviews conducted at a selected sample of high schools and their feeder schools.
Impact of Instruction on CAHSEE
In Chapter 4 we report analysis of district, high school, and feeder school survey and interview responses to determine the impact of instructional trends on success on the CAHSEE. We also compare survey responses between schools with and without relatively high concentrations of at-risk students (i.e., English learners (EL), students receiving special education services (SD), economically disadvantaged, Hispanic, and African American).
Student preparation. We continue to find a substantial proportion of high school teachers reporting that students arrive unprepared for high school courses. Teachers most often cited student motivation, low parental support, and low student attendance as the factors that limit the effectiveness of the courses they teach. This effect was more pronounced for remedial courses than for other courses. Parental support was rated as a greater problem for required supplemental courses targeted to remediation than for any other course type.
Teacher credentialing. Among those factors that were significantly related to higher CAHSEE pass rates were teacher subject-area credentialing, years of teaching experience, and articulation between the feeder middle school and the high school, as well as coordination between special education and general education staff.
We investigated teacher credentialing and the assignment of subject-area credentialed teachers to courses and students. While three quarters of high schools report that nearly all their teachers hold appropriate credentials, in other schools at least a quarter of the teaching staff remains uncredentialed.
Over half of schools report using some mathematics teachers with emergency credentials.
A third of schools report some ELA teachers with emergency credentials.
While EL students reportedly receive instruction from credentialed teachers at nearly the same rate as all students, students receiving special education services are more likely to receive both ELA and mathematics instruction from a teacher who does not hold a subject-area credential.
ELA credentialing is lower in schools with high concentrations of African American students.
Lower percentages of schools with high concentrations of EL, economically disadvantaged, Hispanic, and African American students report math teachers with subject-area credentials than do schools without such high concentrations of at-risk students.
Student readiness for accountability. When interviewed, just over half of general education math and ELA teachers at both high school and feeder school levels stated that the Class of 2006 was ready to be held accountable to the CAHSEE graduation requirement. However, approximately half of special education and EL teachers believe their students are not ready to pass the CAHSEE, although a number stated that students need to be held accountable.
Impact of CAHSEE on Instruction
Increased alignment to standards. Our investigation of trends in California education that may have been influenced by the introduction of the CAHSEE requirement is reported in Chapter 5. Alignment of instruction to California Content Standards has increased steadily over the past several years at both the high school and middle school levels. Efforts are also underway to ensure that the level to which content standards are being taught is consistent across teachers. Nearly all high school and feeder middle school respondents identified one or more systems used to track student proficiency in the content standards.
Content-related professional development for teachers. Most high school and feeder middle school teachers have participated in content-related professional development. Further, schools have focused attention on remedial courses, as evidenced by the fact that assignment of high school teachers to teach remedial courses closely paralleled—and in some cases, exceeded—the education level and years of experience of teachers in related primary courses. High school department heads generally indicated their courses were demanding for students, although some differences were noted in schools with high concentrations of at-risk students.
Identifying/emulating successful programs. Some exemplary programs (e.g., Advancement via Individual Determination (AVID), Student Success Team (SST)) were identified through site visit interviews. These may warrant further targeted evaluation to determine whether they would be effective in additional schools.
Trends in Educational Achievement and Persistence During the Era of CAHSEE
Observed trends in important student outcomes over the past several years may reflect, in part, the far-reaching effects of the CAHSEE requirement for standards-based education and accountability. Since no students have yet been denied a high school diploma by virtue of not passing the CAHSEE, we provide baseline trend information in this report that will be augmented as the CAHSEE requirement takes hold.
Fears of increased attrition not realized
We analyzed enrollment levels, graduation rates, single-year and four-year dropout rates, participation in and performance on college entrance examinations, rates of completion of A–G courses, participation in and success on Advanced Placement (AP) exams, and enrollment rates of California high school graduates as first time freshmen in California college and university systems. One important trend reported previously for the Class of 2004, is that more rather than fewer students are progressing normally from 10th to 11th and 11th to 12th grade for the first high school class subject to the CAHSEE requirement. This trend has continued for the Class of 2006 through 11th grade.
Students’ perceptions of CAHSEE
We also inspected student responses to survey items administered with the CAHSEE for indications of trends in preparation for the exam and subsequent plans.
Increasing numbers take test seriously. Compared to the first time test takers in the Class of 2006, higher percentages of students in the Class of 2007 reported (a) perceiving the exam as important, (b) expectations of high school graduation, (c) plans to go to college, and (d) that CAHSEE test questions addressed topics that had been covered in coursework. Responses of students in the Class of 2006 who tested as sophomores (in 2004) and then retested as juniors (in 2005) were matched and compared. In the 2005 administration, these students were more likely to report perceiving the test as important, expecting to graduate from high school, and reporting they did as well as they could.
Disadvantaged students work harder to prepare. Disadvantaged students or those who did not pass the tests were more likely to (a) perceive the tests as more important to them, (b) make an extra effort besides regular course work to prepare for the tests, (c) indicate that the tests would make their graduating from high school harder, (d) be uncertain about their high school graduation, (e) be nervous when taking the tests, (f) be unfamiliar with the test questions, and (g) report that the test questions were more difficult than regular course work.
Test measures what is taught. A large majority of the ELA and math test takers reported that all or most of the topics included in the test questions were covered in their courses. Students who felt the tests were difficult, most often cited having “forgotten things” they were taught as the reason. About 10 percent of examinees reported that they “did not take the courses that covered these topics” and about 20 percent indicated having trouble with these topics when they were taught. Most test takers indicated that the tests were not more difficult than their course work.
Study of Options for Students Receiving Special Education Services
SB 964, passed in 2004, required a study of options for students receiving special education services who are unable to pass the CAHSEE. The report of this study was released in May 2005 (Rabinowitz et al., 2005). To provide further information on these options, we linked data on the services and programs received in special education programs with CAHSEE outcomes for individual students.
Many Special Education Students Can Pass
Our analyses revealed a strong relationship between the types of special education services a student receives and success on the CAHSEE. About half of the students examined received non-intensive services such as in-class accommodations or a resource specialist, and were able to spend more than 80 percent of their time in regular instruction. About half of these students passed the CAHSEE while still in 10th grade. Students receiving these services who had not passed in the 10th grade showed significant gains when they retested in the 11th grade. It seems likely that, with continued assistance, these students will have a good chance of meeting the CAHSEE requirement. It is thus reasonable to ask that both the schools and these students themselves continue to work to meet the required standards.
More Seriously Disabled Students Require Alternate Goals and Assessments
About one quarter of the students receiving special education services required more intensive assistance. These students participated in regular instruction less than 20 percent of the time, and only about 10 percent of them passed the CAHSEE during the 10th grade. Those who retested in the 11th grade showed only small gains in CAHSEE scores compared to all other students. These students receive services specified by Individualized Education Program (IEP) teams, who have statutory authority for making such judgments. There is no basis for second-guessing the services being provided to these students, although it is important to ask IEP teams to be sure student classifications are appropriate. It is less reasonable to hold these students responsible for mastering the skills assessed by the CAHSEE when they are not receiving instruction related to the skills tested by the exam. Alternate goals and some way of recognizing achievement of these alternate goals are needed for students in this second group.
Better Information and Analysis Required for Some
Another quarter of the students we examined received other combinations of services and showed mixed results on the CAHSEE. More detailed information on the needs of these students and the specific services provided is required to determine which ones have a reasonable chance of meeting the CAHSEE requirement.
Our general conclusion from these results is that it would be a mistake for legislators to impose a single set of alternatives on all students who receive special education services. Students who may be able to master the CAHSEE standards should not be lightly excused from doing so. Other students have little likelihood of mastering the CAHSEE standards and require different goals and options for recognizing accomplishment of these goals.
Policy makers face critical decisions about the CAHSEE as the Class of 2006 nears graduation. As in past years, we offer several general recommendations based on observations and findings from our evaluation activities. These recommendations are targeted to the Board and the Legislature as they consider additions or modifications to policies concerning the CAHSEE and its use. We also offer several more technical recommendations for the continued improvement of the CAHSEE, which are targeted to CDE and to the test developer.
Key Policy Recommendations
General Recommendation 1: Keep the CAHSEE requirement in place for the Class of 2006 and beyond.
Approximately 68,000 students who were not able to demonstrate mastery of essential skills in the 10th grade have been able to do so by the end of 11th grade. While we cannot offer solid evidence, it seems likely that many would not have done so without being identified through their scores as needing additional help and being motivated by the CAHSEE graduation requirement to take advantage of the assistance that was available to them. It is also evident that the requirement motivated schools to expand programs to help students master the required skills both before and after initial CAHSEE testing.
It would be a disservice to students, parents, and educators to send a message that some or all of the students in the Class of 2006 do not have to master language arts and mathematics skills deemed to be critical for success after high school.
General Recommendation 2: Identify specific options for students who are not able to satisfy the CAHSEE requirement and implement them by June 2006.
Nearly 100,000 students in the Class of 2006 did not satisfy the CAHSEE requirement by the end of the 11th grade. With continued effort and help many of these students will be able to satisfy the requirement in time to graduate with their class. However, many of these students, perhaps half, will not. To date, nearly half of English learners and nearly two thirds of students with disabilities have not met the CAHSEE requirement. Score gains from 10th to 11th grade were smaller for these students than for other students. If current trends prevail, a significant number of students including a substantial proportion of English learners and students with disabilities will not have passed the CAHSEE by the end of 12th grade. Many of these students will be denied a diploma for failing to meet other requirements as well.
Our second recommendation is that schools, districts, and the state provide options for students who want to earn a high school diploma but still do not pass the CAHSEE by the end of the 12th grade. We would urge consideration of multiple options to recognize the varying needs of students with different likelihoods of mastering the CAHSEE skills. Some of these may be interim steps while others may be required long term.
We differ strongly from the general conclusion of the SB 964 report (Rabinowitz et al., 2005) that the CAHSEE requirement should be deferred until alternative ways of demonstrating mastery of the standards and alternative diploma options for students unable to demonstrate mastery can be implemented with rigor. We believe it is better to keep the requirement in place and implement options now, improving technical rigor over time. The state should avoid sending the message that students should not continue to strive to master the essential skills, but rather provide options now for students who do not do so.
Some general principles in considering options are:
Insofar as possible, options should be available to all students who need them.
Options should not excuse students and schools from continued effort to develop and demonstrate the skills assessed by the CAHSEE.
Every effort possible should be made to help students master the targeted skills; alternative diploma options should be reserved for students who clearly cannot access the general education curriculum.
Alternative routes should be announced publicly.
Examples of options that could still be implemented for the Class of 2006 are:
Community College Program. Update community college programs that lead to a high school diploma to focus on the CAHSEE skills. Allow students who need it up to two more years to master the CAHSEE skills and receive a diploma through participation in these programs. One advantage of this approach is that it would provide students with instruction in a different setting, not just repeating instruction that did not work before.
Senior-Year Portfolio. Allow districts to develop and implement a senior-year portfolio project for students they believe have mastered the required skills but are unable to demonstrate this mastery on the CAHSEE during the 10th and 11th grade. Additional alternate forms of assessment might also be implemented this year if they can be imported from existing efforts.
Summer Course(s) After 12th Grade. Allow (and encourage) districts to develop a summer program for students who have not been able to pass the CAHSEE and grant diplomas to students who successfully complete this program. Separate ELA and math courses could be offered, with students required to take or pass courses only if they had not yet passed the corresponding test on the CAHSEE.
Additional Years of High School. By statute, students in special education programs can continue their high school education until the age of 22. This option might be expanded to allow other students to take an additional year or two of high school as well. This would be most reasonable if the opportunities provided go beyond the remedial programs to which the students already had access.
Establish an Alternate Diploma or Graduation Certificate. Many districts already offer certificates of completion or other ways of recognizing accomplishment short of meeting the full set of graduation requirements. California might establish a statewide program for recognizing the accomplishment of students who do not meet all diploma requirements but are able to demonstrate mastery of an alternate set of goals. For students in special education programs, this option might involve different mastery goals for each student. This alternative, however, would not have to be limited just to special education students. In the interim, an alternate diploma or certificate might be based on passing one part of the CAHSEE or scoring above a lower set of performance standards. Eventually, however, assessment of mastery should be targeted more directly to alternate content standards set for students unable to master the full content covered by the CAHSEE.
General Recommendation 3: Accelerate efforts to implement a statewide system of student identifiers and develop and maintain a database with information on students who have and have not satisfied the CAHSEE requirements.
It is unfortunate that policy makers have to wait for this report to get any estimate of how many students in the Class of 2006 have and have not satisfied the CAHSEE requirement. And even so, the estimates we provide are approximate and will be subject to some debate. More exact information on the numbers of students yet to meet the CAHSEE requirement for each high school class is needed to design programs to help these students and to estimate funding requirements for these programs.
General Recommendation 4: Collect data from districts on students who are not able to satisfy the CAHSEE requirement by June 2006 and use this information to further refine options for students having difficulty mastering the skills assessed by the CAHSEE.
An important policy question for evaluating the impact of the CAHSEE is how many students will be denied a diploma due to the CAHSEE requirement alone. Currently there is no statewide database with information on satisfaction of other graduation requirements, some of which may be district specific. While there is some uncertainty about who has met the CAHSEE requirement, there is also uncertainty as to how many students have met the algebra course or any other specific graduation requirement. Most schools review graduation requirements with students early in their senior year. With this information, they should be able to respond accurately to a statewide survey fielded in the later half of the school year. Alternatively, CDE might wait until after June to see how many students who were seeking a diploma were actually denied the diploma and why.
Specific Technical Recommendations
A number of more specific technical recommendations are discussed in Chapter 8 of the report. These recommendations are:
Specific Recommendation 1: A number of suggestions for improving specific test questions, particularly with respect to making them accessible to all students, were offered based on the item review. These might provide useful insights as the test development contractor continues to improve and enhance its item development and review procedures.
Specific Recommendation 2: Statistical review of test items should include checks for differential item functioning for students with disabilities.
Specific Recommendation 3: Information on the curriculum and services received by students in special education programs was quite useful. CDE may want to link this information to CAHSEE results on a more regular basis.
Specific Recommendation 4: Conduct a field trial or demonstration project with a small number of districts that already use student identification codes to model the design and use of detailed student data.