Table 2: Immediate Implementation Tasks for Recommendation 2

Download 0.98 Mb.
Size0.98 Mb.
1   2   3   4   5   6   7

Table 2: Immediate Implementation Tasks for Recommendation 2

Task #






Consult with other states that have made the transition to technology-based assessment

CDE staff and other state agency staff

Discuss lessons learned in transition and dual-mode administrations in anticipation of 2014-2015

Incorporation of “Lessons Learned” from other states in future decisions


Consult with district technology directors and other technical experts across the state

CDE staff, district superintendents, and district technology directors

Establish a cross-section of district representatives with whom the CDE meets regularly to determine technology information needs and other preparation requirements

Discussions with district technology representatives


Establish mode participation protocols

CDE and Smarter Balanced

To reduce the capacity strain for resources and to efficiently problem-solve, identify participation protocols for mode (e.g., at a minimum, an entire grade must take the test on computer, or an entire school)

Mode participation protocols


Identify initial participation in paper vs. online administration

CDE staff, district superintendents, and district technology directors

Determine the expected participation rates to establish priorities for solutions during preparation

Initial participation rates


Evaluate technology readiness

CDE and Smarter Balanced

Determine guidelines and recommendations for schools and districts in administration in Year 1

Draft guidelines for choosing administration methods


Establish resolution protocols for Smarter Balanced field test

CDE and Smarter Balanced

With the district technology experts, establish a decision tree that will be used during field testing to reduce ambiguity when the field encounters administration or technology issues

Decision tree for administration and technology issues in field testing


Confirm school readiness for technology-based testing

CDE and Smarter Balanced

Require all schools that will participate in technology-based administration to engage in the field test to familiarize students and test the technology infrastructure

School readiness certification


Resolve issues from field testing

CDE and Smarter Balanced

Administration issues will arise during field testing. With the district technology experts, identify resolutions for these in preparation for the 2014-2015 administration

Resolved Issues Log for 2014-2015

Intermediate Considerations for Recommendation 2:

Though the results of the Smarter Balanced assessments are seemingly a long way off, it is not too early to begin communications with stakeholders about the reports from these assessments. While a focused communication with parents may be premature and cause confusion prior to the year of implementation, designing how that communication will take place is not premature. Likewise, continuing to inform the media about the expected reporting changes through current or developing relationships is time well spent. In addition to preparing for the first administrations of the Smarter Balanced assessments in ELA and mathematics, it would be appropriate for the state to begin considerations about how other content areas will be incorporated via technology-based assessments (TBAs) to maintain coherent operational systems in the schools.

2.1 Communication Documents to Stakeholders

A plan for communicating the change in the ELA and mathematics results to stakeholders would be helpful. While no doubt Smarter Balanced will provide information in this regard for states to use, it is unlikely to be tailored specifically to the changes in the California reporting structure. Communication plans that include mock-ups, two-minute overview web videos, and similar interactive communication media can be established now.

In addition, the state may wish to consider expanding the content of its media briefing prior to the release of the results in upcoming years. In addition to discussing the current year’s results, this briefing can also orient media to the anticipated changes in the program and results in 2014 and to the expectations for the first year of Smarter Balanced assessments in 2015. This preliminary work can foster relationships with media and their education beat reporters that would be helpful in the future. When there is no communications urgency, these relationships will provide opportunities for the media to develop background stories about the anticipated changes in the assessment program.

2.2 Technology-Based Assessments in Other Subjects

In committing to Smarter Balanced, California has also committed, at least conceptually, to the technical requirements of the test-administration engine of the Consortium: even if the state were to choose an alternate test engine than the Smarter Balanced open-source engine, the state must still demonstrate that the tests are identically rendered on the screen. With the challenges of transitioning to technology-delivered tests, it is very unlikely that either the state or the local schools would be willing to introduce two administration engines — one for the Smarter Balanced assessments and a second engine for other assessments that are technology-based.

The advantage in this administration challenge is that Smarter Balanced has committed to an open-source platform for technology-based assessment. This platform will be available to California and future contractors to administer any other assessments. The state will be able to require contractors administering other assessments in its system to use the open-source platform or other authorized variations that adhere to display, functionality, and interoperability standards set by Smarter Balanced. It is important that the CDE consider this a requirement in any future administration of TBAs once the open-source engine of Smarter Balanced is released. Thus, any intermediate-term work should conform to the Smarter Balanced interoperability standards, general system architecture, and display and functionality decisions.

Recommendation 3 – Use the Grade Eleven SBAC ELA and Mathematics Assessments as an Indicator of College Readiness

Use the grade eleven SBAC ELA and mathematics assessments to serve as the indicator of college readiness for entry into college credit-bearing courses, a task that is currently fulfilled through the CST/EAP assessments.

Meeting this recommendation will require time beyond the immediate term. Smarter Balanced has yet to administer the first operational administration of the assessments and, as such, the scores on the assessments indicating college readiness have not yet been identified. Since Smarter Balanced has yet to establish the cut score at grade 11, it has yet to be determined what level of proficiency will replace the Early Assessment Program (EAP) cut score. While this recommendation may not be completed in the next 18 months, California can begin the discussions with its stakeholders, especially those in state institutions of higher education (IHE), about this transition to the Smarter Balanced assessments as a replacement of the EAP mechanism.

Table 3: Immediate Implementation Tasks for Recommendation 3

Task #






College Readiness (CR) Review

CDE staff and IHE representatives

In collaboration with IHE representatives, review the Smarter Balanced content specifications and test blueprints against current EAP expectations

Discussions with IHE representatives on EAP expectations in Smarter Balanced assessments


Gap analysis investigation

CDE staff and IHE representatives

In collaboration with IHE representatives, determine what — if any – content remains necessary to satisfy IHE expectations previously covered by EAP

Gap analysis (if any)


Communication plan

CDE staff and IHE representatives

Draft communications for stakeholders on college readiness indicators — CCSS CR anchors, and as operationalized in Smarter Balanced grade 11 assessments

Communication plans


CCSS/CR professional development


Continue providing professional development on CCSS, in general and as pertains to CR

Professional development materials

Intermediate Considerations for Recommendation 3:

Currently, the California Standards Test (CST)/Early Assessment Program (EAP) assessments are used as an indicator of college readiness, which allows entrance into credit-bearing college level courses without need for remediation. These assessments reflect content assessed by a subset of the test questions on the spring ELA and Algebra II or Summative High School Mathematics CSTs augmented with a set of additional test items developed in collaboration with the CDE and California State Universities (CSU), and now recognized by the California Community College Chancellor’s Office (CCCCO).

As part of the Smarter Balanced assessment system, the Consortium will establish performance benchmarks for career and college readiness with input from K-12 educators and college and university faculty. Preliminary performance standards will be set following the spring 2014 field test and validated after the spring 2015 operational administration.

California has several options, described below, to explore in using the grade 11 Smarter Balanced assessments as indicators of college readiness.

3.1 California College Ready Indicators with Augmentation

California adopted the CCSS with state-specific additions. The first questions for consideration may be to what extent the state-specific additions reflect important aspects of college readiness as determined by California educators and college faculty and whether augmentation of the Smarter Balanced assessments will be needed for the purpose of using the grade 11 Smarter Balanced ELA and mathematics assessments as an indicator of college readiness. This would be the work necessary if the conversations with IHEs indicate there are additional needs that had been previously addressed in the EAP. If augmentation is needed, California would need to invest in additional test development and psychometric work to develop, scale, and establish college readiness performance standards, as applicable.

3.2 College Ready Indicators and System Migration to the CCSS

A second consideration concerns the transition to the CCSS and students’ access to and familiarity with the new standards. California educators are transitioning to the CCSS at varying rates, and the degree of alignment between instruction and assessment will depend on the degree of curriculum implementation. During this transition, it will be important to recognize that there may be an initial drop in test scores on the new Smarter Balanced assessments relative to past performance on the CSTs and the CST/EAP. Thus, there may be students who are capable of success in entry-level college classes who, because of this system transition, may not earn the required score on the new Smarter Balanced assessments. This is a recognizable concern across the Consortium. Therefore, as in the Consortium at large, we expect that the state will need to engage IHEs to evaluate and strategize around the implications of the Smarter Balanced college readiness results for instruction at the high school level and placement decisions at the college level.

3.3 College Ready Indicators and Technology-Based Assessments (TBAs)

A third, and related consideration, concerns the transition to TBAs and the extent to which the test administration is consistent with classroom instruction in terms of delivery mode and use of instructional technology. As with the system migration to the CCSS, this is a consortium-wide concern. Discrepancies between classroom instruction and assessment technology may introduce construct-irrelevant variance that negatively impacts student performance on the assessment for reasons unrelated to their mastery of the content domain of measurement interest. Thus, as with the transition to the CCSS, the state may wish to engage higher education as soon as California Smarter Balanced performance data are available to determine the implications for classroom instruction and college placement decisions. In coordination with Smarter Balanced resources made available from the Consortium, the state may also wish to provide additional resources to schools, teachers, and students to allow students and teachers plenty of opportunity to practice and interact with TBAs to minimize the construct-irrelevant variance that may be introduced by the technology aspect of the assessment.

3.4 Validity Evidence of College Ready Indicators

A fourth consideration relates to the ongoing evaluation of the use of grade 11 SBAC assessment results to indicate college readiness and the implementation of adjustments as needed. Evaluation would include, but not be limited to, validation studies to examine predictive and consequential validity. If applicable, these studies could be done at the SBAC level, with disaggregation at the state level. Alternatively, California may wish to conduct its own studies.

Recommendation 4 – Develop and Administer Science Assessments Aligned to the New Science Standards, Once Adopted

Develop new state science assessments consistent with new science standards, once adopted by the SBE in the fall of 2013, that include item types consistent with the SBAC assessments (e.g., short and extended constructed-response items and performance tasks).

California will need to consider several related components for new science assessments simultaneously. The Next Generation Science Standards (NGSS) have now been published behind the CCSS for mathematics and ELA, requiring a transition that would need to be coordinated with current communication and transition plans. California will also need to identify the manner of administration. For example, key questions must be answered regarding how similar or different the state science assessment will be from the Smarter Balanced assessments:

  • Will they use the same item types, or will science require special additions such as virtual laboratories?

  • Will they allow for use of the same delivery and management systems?

  • Will they be given at the same time of year?

  • Will there be differences in operational administration requirements and approaches?

  • Will they use the same or different scoring approaches?

In coordination with Recommendation 5, the state will also need to consider the alternate assessment in science for those students with the most significant cognitive disabilities, as well as whether California should collaborate with other states in the development of either the general education science test or the alternate science test. All of these tasks will require coherent orchestration.

Table 4: Immediate Implementation Tasks for Recommendation 4

Task #






Develop blueprints

CDE/Testing Contractor

Develop detailed blueprints (BP) for single grades in elementary, middle, and high school based on the NGSS. (Assume 5, 8, and 10 for discussion in this report.)

Draft blueprints at grades 5, 8, and 10


Develop test design

CDE/Testing Contractor

Consider delivery platform/use results from pilot to determine appropriate number and type of items for development; consider options being explored in other states.

Spreadsheet of test design


CDE review


Review and revision of blueprints and test design

Final blueprints


Achievement level descriptors (ALD) drafts

CDE, Testing Contractor, and ALD committee

Design draft ALDs to guide item development

Draft ALDs


Review blueprints and draft ALDs

CDE, Testing Contractor, Assessment Review Panels (ARPs), and SBE

Convene ARP to review and provide recommendations about proposed BPs and draft ALDs

ARP review documentation


New item development

CDE/Testing Contractor

Order and develop items prior to external BP review

Item development


ARP review of new items

CDE, Testing Contractor, ARPs

Conduct ARP review of new items

ARP recommendations of new item development


Recruit for field test (FT)

CDE/Testing Contractor

Complete sampling plan for field test; implement sampling plan with recruiting effort for school districts to participate in the field test

Field test sampling plan


Revise items and begin building FT forms

CDE/Testing Contractor

Apply edits, build paper-pencil forms

Initial online form construction


Complete FT forms building

CDE/Testing Contractor

Complete building forms and certification of print version

Finished forms


Review online forms


Conduct online review of forms

Confirmed online presentation of forms


Setup and test administration training for TBAs

CDE/Testing Contractor

Assist school districts with setting up the TBA system, including local troubleshooting and systems testing; conduct training of district test coordinators and other district personnel on test administration procedures and processes

Training workshops


Design standard-setting

CDE/Testing Contractor

Design a standard-setting plan; begin standard-setting committee recruiting, consulting with other states for comparability as appropriate

Standard-setting plan


Administer FT

CDE/Testing Contractor, school districts

Tests are administered, online and print

Completed FT


Conduct item Analysis

CDE/Testing Contractor

PIA and FIA completed, special studies done

Item statistics


Standard setting

CDE/Testing Contractor, standard setting committee,


Conduct initial standard setting — with other states, as appropriate, for comparability

Recommendations from standard-setting committee


Construct operational forms

CDE/Testing Contractor

Construct operational forms for the first live administration

Initial online form construction


Review online operational forms

CDE/Testing Contractor

Review online presentation of items

Confirmed online presentation of forms

Intermediate Considerations for Recommendation 4:

Implementing both the current version of the NGSS and the desired test-delivery methods will require a coordinated plan. How the science assessment aligns to the delivery and other components of the Smarter Balanced system — if it does so at all — should also be considered.

4.1 Release of the Next Generation Science Standards (NGSS)

The NGSS were released on April 9, 2013. We anticipate that California would implement an assessment aligned to these standards (if and when adopted by the state) no earlier than 2015-2016. As with the transition to the CCSS in ELA and mathematics, the state will have standards and curriculum transition activities that will need to precede the administration of any new science assessments. And prior to those transitions is the adoption of the NGSS by the state, as the Recommendation notes.

Given that the state has not adopted the NGSS, California may wish to consider targeting only those standards for continued item development that are common to the existing California standards and the NGSS, with an emphasis on content that paper-and-pencil testing has previously not been able to assess fully. If adoption is anticipated, the state could consider developing blueprints and item specifications based on the NGSS, as resources allow.

California can then engage in ongoing field test development to address gaps in coverage and further enhance the NGSS-based bank the state would build. This process would engage the CDE and ARPs for feedback and guidance.

4.2 Comparability of Online versus Paper-Pencil Testing (PPT)

Like most states, California is transitioning from paper-and-pencil testing (PPT) to TBAs. During this transition, there may be a need to provide assessments in both modes to allow time for districts and schools to develop the infrastructure needed to administer large-scale assessments online. To this end, TBA and PPT forms would be constructed to measure the same content and constructs.

One of the advantages of TBA is the ability to administer technology-enhanced (TE) and other innovative item types that are technology dependent, such as simulations. These tasks measure performance on standards that previously could not be assessed by means of PPT. Use of such items in TBA presents a challenge for dual-mode administrations in that some types of TE items may not be amenable to the development of paper-and-pencil equivalents. This raises the question of construct comparability across modes, with implications for score use.

The first issue to consider is the extent to which the TBA and PPT forms differ in terms of measurement properties. This can be evaluated at several points during the item development and field test phase using multiple approaches:

  • With regard to item development, expert review of TE items and paper substitutes as pertains to content and construct comparability is essential; expert review would continue with examination/evaluation of item performance at data review meetings, following completion of field test analysis.

  • With regard to construction of TBA and PPT forms, it is necessary to build both to satisfy the blueprint, noting use of TE equivalents in PPT forms as well as use of replacements for those technology-dependent items that do not have paper equivalents. Expert review of resulting forms for content and construct comparability will be needed, as will research studies to investigate score comparability and ongoing examination/evaluation of the psychometric characteristics of items.

  • Expert review and statistical results will inform score comparability across modes. Possible scenarios include the following:

  • If TBA and PPT forms are found to measure somewhat different constructs, treat them as separate and distinct and establish different scales and cut scores and do not compare the results. Alternatively, if in order to support comparisons between the two modes, then two approaches could be considered.

    • Articulate cut scores across modes to allow performance score comparisons (can be done at standard setting and is revisited periodically), or

    • Develop concordance tables to allow cross-mode comparisons at the scale score level.

  • If TBA and PPT forms are found to measure the same general construct, with some differences, treat as related and link TBA and PPT scores through concordance.

  • If TBA and PPT forms are found to measure the same constructs in the same way, treat as interchangeable and put TBA and PPT results on the same scale.

Note that the current question of TE equivalence may be informed by additional research that includes cognitive labs and small-scale pilot studies. Such research will happen both through the work of Smarter Balanced and through any additional studies the state may wish to conduct.

Recommendation 5 – Develop or Use Multistate Consortia Alternate Assessments in ELA, Mathematics, and Science for Students with Severe Cognitive Disabilities

Determine if the National Center and State Collaborative (NCSC) alternate assessment, once it is developed, is appropriate for California students and teachers. Should the NCSC assessment not be suitable, pursue alignment of CAPA to the CCSS using a variety of item types.

In addition to the unique challenges of building an appropriate alternate assessment based on rigorous content standards, the revision of California’s alternate assessments poses further transitional challenges. According to the NCSC website, the assessments being developed by the NCSC will not be available for at least a year after the implementation of Smarter Balanced assessments in those same content areas, with a census field test scheduled for the spring of 2015.

Table 5: Immediate Implementation Tasks for Recommendation 5

Task #






Review blueprints for NCSC assessment of CCSS

CDE/Testing Contractor

Review NCSC blueprints to determine potential use as ELA and mathematics alternate assessment

Blueprint analysis


Conduct NCSC field test

CDE/Testing Contractor, school districts

Participate in the NCSC field test in spring of 2015

Field test administration

Intermediate Considerations for Recommendation 5:

California will need to carefully plan the transition of the alternate assessment to align to the CCSS as efficiently as possible. The NCSC assessment can resolve this issue for the state and many consortia states if NCSC releases the operational alternate assessments on schedule, assumedly in the 2015-2016 school year.

5.1 Availability of the Next Generation Science Standards (NGSS)

As noted under Recommendation 4, the NGSS were published as a final document in April 2013. Currently, the two consortia developing alternate assessments based on alternate achievement standards are not developing a science assessment, so it is likely that California would need to develop a science assessment or perhaps work in collaboration with other states to do so. It would be most desirable that the science assessment mirror the design and administration features of the NCSC ELA and mathematics assessments as closely as possible to increase administrative efficiency and coordination in the field as well as to reduce confusion regarding procedures. Thus, whether a new alternate science assessment is developed by the state or by the current or a newly formed consortium, it will be important to keep in mind the design and logistical considerations that the field will experience in implementing the suite of ELA, mathematics, and science alternate assessments. If the differences become significant, it may create burdens of additional test administration training as well as additional administration time, which is not a goal of the revised assessment system.

Recommendation 6 – Determine the Continued Need and Purpose of Academic Assessments in Languages Other than English Once the SBAC Assessments Are Operational

Once SBAC assessments are fully developed and administered, consult with stakeholders and English learner experts to determine if stand-alone academic assessments in primary languages (languages other than English) are needed to supplement the SBAC assessments; and if so, determine the appropriate purpose for such assessments.

With the goal of student learning and achievement foremost, California should consider a balanced assessment package that includes large-scale assessment as well as classroom measurement, feedback tools, and professional development for all California educators. For students receiving instruction in a language other than English, Smarter Balanced is slated to deliver translated assessments in mathematics. However, California has a history of providing additional primary language content assessments (i.e., Standards-based Test in Spanish), which are developed in Spanish and not translated or transadapted. The state would need to coordinate any additional academic assessments in primary languages with the design and administration of the Smarter Balanced assessments. Recommendation 6 identifies the timeline for the activities to take place after the Smarter Balanced assessments are operational. Thus, there are no immediate activities required for this recommendation. However, the state may wish to engage in preliminary research and consideration with stakeholders as outlined below.

Table 6: Immediate Implementation Tasks for Recommendation 6

Task #






Full literature review

CDE/Testing Contractor

Identify research that will contribute to the deliberations for stakeholders once Smarter Balanced is operational

Literature review


Consultation with experts

CDE and national subject matter experts

Consult with state and national experts to review the literature



Review of advice of national experts


Review advice of national experts provided

Consideration of recommendations


Revision of recommendations


Revisions from stakeholders as well as any revisions to support additional language or academic content from Smarter Balanced

Revision based on expert and stakeholder recommendations


Stakeholder deliberations

CDE, SBE, and other state stakeholders

Begin stakeholder conversations based on preliminary findings and Smarter Balanced implementation

Stakeholder deliberation meetings

Download 0.98 Mb.

Share with your friends:
1   2   3   4   5   6   7

The database is protected by copyright © 2020
send message

    Main page