I Hereby Signify My Approval to Transmit This Response to Standards 19-21 Document to the California Commission on Teacher Credentialing:
Program Standard 19: Assessment Administered for Validity, Accuracy and Fairness
The sponsor of the professional teacher preparation program implements the Teaching Performance Assessment according to the assessment design. In the program, candidate responses to pedagogical assessment tasks are scored in a manner that ensures strong consistency of scoring among assessors, particularly in relation to the established passing standard. The program sponsor periodically monitors the administration, scoring and results of the assessment to ensure equitable treatment of candidates. Prior to initial assessment, each candidate receives the Teaching Performance Expectations and clear, accurate information about the nature of the assessment and the pedagogical tasks.
The sponsor of the professional teacher preparation program implements the Teaching Performance Assessment according to the assessment design.
The SB 2042 requirement that programs of education transition to an outcome-based philosophy has been adopted and implemented by the Biola University School of Education. Under Biola’s SB 2042 submission, the School of Education obtained approval to offer programs that could result in a multi-subject, single-subject, or internship credential. Though each of the credential programs is distinct, all candidates move sequentially through the following courses: at the undergraduate or graduate level:
LEDU 301 Introduction to Teaching and SEED 519 Foundations of Education (These courses are the prerequisite to full admission into the teacher education program and are not cross-listed. LEDU is for undergraduate students while SEED 519 is for graduate students.)
LEDU 330 /SEED 526 Psychological Foundations of Education
LEDU 420 /SEED 520 Elementary Reading Language Arts (multiple subject) or LEDU 425/SEED 525 Secondary Content Area Reading (single subject)
LEDU 341 /SEED 541 Methods of Teaching Linguistically Diverse Students
LEDU 430/SEED 505 Elementary Curriculum (multiple subject) or LEDU 435 /SEED 506 Secondary Curriculum (single subject)
LEDU 440-442 /SEED 512-513 Elementary Student Teaching I-II (multiple subject); LEDU 450-452/SEED 514-515 Secondary Student Teaching I & II (single subject), or SEED 581A-D, Intern Fieldwork I-IV (multiple or single subject)
Biola Teacher Candidates are made aware of the California Teaching Performance Assessment (CA TPA) during the Introduction to Teaching /Foundation of Education Courses. Within these courses, professors lead candidates in an examination of the Teaching Performance Expectation (TPE) standards. Once the candidate’s are familiar with the expectations, the professors assist the candidates in developing an electronic portfolio database that will house all key assignments. A critical component professors embed within their clarification presentations is a review of the manner in which the CA TPA contributes to Biola’s School of Education assessment program. Finally, the professors of the Introduction to Teaching course provide candidates with a copy and an overview of the Teaching Performance Assessment Candidate Handbook (see Appendix A).
Under our temporary Standard 19 submission, Biola University committed to using the CA TPA program. As a result, Biola’s first administration of the TPA tasks occurred in the fall of 2004. As data from the CA TPA tasks has been evaluated, Biola University faculty realized a need to design additional assignments that will prepare the candidates to fully develop the TPE skills embedded within the four CA TPA tasks. After attempting several different minor instructional adjustments, Biola University faculty realized in the spring of 2007 that our teacher preparation program’s scope and sequence had to be reviewed and modified. During our initial conversations, it quickly became apparent that developing program outcomes with key assignments would provide the organizational structure that would afford our candidates the practice necessary to master the skills identified within the TPE standards. As the process unfolded, Biola University School of Education faculty focused on integrating the aspect of TPE twelve which requires candidates to understand the manner in which their biases positively and negatively impact students. Biola University School of Education’s program outcomes were adopted in the spring of 2007. The initial key assignments were identified and developed during the fall 2007 semester. Current plans focus on field testing the key assignments in the spring of 2008 with full implementation occurring during the fall of 2008 (see page 49 in Appendix A for an overview of the Teacher Preparation Student Teaching Outcomes).
Biola University’s Teacher Preparation Program administers the CA TPA tasks in the following order:
TPA #1, Subject Specific Pedagogy, during the fourth course LEDU 341 /SEED 541 Methods of Teaching Linguistically Diverse Students.
TPA #2, Designing Instruction, during the fifth course LEDU 430/SEED 505 (elementary), LEDU 435 /SEED 506 (single-subject) Curriculum.
TPA #3, Assessing Learning, and TPA #4, Culminating Teaching Experience, during the final course LEDU 440-442 /SEED 512-513 (elementary), LEDU 450-452/SEED 514-515 (single-subject) Student Teaching or SEED 581A-D, Intern Fieldwork I-IV.
In effort to ensure that the teacher candidates receive an appropriate level of support, the Biola University Teacher Preparation program agreed to define embedded as making each CA TPA task a course assignment. This resulted in each task-embedded course’s professor of record being assigned the responsibility of ensuring candidate readiness. As candidate preparation strategies evolved, course professors of record expanded their instructional strategies to include an examination of the task’s requirements, a review of the related TPE skills, the highlighting of essential developmental theories, and multiple opportunities to strengthen analytical reflection proficiencies. To make instruction concrete, professors are encouraged to provide benchmark cases for candidate review. Finally to ensure that the candidate’s outside-class time is respected and to prevent the dissidence caused by a TPE score that is inconsistent with a final course grade, the professor of records have agreed to use the score of the task as part of the final course grade (See pages 54-55, 58, 62, 64, 70-74, 76-77, 82-84, 88-90, 94-96 in Appendix A for specifics regarding integration into courses).
The team approach to introducing concepts, practicing skills, and administering the CA TPA tasks created a need to provide accessible and appropriately detailed information. Through a review of current data management procedures, it was determined that current implementation of Biola University Teacher Preparation Program’s FileMaker Pro data-management system would provide a strong foundation for the new record keeping requirements. The data-management program currently in place included five unique screens where relevant information for each candidate was kept. The opening screen entitled “Undergrad/Grad Info” provided a place to record the candidate name, their entry date into the program, and a summary of conversations that occurred during each advising appointment. Simply requiring professors of CA TPA embedded courses to record summaries of CA TPA advising meetings within this field would ensure the necessary documentation regarding support, advice and/or granted modifications to procedures was maintained. In addition to providing a general advising record, Biola University Teacher Preparation Program’s FileMaker Pro database included the screen entitled “Personal Data” which would allow the CA TPA Coordinator to desegregate data by credential focus, candidate age, and/or ethnicity as well as ensuring that continued communication with candidates was sustained (see Appendix B). Thus, developing institutional capacity for CA TPA recording keeping simply required the development of a “TPA Info” screen where candidate task specific and cumulative scores as well as CA TPA task-specific formal actions could be recorded and monitored (see Appendix B).
Biola University Teacher Preparation Program’s commitment to utilizing a team approach moves beyond professor agreement to record keeping to sharing information through a unified dedication to establish procedures whereby joint decision making is a hallmark of Program meetings. As the complexity of the CA TPA practices have grown, agenda items related to the CA TPA have become common. Consistent energy is placed on evaluating the impact of current instructional decisions as well as the impact on a candidate’s life (see Appendix C – minutes for September 18, 2007). Furthermore, when evaluating candidate appeals, consideration is given to the impact an individualized adaptation revision will have on the candidate’s ability to successfully complete CA TPA expectations (see Appendix C – minutes for April 17, 2007).
In summary, Biola University Teacher Preparation Program has implemented the CA TPA by sequentially embedding tasks into distinct courses. Ensuring that essential information is readily available for advising has been accomplished through a modification to the FileMaker Pro database which also provides the information necessary to disaggregate achievement statistics. Finally, institutional capacity is ensured by assigning the Teacher Preparation Program committee responsibility in reviewing CA TPA policies and procedures as a means for making collaborative program decisions.
Appendix C: Embedding CA TPA Data into Program Decisions
In the program, candidate responses to pedagogical assessment tasks are scored in a manner that ensures strong consistency of scoring among assessors, particularly in relation to the established passing standard.
Ensuring the accurate scoring of CA TPA tasks is a multi-step process which begins with the careful monitoring of the candidate’s submission. Being a small program with high tuition costs, reducing additional expenses to candidates requires Biola University’s Teacher Preparation program to rely, in part, on using professors and student teaching supervisors as assessors. The potential for conflict of interest to arise due to the relationship between the candidate and the assessor is a reality that must be addressed. Several intentional procedures have been put into place to minimize unfair scoring. First, the professor of record assumes responsibility for reviewing the Task Submission Checklists (see Appendix D) with the candidates. This checklist clearly provides the candidate with specific guidelines that ensures a commonality of appearance for each task submitted. By eliminating a candidate’s ability to creatively utilize different formatting options, the ability of an assessor to speculation of task ownership based on appearance has been reduced.
Second, the checklist (see Appendix D) clarifies for the candidate the need for anonymity by reminding the candidate to only identify themselves by student ID number. Further the checklist states that documents where the candidate’s name appears, such as the Code of Honor and applicable release forms, are to be bundled and submitted separately so that these forms can be filed before the task is passed on to the assessors. In reviewing the strengths and weakness associated with the adoption of using a candidate’s ID number for identification purposes revealed that a professor’s access to the FileMaker Pro database could produce a situation where curiosity would compromise the program’s goal for blind scoring. To resolve the database dilemma, a commitment was made to use professors only as second assessors. Since university supervisors work with a minimum number of candidates it is easy to ensure that a supervisor never receives the tasks of a candidate with whom they have a relationship. Additional, the policy preventing supervisor access to the FileMaker Pro database ensures candidate anonymity is maintained.
Finally, the checklist (see Appendix D) places a certain degree of responsibility on the candidate by creating policies that include significant penalties for not meeting expectations. The stated consequences were created to explicitly remind the candidates of the significant role the CA TPA tasks have in the credentialing process. Though mercy will be administered to candidates who have experienced unexpected circumstances, the failure to fully abide by submission guidelines will result in the task being returned to the candidate so as to protect the integrity of the assessors. Furthermore, the checklist allows assessors to score without a sense of sympathy regarding what was submitted because the assessor has been assured that the candidate was reminded of the need to assume responsibility for submission completeness.
Assigning tasks to assessors is the responsibility of the Biola University’s CA TPA Coordinator. The assignment process begins with a review of the CA TPA Assessor Chart (see Appendix E) which organizes assessors who are in positions that result in interaction with the candidates as faculty and lists assessors who have no interaction with candidates in the middle of the page under the generic title assessor. After identifying the pool of assessors, the CA TPA coordinator contacts each assessor via mail to determine who will be scoring and the number of tasks they desire to score (see Appendix E). Based on the response, the CA TPA Coordinator uses the CA TPA Assignment and Score Level Record form (see Appendix E) to assign and distribute tasks to the various assessors. The first scorer column in the CA TPA Assignment and Score Level Record form is filled-in with the number of the various assessors who have no interaction with candidates while second scorer column is filled-in with the number of the different faculty status assessors. In cases where there are not enough non-faculty assessors, the CA TPA Coordinator will seek to assign a faculty-status assessor to a candidate in which no interaction has occurred. In the rare case where a faculty-status assessor needs to score the task of a candidate with whom interaction has occurred, the CA TPA Coordinator will ensure that this candidate’s task is automatically double scoring.
As a rule, Biola University’s Teacher Preparation Program requires that 50 percent of the submitted tasks be doubled scored. Determining which tasks are doubled scored begins with a random selection process that involves sequentially organizing the candidates’ second paper copy submissions and then going through the pile pulling out selected number tasks (e.g., every second, third, fourth, etc) until the fifty percent number is reached. Since Biola University’s Teacher Preparation Program has selected a passing rate of three or higher on every task, any task that was initially scored once and awarded a score less than three will be added to the list of tasks that are doubled scored. If during the process of double scoring, a task receives two different scores, a third assessment of the task will be completed. Typically, it is the responsibility of Biola University’s CA TPA Coordinator to be the third assessor.
The CA TPA Assignment and Score Level Record form (see Appendix E) also serves as the foundational document for determining an assessor’s ability to reliably score a CA TPA task. By disaggregating data according to assessor number, Biola University’s CA TPA Coordinator is able to monitor assessor accuracy (see Appendix F). The CA TPA Coordinator begins the assessor evaluation process by determining the semester statistics found within the Summary of Number of Times a Task was Scored chart (see Appendix F). Upon completion of the chart, the CA TPA Coordinator first verifies that no more than fifty percent of tasks were single scored. Next, the CA TPA Coordinator reviews the number of tasks that were awarded a middle score (i.e., tasks that are tripled scored and receive three distinct scores). If less than ten percent of the total tasks scored received a middle score, a reliable scoring session is declared. If more than ten percent received a middle score the Assessor Scoring Records (see Appendix F) are scrutinized with the intent of discerning which assessor had an unreliable semester. Any task singled scored by an identified unreliable assessor will automatically be double scored.
Before distributing the scores to candidates, the Assessor Scoring Records (see Appendix F) is used to complete a final check for consistency of scoring. Based on the original training calibration rate established by the State (matching a minimum of 50% and no more than one task off by more than one), Biola University’s CA TPA Coordinator reviews the verified scores columns for each assessor. Any assessor who has verified scores match or succeed the State’s original calibration rate is considered accurate. Any assessor who is below the State level will have the remainder of the single scored tasks re-assessed.
In an effort to monitor and manage trends, semester data is further recorded in the Summary of Number of Times a Task was Scored Distribution Record chart (see Appendix F). Any significant movement in percentages will result in a further evaluation of scoring policies and procedures with the intent of determining areas where needed adjustments must be made. Finally, each semester assessors will be given their Individual Scoring Record (see Appendix F) as a tool to verify their calibration success and struggles.
In cases where a candidate believes that they received an unfair assessment, an appointment with a School administrator, preferably the CA TPA Coordinator will be required. During the meeting, the candidate and administrator will review the appeal form (see Appendix G). Focus will be placed on helping the candidate to use the rubric to clearly articulate reasons that would support their claim of an unfair assessment. Regardless of the outcome of the initial conversation, the candidate will be informed of next steps (letter of request, submission of a fee, re-scoring, adjustment of student records). It is our desire to use this meeting as a means of ensuring the fair and equitable treatment of our candidates as well as securing data regarding areas where the Biola University Teacher Preparation Program may need to become more sensitive. Appeal forms will be signed and photo-copied with the original being given to the candidate and the reproduction being placed in the candidate’s file.
Appendix D: Task Submission Checklists
Appendix E: Assessor Assignments
Appendix F: Maintaining Scoring Accuracy
Appendix G: TPA Score Appeal Process
The program sponsor periodically monitors the administration, scoring and results of the assessment to ensure equitable treatment of candidates.
Though Biola University’s Teacher Preparation Program requires candidates to score a three or higher on each CA TPA task as a condition for meeting preliminary credentialing requirements, the faculty members of the program also believe that the tasks produce an incomplete picture of a candidate’s ability. As a result, candidates in Biola University’s Teacher Preparation Program are asked to demonstrate proficiency through the successful completion of many different developmentally-sequenced formative and summative assignments (see Appendix H). Thus, to ensure that the time and energy needed to complete all program requirements are compatible with the time and energy needed to work on the CA TPA tasks the following intentional steps have been implemented:
Teacher Preparation Program agenda items that focus on performance results (see Appendix C) so that each professor remains aware of the successes and struggles of the candidates
Teacher Preparation Program agenda items that focus on implementation policies and procedures (see Appendix C) with the intent of seeking agreement that will ensure consistent implementation.
Semester memos sent directly to professors of CA TPA embedded courses (see Appendix I) reminding them of essential professorial administrative and instructional tasks that have been discussed and approved for implementation.
Verifying that each professor has fulfilled their CA TPA responsibilities is complex. The legal and institutional requirements associated with a candidate’s right to privacy as well as a professor’s privilege to enjoy academic freedom prevents a dictatorial approach to administration. However, as is demonstrated by Biola University’s commitment and success to providing disability services, creating a cooperative environment that seeks to guarantee fair and equitable access to academic services is possible. Thus, Biola University’s CA TPA system is dedicated to building off from the policies and procedures already in place. Specifically, each professor will remind candidates of University services through an announcement in their syllabi (see Appendix A). Next each education professor’s commitment to model best practice pedagogy has already created a culture where it is common for professors to make adaptations that utilize candidate strengths. As a result, the requirement in the Professor List of Responsibilities (see Appendix I) for each professor to inform the CA TPA Coordinator of any candidate that needs adaptations is requesting a small change to current practices that will not feel inappropriate or intrusive. The ability for the CA TPA Coordinator to record a summary of agreed-upon adaptations into the candidate’s FileMaker Pro CA TPA screen (see Appendix B) ensures the construction of usable documentation. Finally, starting in the fall of 2007, the CA TPA Coordinator assumed responsibility in preparing a report that disaggregates CA TPA passing rates based on ethnicity and special adaptation status (see Appendix J). With the implementation of the Disaggregate Passing Rate chart, any significant deviation from Biola University’s overall and/or summary of the last five administration statistics can now be further examined. Unacceptable deviations from the norm will become a corrective action agenda item for the Biola University Teacher Preparation committee.
Biola University Teacher Preparation Program’s concern for equity moves beyond gender, ethnicity, and disabilities equity/access issues. It is the goal of our program to assist each candidate in fully developing their individual gifts. An essential step in making sure that each CA TPA task is scored fairly, despite the probability that candidates will provide a great variance within their responses, depends upon the assessor’s ability to continuously return to the TPE criteria. A tool designed to assist the assessors in maintaining a focus on the TPE standards is the Official Student Report (see Appendix K). This report, completed after the assessor has finished all Record of Evidence obligations, requires the assessor to provide each candidate with a list of specific TPE skills that were not apparent within the tasks in a clear, consistent, and connected manner. An unanticipated benefit of the Official Student Report has been its usefulness in providing insight into potential reasons why an assessor’s calibration may be drifting. In instances where the Biola University’s CA TPA Coordinator has identified an assessor that is neglecting to fully evaluate specific TPE standards, a meeting will be called so that the assessor can be assisted with calibration alignment.
The Official Student Report (see Appendix K) also allows the Biola University CA TPA coordinator to generate, on a semester basis, data that identifies the level of success to which each TPE skill has been mastered. Unfortunately, the identification of a TPE weakness is not synonymous with establishing an awareness of a typical Biola University candidate’s current understanding. Transforming the identified TPE weakness into useful information will require a content analysis. At present, Biola University’s CA TPA Coordinator has not conducted a content analysis because of complexity involved in retrieving an appropriate sample of candidate work. Developing the system to store and utilize the CA TPA tasks has been very difficult resulting in continuous revisions over the last few semesters. Learning from our struggles, a revised data-management system is being implemented during the fall of 2007. The essential elements in the new data management system include using a University provided computer to:
Create a computer file named after the semester
Further organize the semester file by creating a sub-file for each of the four TPA tasks
Place within each TPA task sub-folder a sub-sub-folder for the electronically submitted tasks (see Appendix D) and a sub-sub-folder for the accompanying Official Student Report (see Appendix K)
Save, in the appropriate submitted task sub-sub-folder, each candidate’s electronic submission using the candidate’s ID number as the document name
Save, in the Official Student Report sub-sub-folder, each candidate’s Official Score Report where once again the candidate’s ID number is used as the document name
Ensure that CA TPA data is not lost due to computer malfunction by burning a CD that will be removed and filed by semester with the applicable Record of Evidence forms in the CA TPA Coordinator’s office.
Realizing that a candidate’s responses to CA TPA task prompts is personal property; conducting a content analysis (identifying and recording typical strategies employed, range of places data is collected, preferred adaptation approaches, and examples of representative score-level responses) must not begin until appropriate permission has been granted. To secure candidate consent, Biola University’s CA TPA Coordinator has revised and obtained approval to use the Candidate Release Form developed by the University of San Francisco (see Appendix L). Currently, the forms are passed out during the LEDU 440-442/SEED 512-513 Elementary Student Teaching and LEDU 450-452/SEED 514-515 Secondary Student Teaching courses. The late request (candidates have already submitted two tasks prior to this course see Appendix H) prevents Biola University’s CA TPA Coordinator from conducting a content analysis until the candidate is ready to submit their third task. Even with an understanding of the limitation associated with the late request, the Teacher Preparation faculty believes that the selection of this late date is beneficial because it allows the candidate a deeper understanding of the personal nature of the task thus allowing the candidate to make a more informed decision. Additionally, the complexity involved in performing a content analysis has led to a decision to only complete this type of study once per year. To accomplish its research goal, Biola University will systematically re-label all saved files based on the candidate’s willingness to share work and to use only the work of the candidates who have provided permission.
Securing release forms is also critical to achieving Biola University’s goal of honoring the SB 2042 provision to increase the intentional communication between districts and Universities. It is the intention of the Teacher Preparation Program to use the content analysis studies to seek program improvement advice from district personnel through the development of relationships that allow annual meetings where Biola University statistics can be compared with current district practices. Finally, in an effort to ensure the accuracy of Biola University assessors, tasks with signed release forms will be provided to meet research and/or training requests associated with any CA TPA networking or state meeting agenda.
Appendix A: CA TPA Embedded Learning Experiences
Appendix B: FileMaker Pro Data Management Fields
Appendix C: Embedding CA TPA Data into Program Decisions
Prior to initial assessment, each candidate receives the Teaching Performance Expectations and clear, accurate information about the nature of the assessment and the pedagogical tasks.
The foundational belief to Biola University’s Teacher Preparation Program is to embed the CA TPA tasks in a fashion that supports the candidate while avoiding the perception that professors are simply teaching to the tasks. To accomplish our goal, candidates are introduced to the CA TPA tasks and the CA TPE standards during the very first course in the program (see Introduction to CA TPA in Appendix A). After the introduction, specific reference to the CA TPA tasks virtually disappear until the candidate enrolls in a CA TPA task-embedded course. Once enrolled, the candidate immediately gains access to the professor’s blackboard site where the Candidate Handbook, the task-specific template, and a score-level three benchmark sample are provided (see Appendix I).
Based on information gathered during the fall of 2004, the faculty of Biola University learned that simply providing the candidate with state-produced materials was not enough to generate passing scores. Reflecting upon the results of our first CA TPA implementation we discovered that a professor’s desire to produce excellent teachers often created a tension between the need to cover essential TPE content and the goal to provide candidate’s the extra support necessary to ensure CA TPA success. Resolving the identified stress began with the CA TPA Coordinator developing an intentional plan that focused on supporting the professors of CA TPA embedded courses. Work began by carefully examining the Candidate Handbook and the task templates to determine the least obtrusive ways to integrate CA TPA skills into each CA TPA embedded course.
As the study progressed, it became apparent that each CA TPA task could be broken down into a typical teaching-learning skill (e.g., establishing goals, learning about students, planning for instruction etc.). The transition of viewing the CA TPA tasks as steps and prompts to teaching-learning processes was the breakthrough that allowed professors to realize that supporting candidates involved little more than intentionally connecting content already embedded within their course with CA TPA requirements. During the next few semesters, candidate presentations (see Appendix M) were developed that allowed greater uniformity of candidate support across the various CA TPA embedded courses. Typical professorial assistance now entails:
Providing state-produced materials on their course blackboard site (see Appendix I).
In class, briefly reviewing the Candidate Handbook focusing on explaining to the candidate the importance of reading the material prior to completing the task and immediately upon finishing the task as a means of ensuring that concentration on the appropriate, unique characteristics of the CA TPA tasks is maintained.
In class, briefly reviewing the distinct characteristics of each rubric level.
Presenting the CA TPE course material as assigned in Biola University’s SB 2042 submission (see Appendix A).
Coordination of the material in the Student Support Help Presentation (see Appendix M) with a benchmark level three and/or four task.
Small group investigation of the benchmark level three task with a focus on revisions that would move the example task to a score level four
The natural manner in which the CA TPA tasks have been embedded within the Teacher Preparation courses can create a perception among the candidates and professors that the tasks mirror common course assignments. This perception assists Biola University’s Teacher Preparation program in meeting the formative nature of the CA TPA program. Yet, it is also important to remember the summative aspect of the CA TPA tasks. Three steps are now being implemented to ensure that each task submitted represents the candidate’s own performance. The steps are:
Professors are asked to check each candidate’s submission to verify that the content area and grade level of the task submitted matches the candidate’s fieldwork placement (see Appendix I).
Professors are asked to spend time reviewing with the candidates Biola University’s Code of Honor (see Appendix N). The goal of this document is to assist candidates in understanding what constitutes an acceptable level of collaboration. Phrases such as majority and less than fifty percent have been chosen specifically to alert candidates to the importance of matching pedagogical decisions to the unique students with whom they are working.
Candidates are asked to sign and submitted a Code of Honor (see Appendix N) document with each CA TPA task submission. Through the act of signing the document, candidates are affirming that the work they are submitting is original. Thus, any act of plagiarism now moves into an intentional act that can then be dealt with according to typical University protocol.
Finally, in an effort to monitor the honesty of our candidates, starting the in spring of 2008 Biola University’s CA TPA Coordinator will experiment with running internet searches on a random sample of candidate CA TPA task submissions. Initial work will simply involve lifting key phrases from a CA TPA document and running a Google search. As sophistication for completing internet searches develops, this aspect of our checks and balance system will be further revised.