Publications

CRESST Recommendation for New Assessment System for UC Admissions

May 20, 2020

Background

College and university admissions processes are—much like our planet is with coronavirus—in a state of uncertainty and transition. What we thought may have worked well enough to get by is no longer viable. As with many aspects of our lives, the University of California (UC) admissions process—specifically the standardized testing component—requires reform. Adjustments to existing assessments are simply not enough. With all of this uncertainty, it is easy to lose sight of the fact that we needed reliable and valid information about college applicants before the pandemic and will still need it when it is over.

With careful review, the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) at UCLA recommends a bold research-based vision for an assessment solution that is equitable, fair, and predictive of success in the UC system and beyond. CRESST is not recommending a specific admissions process nor use of an existing or retrofitted large-scale assessment. Rather, CRESST—based on its over 50 years of assessment and evaluation expertise— is recommending the development of a completely new assessment system for use in the UC admissions process.

Li Cai, Director of CRESST at UCLA, and Eva Baker, Founding Director of CRESST, are members of the University of California Standardized Testing Task Force. Tasked by University President Janet Napolitano and created by Academic Senate Chair Robert May, and comprising 17 faculty members across all 10 campuses as well as a student representative, the Task Force spent over a year exhaustively researching the impact tests like the SAT and ACT have on student success and diversity in the context of a broad review “to consider whether the University and its students are best served by UC’s current testing practices, a modification of current practices, another testing approach, or the elimination of testing.” The resulting Task Force report provides evidence and recommendations for the University of California Board of Regents as they decide on standardized testing in the UC admissions process.

Large-scale assessments have been publicly scrutinized for decades. Educators teach to the test; too much classroom time is spent on test preparation; only certain students can afford to take certain tests; and inequities associated with the availability of paid test preparation may make success more likely for certain subgroups of students than others. Much of the criticism is fair—tests can be unfair if not designed with equity in mind; test scores may have questionable validity and not assess the knowledge and skills they were intended to assess; and they can be expensive for the test-taker, school, and state. When the Task Force began its work, it was a popular notion that current standardized tests were the main culprit in making UC less diverse.

The Task Force in its review found—to some surprise—that tests such as the SAT provide a useful tool to support admissions leading to a more diverse pool of admitted students than UC otherwise would have had. The Task Force found that approximately 25% of low-income, first-generation, and underrepresented minor students earned their guaranteed admission into UC because of test scores. The Task Force’s report showed that test scores are better predictors of success for underrepresented minority students, first-generation students, and those whose families are low- income. Using recent UC-specific data, the Task Force found that test scores remain predictive of success even after student demographics are taken into account. That is especially true when compared to high school grades, whose predictive power has gone down due, in part, to grade inflation. These findings tell us that there is value in the evidence that educational assessments provide, but they can be improved—and so can the UC admissions process as a result. In a vote of confidence in its veracity, the Task Force report was unanimously endorsed by faculty members of the UC’s Academic Senate, 51–0.

This document serves as an acknowledgment by CRESST of the Task Force’s comprehensive and exhaustive review of both the assessments currently used by UC in its admissions process, and going beyond that acknowledgment, a proposal of a bold, new, better assessment system—one that addresses from its inception the design, equity, quality, and cost concerns that in some cases have been lost in the use of assessments for UC admissions and for accountability purposes.

The New Assessment System

CRESST wholeheartedly endorses the bold new vision proposed by the Task Force, that it is possible “to create fundamentally different kinds of assessments that satisfy all University of California Board of Admissions and Relations with Schools (BOARS) principles on testing, while retaining predictive power, providing improved feedback to students and teachers, and broadening the demographic range of students UC is able to admit.” It is no exaggeration that current standardized tests, whether they were originally developed for university admissions purposes (e.g., the SAT and ACT) or for K–12 school accountability purposes (SBAC), tend to measure a far narrower piece of achievement and potential than covered by K–12 education or expected from higher education institutions.

Design

This new UC assessment will be an admissions test as opposed to a K–12 school accountability test as mandated by federal education legislation. It will be a high-stakes assessment—it, along with other factors, informs the admission of a specific applicant to the highly selective UC system, and to a degree, the applicant’s success at a UC campus and beyond. While existing K–12 accountability tests are helpful in, for example, informing placement decisions (e.g., the use of SBAC at California State University campuses) and capturing the differences between larger, aggregated groups such as schools or districts for school accountability, their test design does not support the differentiation that is needed to select among individual applicants with the proficiency and preparedness to succeed in the UC system. Feedback from the assessment is focused far more on informing teachers and schools than on informing students about their performance. This is evident in the quality and depth of overall scores and subscores.

Fairness and Equity

Achievement gaps are unfortunately an enduring issue across all large-scale assessments, including accountability tests such as SBAC that is supposed to be closely aligned to content instruction based on the Common Core—there is strong evidence that the gaps in average achievement across subgroups of students along economic status or race/ethnicity lines are not getting any smaller in California over the last five years of Common Core implementation.

All tests, independent of their claimed design attributes, almost invariably reflect disparities related to background and prior experience unless there have been effective interventions in between to mitigate their influence, especially for the students who were systematically disadvantaged. The current assessments arrive very late in an applicant’s high school career, leaving no room for feedback and corrective action even if desired. With structural inequalities at their present level in the state and across the nation, it is little wonder that there is a lack of progress in closing the gap.

While the Task Force found that standardized tests have been a useful tool in admitting a more diverse pool of students than the UC would have gotten, and to a substantial degree admissions practices at UC already compensate for differences among student demographic groups—we know we can do better. There is already research and evidence that an assessment system that takes full advantage of new interactive technologies, including computer-based simulations and performance tasks; that elicits real student performance; and that provides immediate feedback to the students, can reveal and foster reasoning, potential, problem-solving, and critical-thinking skills, just to name a few—all factors important to college and career success—more than existing standardized tests.

The new assessment system should not be a “done once” test in 11th grade. As per the Task Force’s Recommendation 6, “[the new test] will be continuously accessible to students” and “will assess a broader array of student learning and capabilities than do current tests. This new assessment should provide more feedback to schools and to students and enable UC to admit classes of students more representative of the diversity of the state.” The continuous availability will shift the focus from taking a static snapshot of the student, to the longitudinal measurement of potential, change, and growth. It also allows multiple and more reliable data points to capture the developmental trend of a broader range of skills and attributes related to an applicant’s success at UC, reflecting the principle that the University “should select those students who demonstrate a strong likelihood that they will persist to graduation.”

Technology and Delivery

The design of tasks and items in current assessments, while adhering to industry standards, does not take full advantage of the promise of technology and the new ways in which tasks may be scored.

Using SBAC as an example, no matter how many scorable “items” are grouped under one performance task, they are based on a single task assigned to a student in a year. The usage of single performance tasks creates concerns of fairness and comparability. The design of the new assessments should include many more scored data points on performance in simulations, real- world-relevant tasks, and perhaps even game-like settings. These performance tasks may also be used to measure “non-cognitive” attributes such as interest, persistence, and curiosity, should there be interest among the stakeholders in this new assessment system.

This new assessment should be delivered digitally. The types of items in existing large-scale assessments do not fully measure the range of possible performance that students, particularly subgroups of students whose first language is not English, can give to indicate their proficiency or achievement. The design of a new assessment provides the opportunity to move beyond the constraints of the performance task structure of existing accountability tests. Delivery of current assessments requires in-person participation at testing sites. The in-person assumption is seriously challenged for the first time in decades by the COVID-19 pandemic, the closing of schools, and limitations of large-group gatherings such as those required by large-scale test administration due to public health measures implemented to slow the spread of the disease.

Cost

Large-scale assessments are expensive—to develop, to take, to maintain, and to score—for the test- takers, their families, and for local and state education agencies. Facing deep state and UC budget cuts, one may wonder whether the task is financially viable. We believe that the UC, because of its size and reputation, is strategically well poised to lead a national coalition around this important assessment reform, precisely at this moment in history. Novel methods for automating the creation of items and performance tasks, field testing design, and automated scoring with AI also indicate potential for substantial reduction in the ongoing cost of maintenance.

It is our belief that the new UC assessment system, along with learning and assessment preparation, should be freely available to all students in California. Already, a large percentage of students in California benefit from free in-school SAT testing and fee-reduction vouchers. The current model of test-takers paying for admission testing services is outdated and needs to change. The in-school SAT testing that is free to students is a good step in shifting the burden away from the students and their families. Fee-reduction vouchers shift the burden to the assessment developers themselves.

Because higher education institutions are the ultimate ongoing users of the assessment results, a clear realignment of interest with responsibility is needed here. The users of an assessment product can, should, and will demand the development of an assessment that is predictive, informative, and fair. If the broad assessment system proves to be valid, reliable, fair, and predictive in other settings, for example, career-related, other users may soon follow.

Concluding Remarks

As researchers, UC faculty, and Task Force members, it is our responsibility to both review existing data and research from existing assessments, and to visualize how the UC’s admissions process will be improved with a new assessment system. We, as experts in assessment, recognize and recommend moving beyond the retrofit and recycling of existing assessments.

California has been a bold leader in evidence-based decision making at a time when this key pillar of our society is under assault. As the University of California proudly says, we are “the only world- class public research university for, by, and of California.” This is an opportunity for the UC to live up to that standard, to lead and to shape the UC admissions process to reflect the diversity and opportunity of California’s students. Join us.

0 likes
CRESST

Author

CRESST

Your email address will not be published.