Update:
This is a developing story and will be updated throughout the day.
How are computers scoring STAAR essays? Texas superintendents, lawmaker want answers
Educators and legislators are concerned about transparency and a spike in high schoolers scoring zero points on written answers.
Texas superintendents — and at least one lawmaker — want answers from the state education commissioner about how computers are scoring STAAR essays.
The Texas Education Agency quietly debuted a new system for examining student answers on the State of Texas Assessments of Academic Readiness, or STAAR, in December. Roughly three-quarters of written responses are scored by a computer rather than a person.
“This is surprising news to me as a member of the House Public Education Committee, as I do not recall ever receiving notice of this novel and experimental method for grading high-stakes, STAAR tests,” Rep. Gina Hinojosa, D-Austin, wrote in a recent letter to Commissioner Mike Morath, which was also shared with The Dallas Morning News.
Superintendents across the state also were caught off guard until recently. Many school districts already are suing the state over changes to the academic accountability system that’s largely based on STAAR performance.
The News reported on the rollout of computer scoring Wednesday.
The use of computers to score essays “was never communicated to school districts; yet this seems to be an unprecedented change that a ‘heads up’ would be reasonably warranted,” HD Chambers, director of the Texas School Alliance, wrote to Morath in a letter shared with The News.
TEA spokesman Jake Kobersky said in a statement that the agency is developing a comprehensive presentation for educators, explaining the changes in detail and addressing outstanding questions.
He added that the agency alerted the House Public Education Committee in August 2022 that it was pursuing automated scoring.
The final bulletpoint on an 18-page slideshow read: “TEA is pursuing automation for scoring where appropriate to reduce costs while ensuring reliability. Full human scoring is not possible under item-level computer-adaptive (B), and full human scoring with no automation under the current system would require at least $15-20M more per year.”
The new scoring method rolled out amid a broader STAAR redesign. The revamped test — which launched last year — has a cap on multiple choice questions and essays at every grade level. State officials say it would cost millions more to have only humans score the test.
The “automated scoring engines” are programmed to emulate how humans would assess an essay, and they don’t learn beyond a single question. The computer determines how to score written answers after analyzing thousands of students’ responses that were previously scored by people.
Among the district leaders’ biggest concerns is a huge spike in low scores among high schoolers under the new system.
Roughly eight in 10 written responses on the most recent English II End of Course exam received zero points this fall.
For the spring test — the first iteration of the redesigned test, but scored only by humans — roughly a quarter of responses scored zero points in the same subject.
Members of the Texas School Alliance, which represents 46 districts, “examined their individual district results and found shockingly consistent scoring differences.”
Chris Rozunick, the director of the state’s assessment development division, previously told The News that she understands why people connect the spike in zeroes to the rollout of automated scoring based on the timing. But she insists that the two are unrelated.
Many students who take STAAR in the fall are “re-testers” who did not meet grade level on a previous test attempt. Spring testers tend to perform better, according to agency officials who were asked to explain the spike in low scores in the fall.
“It really is the population of testers much more than anything else,” Rozunick said.
Kobersky added that, under the previous STAAR design, a score of zero was reserved for “unscorable responses,” meaning the question was left blank or written in a nonsensical way. The redesigned test rubric allows for a zero both if a response is unscorable or if it’s the value of the response as determined by the scorer, he said.
Some district leaders requested the state education agency provide them images of students’ responses so that they could “better understand what led to the significant increase in the number of zeroes, and most importantly how to help students write their responses” to receive better scores.
“Each request has been denied,” Chambers wrote in his letter to Morath.
Kobersky said fall questions are not released because they can be reused for other tests.
TEA officials say a technical report, with a detailed overview of the system, will be available later this year.
STAAR scores are of tremendous importance to district leaders, families and communities. Schools are graded on the state’s academic accountability system largely based on how students perform on these standardized tests.
“As with all aspects of the STAAR test and the A-F accountability system, it is important that there is transparency, accuracy and fairness in these high-stakes results,” Hinojosa wrote.
The DMN Education Lab deepens the coverage and conversation about urgent education issues critical to the future of North Texas.
The DMN Education Lab is a community-funded journalism initiative, with support from Bobby and Lottye Lyle, Communities Foundation of Texas, The Dallas Foundation, Dallas Regional Chamber, Deedie Rose, Garrett and Cecilia Boone, The Meadows Foundation, The Murrell Foundation, Solutions Journalism Network, Southern Methodist University, Sydney Smith Hicks and the University of Texas at Dallas. The Dallas Morning News retains full editorial control of the Education Lab’s journalism.
