How to identify testing bias

When Sheila Akbar, director of education at test prep company Signet Education, took a practice exam for new SAT, one passage threw her off. In the reading section, students were asked to analyze a letter exchange between two women in the 1830s arguing about whether they should become involved in the abolitionist movement. (The SAT often incorporates essays, letters and other historical documents from civil rights struggles). One woman was for, the other, who was against, argued that a woman’s place is in the home.

Akbar, a seasoned test taker, was rattled.

“It literally said women are inferior to men. I found myself getting philosophically engaged, and getting angry,” she said. “And then I thought, ‘Oh wait, I’m on the clock, I need to get to my questions and stay focused.’ And then I started thinking, ‘I’m in a room of 16, 17-year-old kids. What if any of them are upset by this? What if they don’t have the ability to say this is an intellectual exercise, and say I need to push my feelings aside and just focus on the questions?’”

Though the letters were meant to be interpreted within a historical context, Akbar says that any type of bias can distract from the task at hand — acing the exam.

Though the SAT was recently revamped, no test is perfect, says Akbar. She identified different types of testing bias — in all types of tests— that students should be aware of.

Language bias
One of the reasons many tests, including the SAT, have removed analogies and other vocabulary-heavy questions is because certain words used — such as “yacht” or “polo” — were familiar to some (wealthier) students, while others did not have these words in their vocabularies, explains the expert. Though class and racial bias are most talked about, testing bias takes many forms.

Stereotype threat
A question in the math section about gender differences in enrollment in high school math classes also set off a red flag for Akbar. The question indicated that in all classes except for one, more boys were enrolled than girls, which, says Akbar, might pull a female student’s focus off the question and onto the stereotypes about girls and math.

“We know that when minorities groups are faced with stereotypes about their own demographic, they become extra anxious about not confirming that stereotype, and that actually leads them to perform worse, since they’re worried about something other than what the test items actually are.”

Administration bias
It’s not just tests themselves that can be biased, but the way they’re administered. If a test proctor isn’t following protocol, or test-takers haven’t been given the the correct amount of time, results can be skewed. Though test-takers have the right to report any kind of break in protocol, they often don’t. Says Akbar, “It’s a little unrealistic to expect a 17-year-old to make a complaint to a company that controls their scores.”

Navigating bias can be tricky, but at the very least test-takers can make themselves aware of the pitfalls.

“Always do official practice tests in a timed, realistic setting,” says Akbar. “That way, if there are elements of bias, you’ll be aware of it before you go into the full test.”