Blind audition study: Truth or myth? | FACTUAL FEMINIST

The blind audition study is one of the
most celebrated social science papers of all time. It’s famous for showing that
when orchestras auditioned musicians blindly behind a screen, women’s success
rates soared. But it turns out, the study showed no such thing. What’s going on? That’s coming up
next on the Factual Feminst. During the 1970s and ’80s, the nation’s orchestras became more open and
democratic, and to ensure impartiality, several introduced blind auditions. Two
economists, Claudia Goldin of Harvard and Cecilia Rouse of Princeton, noticed that
women’s success rates increased along with the adoption of screens. Was that a
coincidence, or the result of the screens? That’s the question they tried to answer
in their 2000 paper “Orchestrating Impartiality: The Impact of Blind
Auditions on Female Musicians.” They collected four decades of audition
records and rosters from eight major orchestras and crunched the numbers. Now,
their paper includes multiple warnings about small sample sizes and
contradictory results — but few readers seem to have noticed. What caught
everyone’s attention were some strong claims in the final paragraph: “We find
the screen increases by 50% the probability that a woman will be
advanced from certain preliminary rounds.” According to Google, the study has
received 1,500 citations in academic articles and thousands of media mentions,
and showcased in so many diversity workshops that one recent attendee
begged never to hear about it again. Now, the study’s appeal is clear: two
prominent economists, in a top journal, wielding state-of-the-art econometrics,
captured and quantified gender bias — and they also documented a solution. Or so it
seemed. The research went uncriticized for nearly two decades. That changed recently
when a few scholars and data scientists went back and read the whole study. They found
a tangle of small, ambiguous, contradictory findings. For example, the
screen seemed to help women in preliminary audition rounds, but in the
semi-final rounds, they didn’t. And none of the findings were strong
enough to draw a broad conclusions one way or another. And the authors say as
much, albeit ambiguously throughout the paper. So where did Goldin and Rouse get their totemic conclusion that blind auditions
dramatically improved the success of women candidates?
Well, after warning that their findings were not statistically significant,
they simply declared them to be economically significant. But what does
that mean in this context? “That doesn’t mean anything at all,” says Columbia
University data scientist Andrew Gelman. In a recent commentary on the study, he
said that they’re just “fine words” that really mean that our data is “too noisy
to form any strong conclusions.” Now my guess is that the authors thought they
detected something with real-world relevance somewhere in all that
noise — but that’s a reason to call for more research, not to declare the
transformative power of screens in women’s quest for equality. Still some
may think it seems obvious that the screens contributed to equal hiring, but
it’s not. The screens may have been a reflection of changing attitudes — and it
was those attitudes, not the screens, that helped women. After all, women didn’t need
blind auditions to move ahead in law or business, medicine, or even in the
Cleveland Orchestra, which had not resorted to blind auditions, according to
the study. Now, Gelman and other critics — they don’t
deny the reality of gender bias and they don’t question the potential merits of
blind auditions as a means of achieving impartiality. But
Goldin and Rouse verified nothing about the special benefits for women, and nor
has anyone else. The subsequent research on blind
recruitment is just a morass of baseless claims or retracted statements,
contradictory findings… So how did such an equivocal study achieve iconic status?
Well, a lot of the credit goes to confused but influential fans, such as
the writer Malcom Gladwell and the Harvard psychologist Mahzarin Banaji.
They saw the study as powerful, indisputable proof of the ubiquity of
gender bias, as well as a way to counter it. According to Gladwell, “Orchestras in
the 1980s started putting up screens in audition rooms and immediately —
immediately — orchestras started hiring women left and
right.” And here’s professor Banaji in a 2017 TEDx talk: “In the late 1970s,
American orchestras were almost entirely all male . . . Once the curtain dropped, the
case study shows that the number of women who were selected doubled — they
went up 50%.” But it didn’t show that. Truth matters. Overhyped claims create
confusion and undermine public trust, and they don’t solve problems. Sex
discrimination in the workplace is a serious matter,
but improvements require solid data, replicable research, and careful
evaluations of causation. As the scholar Alice Dreger says, “Carpe datum . . . Evidence
is an ethical issue.” If you found value in this video, please show your support by subscribing to the series, and follow me on Twitter, and listen to my podcast,
“The Femsplainers.” Thank you for watching the Factual Feminist. Wave goodbye, Izzie.

Leave a Reply

Your email address will not be published. Required fields are marked *