Study: Subconscious human bias exists in tournament selection
Economists found membership in BCS conference is worth an extra 1.75 seeds
Authors aren't accusing committee of fraud; they're saying members are human
NCAA says sample size is too small, but data backs up economists' claim
In the minutes after the NCAA selection committee announced this year's tournament bracket on Sunday night, the most curious decisions became instant debate fodder. On CBS, SI's Seth Davis wondered aloud about the respect given to the Pac-10 -- a relatively weak conference this season -- in the form of a No. 8 seed for regular-season champ Cal. On ESPN, Dick Vitale railed against the committee for awarding Wake Forest an at-large bid and leaving out Virginia Tech, which had a better overall record, a better ACC record and a head-to-head win against the Demon Deacons.
Had either read a soon-to-be published study by three economists, they might not have been so surprised.
The study, by Jay Coleman, Mike DuMond and Allen Lynch, looked at selection data from 10 tournaments (1999-2008) and found that when seeding the tournament, membership in one of the six BCS conferences is worth an average of an extra 1.75 seeds. The study also found that having a conference representative on the 10-member selection committee resulted not only in a higher seed but also in a better chance of getting an at-large bid. According to the authors, a true bubble team (one with a 50-50 chance of getting in or being left out) would have a 49 percent better chance of getting in if its athletic director is on the committee, a 41 percent better chance if its conference commissioner was on the committee and a 23 percent better chance if a fellow conference AD is a member of the committee.
According to the researchers, Wake Forest would have beaten out Virginia Tech this year even after removing the controls for selection committee bias. Hokies fans should be angrier about their team's abysmal out-of-conference schedule, but it probably didn't escape their notice that Wake Forest athletic director Ron Wellman is a member of the selection committee.
The study's authors aren't accusing Wellman or any other selection committee member of deliberately rigging the process. In fact, they realize Wellman wouldn't have even been allowed in the room when the committee voted to grant Wake Forest an at-large bid, nor would he have been allowed to offer an opinion on fellow ACC member Virginia Tech. What the authors are suggesting is that the selection process setup allows subconscious biases to creep into the proceedings.
"We're accusing the committee of being human," said Coleman, a professor at the University of North Florida. "It's human nature. We all are biased at some level."
While the authors' findings probably wouldn't surprise anyone who has ever sat on a barstool and debated the selection committee's seeding and at-large choices, this is the first time anyone has performed a statistical analysis to determine whether the conventional wisdom is true. The appearance of bias, meanwhile, is a sensitive subject to the NCAA, which distributes millions of dollars to conferences and schools based on how many games each school plays in the tournament.
Greg Shaheen, the NCAA's senior vice president of basketball and business strategies, said the committee goes out of its way to eliminate bias, and he pointed out that -- at least with the finding of bias in at-large selections -- the selections in question accounted for only about three percent of the selections studied, making the sample size relatively small. Shaheen also said the authors did not account for the rules that govern seeding and bracketing the tournament.
After reviewing the study, Tom Paskus, the NCAA's head research scientist, expressed several concerns with the study's methodology. In one case, he called a particular set of data "nonsensical." The authors counter that the study was evaluated independently before it was approved for publication in Managerial and Decision Economics. Emory University professor Paul Rubin, the journal's editor, confirmed in a phone interview that the study was peer reviewed by economist who specializes in sports. In the world of academic journals, such a reviewer is called, coincidentally enough, a referee. Rubin declined to name the referee in order to protect the double-blind review process, but he said the referee approved the paper for publication after one round of revisions.
SI.com also sent the study to be reviewed by University of Michigan sports management professor Rod Fort, one of the nation's leading researchers on the business of college sports. Fort echoed Shaheen's sample-size concern, but he said he found no major methodological flaws.
The project began 10 years ago as particularly nerdy version of a traditional March pastime: forecasting the bracket. Coleman stumbled upon Jerry Palm's collegerpi.com, a trove of roundball data that included almost every statistic an economist could want to build a model. "For folks like the three of us," Coleman said, "something like that is a gold mine." So Coleman brought in Lynch, who is now a professor at Mercer University in Macon, Ga. (Lynch and DuMond also are co-designers of the college football recruit prediction model that helped inspire SI.com's 2009State of Recruiting project.) The pair used the usual factors such as record, conference record, RPI, quality wins and other on-court statistics to build the model.
After several years of predicting the tournament field using their Dance Card formula, Coleman and Lynch noticed they usually missed two or three at-large teams a year. They also noticed the teams they missed on seemed to have similar characteristics. They realized they needed to change the formula after the 2006 tournament.
For most, the defining memory of the 2006 tournament is watching George Mason, an at-large team from the Colonial Athletic Association, crash the Final Four. For Coleman, DuMond and Lynch, it was that selection Sunday. Air Force hadn't even bothered to gather as a team to watch the selection show. The Falcons got an at-large bid. Meanwhile, George Mason's situation looked dire thanks to a pair of losses to Hofstra -- a team with a similar resume and an RPI of 30. George Mason got an at-large bid. Hofstra didn't. "I need someone to explain this process to me," Hofstra coach Tom Pecora told The Buffalo News.
Little did Pecora know Coleman and Lynch were about to recruit the man who might provide an explanation. DuMond, steamed that his alma mater, Florida State, had seen its bubble burst that year, happily agreed to join the project. The information Coleman and Lynch sought was in DuMond's professional wheelhouse. A principal at Charles River Associates in Tallahassee, Fla., DuMond specializes in discrimination issues involving labor and employment, so he knows how to mine data for bias. In looking at the at-large bids that year, DuMond noticed a common complaint that can best be described with a passage from a story John Markon wrote for the March 14, 2006 edition of the Richmond (Va.) Times-Dispatch.
This year, everyone's complaining about the same three or four choices - Air Force, Utah State, California and George Mason. We've heard all the arguments based on RPI, quality losses, record in the most recent 10 games, etc.
Let's make it slightly simpler:
Air Force: A Mountain West Conference AD, Chris Hill of Utah, was on the committee.
California: A Pacific 10 Conference AD, Dan Guerrero of UCLA, was on the committee.
Utah State: Western Athletic Conference Commissioner Karl Benson was on the committee.
George Mason: Mason AD Tom O'Connor was on the committee.
More College Basketball
College Basketball Truth & Rumors