“Gaming the College Rankings” – HigherEdJobs

Once again, the reliability of college rankings has been thrust into the spotlight, thanks to recent charges that universities such as Columbia, USC, and Temple have misreported data to U.S. News & World Report. Ever since U.S. News began ranking colleges in 1983, educators have devoted endless hours to employing gaming techniques ranging from distortion of academic policies to outright falsification of data. Why do schools cheat, and what can be done about it?

The primary targets of gaming behavior are the “best-college” rankings, such as those published annually by U.S. News, the Wall Street Journal, Forbes, Money, Washington Monthly, and Niche.com. These publications purport to construct a single, comprehensive measure of institutional quality that produces an ordinal listing of schools from “best” to (presumably) worst. They all utilize formulas based on a selection of variables, weighting factors, and statistical techniques dreamed up by the rankers, with little or no scientific justification or even scholarly input. And they all rely heavily on unaudited data self-reported by the ranked institutions themselves.

Since most ranking publications publish a general description of their methodology, institutions can identify ways in which to improve their score by manipulating the statistics they report. The most egregious form of gaming is the intentional submission of false information. Over the years, dozens of institutions have been found to have reported falsified data on such variables as their entering students’ average SAT scores, acceptance rates, yield rates, student/faculty ratios, faculty salaries, graduation rates, or alumni giving percentages. Though the number of documented cases is small relative to the total amount of reporting, most knowledgeable observers believe that the public revelations represent the tip of a very large iceberg.

A somewhat less corrupt, but surely more widespread, form of data manipulation is sometimes referred to as “massaging” the numbers. Examples include counting non-teaching faculty in student/faculty ratios, omitting whole categories of low-scoring admittees (such as recruited athletes or foreign students) from average SAT scores, classifying medical-care expenditures in a university hospital as “instructional,” or statistically removing chronic nondonors from the alumni giving database.

At the other end of the gaming spectrum are all the ways in which institutions alter their practices and policies simply to improve their scores in the rankers’ formulas. While this type of gaming is less morally objectionable than data manipulation, it often comes at a cost to the quality of the academic program. For example, schools seeking to report a high percentage of small classes (customarily calculated in the fall semester) sometimes shift large lecture classes to the spring semester. By doing so, they disadvantage students who need to take introductory courses early in their college careers. Similarly, the mad rush to improve graduation rates — heavily weighted in most ranking formulas — has led to charges of grade inflation and elimination of science and math requirements. The scramble to improve rankings can also undermine institutions’ professed goals of increasing racial, ethnic, and socioeconomic diversity. Examples include changing admissions criteria to give greater weight to SAT scores, expanding early-decision admissions programs, and shifting the financial aid budget away from need-based aid to allow for more “merit” grants.

Those who misreport rankings data rarely suffer a severe penalty for doing so. Even when schools have been caught red-handed, U.S. News has merely dropped them from the rankings for one year or required their presidents to sign a form attesting to the accuracy of subsequent submissions. Occasionally, outside entities impose more severe sanctions. For example, the law-school accrediting agency required one data falsifier to report its transgressions on the school’s website for three years. Most ominously for rankings cheaters, a federal court recently convicted the Temple Business School dean of wire fraud for repeatedly lying to U.S. News. But the facts of that case were sufficiently unusual that the odds of other cheaters facing prison time are quite low.

The problem of misreporting might be solved if colleges were required to obtain independent audits of the data they submit to the rankers. After all, we require schools to obtain financial audits before allowing them to sell bonds to investors. Why not require them to conduct “academic audits” before asking applicants’ families to pay hefty tuitions to schools chosen on the basis of their rankings?

The second form of gaming — chasing higher numbers by adopting tactics that harm the academic program or undermine institutional values — is also hard to prevent. So long as rankings influence applicant decision-making, schools will have a strong incentive to abandon their principles and march to the beat of the rankers’ drum. Perhaps college accreditors could require schools to report any policy changes they have made in order to improve their rankings and to explain how those changes affect the quality of their academic program and the attainment of diversity-related goals.

Instead of compromising their principles, colleges should ideally follow the lead of schools like Reed College, where I served as president, and renounce the rankings altogether. Unfortunately, at the present time, only about 15 percent of schools ranked by U.S. News follow that advice. And most of them derive little benefit from being included due to their low rankings. Recognizing the intellectual emptiness and the distorting — if not to say corrupting — effects of rankings, more of the selective schools should have the courage to walk away from them. I was pleased to see that Columbia University has just announced a decision to cease cooperating with U.S. News for at least the next year, while conducting an audit of its recently criticized reporting procedures. If Columbia would seize this opportunity to drop out permanently, other schools might be inspired to follow suit, thus depriving U.S. News of its undeserved power over higher education.

Keep your fingers crossed.