What happens when the public lose confidence in academic findings?
In an important new paper, Rafael Ahlskog and Sven Oskarsson show that half of the effect size in observational studies, even in conservative estimates, is composed of confounding. In laypeople’s terms, this means that many academic studies considerably overestimate the effect of the variables which they study. Such results erode confidence in academic work, yet there have been a series of these findings. In a 2018 paper, 29 expert teams used a single dataset to analyze whether football referees were more likely to red card dark-skin-toned players, the teams coming to radically different results, none of which could be proved incorrect. Such findings follow the replication crisis. In 2015, researchers replicated 100 leading studies in the field of psychology, finding that only 36% of the replications yielded significant findings. Moreover, the mean effect size in the replications was about half the effect reported in the original studies. In other fields, similar crises followed.
Within academia, such developments are well known. Yet public awareness is limited and few reflect on practical consequences for politics. Though such challenges do not affect academic confidence in the reality of (say) climate change and Coronavirus, there are significant consequences for policies which address such issues; if scholars worry about findings, how can we be confident that policy interventions are effective? Admittedly, political implications are indirect. Decades ago, postmodern philosophers deconstructed social reality, engendering loss of confidence in liberal democracy among certain intellectuals. Yet this had questionable impact on practical politics. At this time, politicians such as Ronald Reagan and George W. Bush promoted liberal democracy with an evangelical zeal, reflecting a popular culture in which confidence in democracy remained overwhelming. But recent trends are distinct. Quantitative social scientists now worry about the validity of methods, previous crises of confidence being confined to the humanities.
Such developments may pose greatest risk to progressive ideologies, these being predicated on the notion that politicians can improve society. For example, the Blair governments championed evidence-based policymaking. One response to these developments is to embrace ambiguity, some postmodernists doing this decades ago. In recent years, social justice ideology has popularized many postmodernist ideas. But ironically, much of social justice ideology involves increased certainty, adherents decrying power structures and emphasizing the importance of race and gender. The cancellations associated with social justice ideology are unequivocal acts of certainty. This paradox reflects the fact that social justice ideology has mass appeal. Certain intellectuals might be able to embrace radical ambiguity, yet this is too much for most people! Those faced with ambiguity often embrace certainty, this being one interpretation of the appeal of social justice ideology.
A second response is to become more wary of unsettling existing political institutions and social traditions. In recent years, other developments have supported such a conservative approach. Many scholars favour cultural evolution theory; this posits that social institutions tend to be efficient, reflecting developmental processes akin to biological evolution. The theory evokes Burkean arguments against political change, these emphasizing that traditional institutions are depositories of wisdom. These arguments are not merely reactionary, cultural evolution theory demonstrating that historic survival makes institutions efficient. Given this efficiency and weakening evidential bases for policy interventions, arguments for the status quo strengthen.
Yet our world faces existential challenges, such as climate change, associated with existing institutions and undermining the conservative position. Following these problems, organizations such as Extinction Rebellion conclude that we need rapid, systemic change. Such solutions are appealing, but scarcely resolve the problem of uncertainty. Left-wingers propose a gamble, merely hoping that the new system will be better than the old one. Given the epistemic uncertainty of most left-wing thought, notwithstanding sporadic rigidities, I do not see how activists can deny this. As a Burkean social democrat, I am more cautious, preferring to seek solutions using existing institutional frameworks. Obviously, this choice is scarcely objective, being bound up with factors such as personality. Ultimately, there is no ‘correct’ approach.
One can exaggerate the significance of such developments. They scarcely sound the death knell for academia. Many findings are robust; for example, Ahlskog and Oskarsson conclude that half of effect sizes remain valid. Notwithstanding these caveats, such developments increase pressures for polarization, further straining liberal democracy after a tumultuous decade. We may wonder how much stress the old system can take.
If you enjoyed reading this, do think about subscribing! Subscription is free – all it means is that you’ll receive a weekly email. But every new subscriber makes me very happy 😊 😊 😊