The scientific method is one of the most incredible tools we have for understanding the world. It allows us to drastically minimize the risk of distorting reality based on our hopes and expectations. In this sense, science is theoretically very powerful.

But science, like so many other things, is a human creation, with its flaws, limitations, and pitfalls.

Thus, to avoid blindly regarding science and its practice as an absolute source of truth, we wish in this article to explain what it truly is, including in our time, and where it shows serious limitations.

Science has a history

The scientific discipline is an ancient concept. Humanity’s desire to understand the world is timeless, and the way methods have been refined over time depends on various factors: technical progress, political and religious agendas, cultures of knowledge, practical opportunities to pursue research, funding sources, academic cultures, and so on.

In this regard, the tools of science, used to apply what is known as the scientific method, are varied and evolve over time. There are also different methods, depending on the objectives and the subjects being addressed.

Different peoples will not have exactly the same priorities either. Some will focus on war, medicine, or art, depending on the historical context.

Scientific taxonomy (the way information is classified), meanwhile, varies by language and discipline. A genetic taxonomy will not be the same as an anatomical taxonomy, even though both are scientifically valid.

In this respect, science is closely intertwined with human history; it is a dynamic field that is constantly evolving and subject to specific constraints.

Science is arbitrary

This point is probably one of the least understood, even within scientific institutions.

We practice science based on a reality that gives us no guidance on how to do so. We observe and readily accept that our world is intelligible, but nothing in nature tells us how to go about understanding it. We are forced to develop our own tools to do so, and these tools always require us to make arbitrary decisions sooner or later.

Presenting study results with 99% confidence intervals is in itself an act of arbitrariness. Deciding that confidence is sufficient based on 99% validation of the underlying hypothesis is not justified by reality; it is a human interpretation that proves relatively satisfactory in our experience of understanding the world.

These arbitrary limits are not unjustified, but they are in fact impossible to prove definitively. They also evolve over time, and two different eras will not have exactly the same tools or the same state of the art in science.

This arbitrary dimension of science is essential to understand in order to be able to assess the value of the studies produced and avoid falling into the easy trap of oracular science. It also keeps science alive by leaving the door open to improving the very tools we use.

Science is fallible

It can be hard for many science enthusiasts to admit, but scientists make mistakes (sometimes serious ones) and this remains true today.

The conditions under which science is practiced are varied:

  • technical constraints

  • budgetary constraints (public or private funding)

  • political constraints (instability, censorship, ideologies, or other factors)

  • psycho-egoistic constraints (pride, bias, intellectual limitations, or other)

  • cultural constraints (visibility of research, social pressures, or other)

  • systemic constraints (publication frequency and positivity, loyalty to current and former peers, or other)

Each of these constraints can either encourage or hinder the proper practice of science. A generous budget often allows for the production of better studies than a tight budget. A culture of research freedom opens up more possibilities than a culture of censorship and intellectual conformity, etc.

Probably due to the negative impact of these constraints, in recent years, researchers have identified a significant phenomenon that is gaining momentum and demonstrates a significant fallibility in current science.

The Reproducibility Crisis

The most well-known indicator of the limitations of science as a practice is what is known as the “reproducibility crisis.” This is a well-documented and quantified observation (see the Wikipedia article on the subject, which is quite good) that tells us that a very large number of published scientific studies are not reproducible, either because the information needed to do so is missing or because the results are not corroborated.

These difficulties in reproducibility vary by field, but it is worth noting that psychology and oncology are particularly affected, even though these are two disciplines with enormous implications for our lives, particularly due to the health policies they inform.

As an example, we can cite this Reproducibility Project study, led by the Center for Open Science in Virginia, which aimed to assess the degree of reproducibility of psychology studies dating back to 2008. The authors of this study found that the choice of statistical analysis methods had a significant impact on the results, and they were ultimately able to replicate only 36% of the published studies, with effects that were, on average, half as strong.

In 2012, C. Glenn Begley and Lee Ellis published a paper reporting a reproducibility rate of 11% for oncology studies (out of 53 studies analyzed).

Journalistic neutrality aside, this rate is scandalously low, and the downplaying of the reproducibility problem by most scientists and the journals that publish them is a disgrace.

This misguided approach can be attributed to the constraints mentioned earlier — which are also noted in the Wikipedia article — but we would now like to address one in particular that we find especially troubling.

Systemic incompetence

The issue of incompetence is a sensitive topic in general, both because it touches on researchers’ self-esteem and credibility and because such criticisms are often met with arguments along the lines of “you’re not a scientist, so you can’t be any more right than they are.” (that we sometimes call the resumé argument)

That is why it is preferable to approach the issue from the perspective of the research itself, particularly studies that address the question directly.

Here are three examples of studies whose objective was to assess scientists’—and, among them, statisticians’—mastery of statistics:

A rough estimate based on these studies suggests that approximately 90% to 99% of the scientists and researchers surveyed—regardless of their level of education (whether they are still students, doctoral candidates, or full-fledged researchers) and regardless of whether they are statisticians or not—do not fully understand what a p-value and a confidence interval are (two of the most important basic statistical tools in modern science within the frequentist paradigm).

These studies strongly suggest that systemic incompetence is at work in scientific circles and that teachers themselves are passing on their own misconceptions of science to their students. The fact that we observe the same or nearly the same results in the Netherlands, China, and Sweden also tends to show that the phenomenon is global and not specific to certain countries.

Another study, conducted in 2021 by Mauro Federico Andreu and his colleagues, showed that in Argentina, approximately 63% of physicians and respiratory therapists did not fully understand what the p-value means.

Further studies would obviously be welcome to refine this finding, but it actually aligns with a body of evidence suggesting that the state of the art in scientific research is questionable, if not catastrophic in some cases.

This problem of understanding basic statistical tools is all the more concerning because these tools directly impact the understanding of the value of studies as well as their significance within the research corpus. They also influence future research and may unduly support political and medical decisions.

Argumentative caution

This brief overview of the nature of science and some of its limitations is intended to encourage us all to exercise great caution in how we approach scientific studies.

Indeed, it is not enough to say that “such-and-such a study has shown that” for our argument to be truly sound. In fact, we must very carefully read and verify the studies we cite in order to exercise the most meticulous scrutiny (which, in theory, should be carried out by the institutions themselves and international journals).

This can only benefit our quest for explanatory accuracy and may also help us develop more cautious, precise, and, possibly, less peremptory argumentative approaches. It also helps demystify the scientific world and its claims, which should be the hallmark of a scientific approach worthy of the name (sticking only to solidly supported facts).

The use of meta-analyses is also valuable for highlighting trends at the macro level, including those concerning methodological weaknesses.