December 17, 2024

If you are not an expert in a field but want to draw from the scientific knowledge base, how do you know what constitutes high-quality research? If you ask scientists, they will very likely tell you to read articles in journals that are well-respected in their scientific community. And this is not bad advice. The current academic system relies heavily on publishers for quality assurance. This system is built on the premise that editors and peer review effectively filter out poor-quality work, ensuring that only trustworthy and interesting studies make it into “respected” journals. In this blog post, I explain why this approach may be short-sighted and propose a long-term vision for what alternatives could look like. 

Quality Assurance is More Important Than Ever 

Let’s first consider how research in academia is conducted. Compared to industry, which is designed to optimize productivity, academia has the dual purpose of conducting (fundamental) research and training new scientists. Much of the data collection and analysis is conducted by PhD students and "early-career" researchers, who often work fairly independently and are responsible for all aspects of their projects. While this is a valuable learning experience, it also means that individual researchers—no matter how qualified, experienced, or smart—may not be able to manage every aspect perfectly. How could they? They are just humans, often working under immense stress. This stress is largely related to the precarious working conditions of academics (see #IamHannah for the prime problematic case Germany). The race to publish and the competitive nature of academia create additional pressures, often incentivizing researchers to cut corners in quality assurance (also see my last blog post). This raises a critical question: How is it checked that all the work is done correctly? 

Internal Review 

In research meetings and at conferences, methods and results are discussed conceptually with colleagues. This provides a valuable framework to ensure that the choice of research methods and the interpretation of results are sound. 

What is generally not systematically reviewed, however, are the actual data and code. To be fair, when data are complex, thorough quality checks that go beyond looking at random samples is not quickly done and requires significant experience. Code review is a valuable practice, but I’ve seen very few research groups implementing it. Often, researchers are too occupied with their own projects to engage deeply with others' work. From my experience, it is only when results are unexpected that supervisors and colleagues take a closer look at what has been done. 

The reliance on individuals for the execution and oversight of entire research projects does not make it easy to meet high quality standards. I once took over a former PhD student’s project and found it a nightmare to search for the right dataset, match code to data, and reproduce the statistics published in their monograph. Eventually, I had to redo the analysis from scratch. This is not to say that the former student did not produce good-quality work; it highlights how challenging quality assurance can be, even for an experienced researcher, if documentation and transparency are not prioritized. Mistakes can easily slip through the cracks. 

Peer Review 

While internal review provides a foundational check, peer review serves as the final gatekeeper for publication. The idea of peer review makes a lot of sense: Scientists check other scientists’ work and provide feedback.

What do reviewers actually do? Reviewers receive a PDF and are asked to judge the quality of its content, often because at least one aspect of the work—like a method or disease—overlaps with their expertise. They provide comments and the work only gets published when the original authors have adequately addressed them. The scientific community takes pride in this process, believing it prevents flawed work from being published and ensures the trustworthiness of what does get published. This is why peer-reviewed journals are generally respected sources of scientific knowledge. 

For researchers, peer review is both a curse and a blessing. Helpful peer reviews can provide valuable new perspectives and ideas for improving aspects of the manuscript. Even comments that are based on misunderstandings can be helpful as they highlight areas that need clarification. However, reviewers can also make unfeasible suggestions or suggest changes that serve to publicize their own work. Responding to reviews is often a balancing act between sticking to the aims and ideas of your work, being pragmatic and pleasing editors and reviewers. 

Limitations of Peer Review 

Nowadays, it is not easy for journals to find peer reviewers. I’ve been invited to review papers where I wasn’t sure why I was chosen. Sometimes, I knew something about the methods but not the application (or vice versa), and I’d disclose these limitations in my review. Yet, I never felt that editors considered this acknowledgment in their decisions. Editors nowadays are also struggling to find researchers willing to review manuscripts, so they cannot afford to be too strict with whom they accept as reviewers. 

Even when peer reviewers are well-qualified, they can only comment based on the information they receive. As a reviewer, you can’t confirm that the machines used were optimally calibrated, that no environmental factors interfered with the study, that standard operating procedures (SOPs) were followed, that all data were accurately entered, or that the code is bug-free. You simply don’t have the time or resources to verify every detail. 

Luckily, more and more journals now encourage or require authors to not only submit the PDF but also data and code. This is surely a step in the right direction, with the challenge that reviewers – if they can be found and if they can find a bit of time for review – are unlikely going to thoroughly checking these additional submissions. 

I remember my first paper review of a PDF taking me 2-3 hours because I was thorough, checking references to learn more if I wasn't certain about a statement. At some point, I heard that senior researchers could review a paper in under an hour. I recall thinking that their experience allowed them to quickly spot loopholes and flaws. Eventually, I also became quicker. It’s easy to be fast when the work is poor—you point out the major flaws and disregard the rest, as the paper likely won’t be published anyway. Good work, however, is more challenging to review. Under time pressure, I would only follow up on things that stood out, and I realized this process required a great deal of trust in the authors. If the paper wasn’t directly in my field, I had to trust that the authors didn’t overlook relevant literature and that their methodological parameters were appropriate. Still, as the reviewer, I was responsible for having “quality assured” this work and giving it a place in the scientific literature. 

Can We Really Trust? 

This is the fundamental question: Can we trust that authors have conducted their research thoroughly and responsibly? In a way, this should not be a problem. Scientists are highly motivated individuals, trying to find truths about the world. However, the current environment is characterized by immense pressure and an incentivization system that does not always reward thorough and honest research. It is focused around the one and only goal of getting as many papers as possible into journals that select for "shiny results". Researchers are increasingly facing the dilemma of doing responsible, thorough work and risk falling behind in the race for the next job or grant—or cutting corners to produce one more paper. 

Papers are how researchers present themselves to the world, but due to the selection of "shinyness" the content often needs to be taken with a pinch of salt. For example, when a figure caption says “Representative example dataset,” you can be pretty certain that the more truthful description would have been “The best-looking dataset of my project.” This is not to say that all science is unreliable, but it highlights the need to approach published work critically and understand the context in which it is created. 

We Do Complex Research; Honest errors Are Unavoidable 

Even if researchers conduct their work as responsibly as possible, errors can never be completely avoided. And indeed, many papers contain errors. A recent Nature publication reports that the number of retracted papers in 2023 exceeded 10,000. Retractions occur when a paper is discovered to be a sham or when a mistake invalidates a paper’s conclusions. I’m actually surprised this number is so high—not because I believe errors are rare, but because people typically don’t actively search for mistakes. They often find them accidentally. If more effort were put into finding errors, that number would likely be even higher. This number matters, because it challenges public trust in science and use of resources. 

What the public often does not realize is that errors are in a way part of the scientific process. This lack of realization is also the fault of the error culture we have in academia. Avoiding and constructively addressing errors should be an important part of the scientific workflow, but this is not always happening (also see this interesting piece of error culture). 

I once discovered that an analysis script used in several studies had an improperly set parameter. At the time, I was surprised no one had noticed, but now I see that it’s not surprising. Research is complex, with numerous small steps implemented by different people, making it easy to miss something. In this case, the incorrect parameter didn’t greatly impact the results, so it did not seem like a big deal, the parameter was corrected for future analyses and that was it. 

But it makes me wonder—if quality assurance is supposed to be our highest priority, what should I, as an individual researcher, do in such a situation? Should I convince everyone to redo all the analyses and write correction papers? Should I directly contact the journals if there is resistance from the original authors? And how would this affect my standing as a scientist if I caused so much trouble for others, especially when the scientific knowledge base is not majorly affected? Would this create more stress than it is worth? Or, on the contrary, would it set a good example for others? My feeling is that in the current system, the costs of following up on small mistakes massively outweigh the benefits. 

Possible Solutions 

Given the current challenges, it's clear that rethinking quality assurance is not just desirable but essential. To address these challenges effectively, we need to rethink our approach to quality assurance in science. Here are some potential solutions: 

1. Encourage Teamwork: The current system places too much responsibility on individuals. Promoting collaborative research environments with diverse expertise and shared responsibility can improve the quality of research outputs and their trustworthiness. Additionally, if each research group includes individuals who specialize in specific aspects of the research workflow, such as coding or experimental design, these experts can act as consultants for all projects in the group. This approach relies on creating more permanent academic positions, allowing these specialists to provide consistent and high-quality input across multiple projects without bearing sole responsibility for their outcomes.
   
   
2. Promote Open Science: Open science practices encourage transparency by making research protocols, data, and methods publicly available. This allows others to scrutinize the entire research process, verify results, and conduct thorough quality checks. With such information accessible, reliance on a single peer-review process during journal submission decreases. Open science also makes it easier to identify limitations and errors even after publication, promoting the continuous updating and self-correcting nature of the scientific process.
   
   
3. Implement More Rigorous Review Processes: While peer review is valuable, it should be enhanced with additional layers of scrutiny, such as code reviews and thorough checks on data management and quality. Given the increased workload this would entail, such measures are feasible only if we reduce the number of articles submitted to journals or if journals, which collect substantial fees, step up to support the process. This could include paying researchers for their reviewing efforts or hiring dedicated staff to handle aspects of quality assurance, such as code and data reviews. Such changes would not only enhance the reliability of published work but also make more effective use of the significant resources currently allocated to publishing.
   
   
4. Invest in institutional Quality Assurance: Research institutions and funders should allocate resources toward dedicated quality assurance processes. This might involve creating roles for research quality officers to oversee compliance with best practices and provide training in data management and analysis. Right now, this may seem unrealistic in practice, and one might ask, “How will institutions pay for this?” The amount of money they currently pay to publishers should be more than enough. If journals are unwilling to assume responsibilities like compensating reviewers or hiring dedicated staff for quality assurance—as outlined in point 3—then redirecting these funds directly toward supporting the scientific community becomes a valid alternative. This can only happen if the we move away from this journal-publication-centered-world.
   
   
5. Revise Incentive Structures: The academic system currently rewards quantity over quality in publications. A shift in focus is needed to prioritize reproducibility and open science practices in hiring and promotion decisions. Recognizing and valuing researchers who identify and correct errors in published work can further encourage accountability and foster a more trustworthy research culture.

These changes – if they happen at all - won’t happen overnight. They require commitment from researchers, institutions, funders, and publishers alike. However, by addressing the shortcomings in our quality assurance practices, we can build a more reliable scientific ecosystem that benefits researchers and society as a whole.