Science thrives on critical self-control – results and analyzes should be reproducible. Scientists have now tested how well this works in an experiment: They let 70 research teams independently analyze the same 108 brain scan data sets. The task was to confirm or refute nine hypotheses based on the data. It turned out that the interim results after the data evaluation were still relatively similar; there were significant discrepancies in the hypotheses in five out of nine. Transparency and critical review are therefore particularly important for complex data, the researchers say.
When it comes to complex facts or new phenomena, scientists often only work gradually to understand the whole picture. The current corona pandemic illustrates this typical process of gaining knowledge very clearly: initially, little was known about the virus and its effects, and the data were based on small, often local studies. As more data has been added, some initial assumptions have been confirmed, others have had to be revised or withdrawn. In the course of the pandemic, the statements and recommendations of the experts have also changed in part – keyword masks. This caused some misunderstanding and uncertainty among the population. However, the ability to continually review old assumptions based on new findings and correct them if necessary is ultimately the engine of scientific progress – and an important pillar of internal quality control.
108 brain scans and nine hypotheses
But what happens when 70 research teams independently analyze the same data sets to test the same hypotheses? An international team of researchers has now tried this out in practice. “The scientific process comprises many steps: A theory is developed, hypotheses are created, and finally data is collected and evaluated,” explains co-author Simon Eickhoff from Research Center Jülich. “Each of these steps can potentially influence the final conclusions, but to what extent? For example, will different researchers come to different conclusions based on the same data and hypotheses? ”To test this, the researchers, led by Tom Schonberg from the University of Tel Aviv, selected images of the brain activity of 108 subjects who participated in a neuropsychological study had. These images, taken with functional magnetic resonance imaging (fMRI), showed which areas of the brain were active when the participants made certain financial decisions.
The scientists sent these 108 data sets to 70 research teams around the world. Each team analyzed this data using their respective standard methods and checked nine predefined hypotheses based on the results. As a final result, you should answer the hypotheses with either yes or no. “Each of these hypotheses asked how certain aspects of decision making affect brain activity,” explains Eickhoff. The analysis teams had three months to evaluate the data. They then provided Schonberg and his team with their results for the various hypotheses, their interim results and detailed information about their approach to the analysis.
Deviations in the conclusions despite similar analysis results
The comparisons of all results sometimes showed clear differences: in five of the hypotheses, the team’s conclusions differed significantly, for the remaining four there tended to be agreement. “Interestingly, the data sets on which the teams base the analyzes still show a relatively high degree of agreement between all teams,” explains co-author Felix Holzmeister from the University of Innsbruck. This was also confirmed by a meta-analysis that compared all data analyzes and interim results carried out by the teams. It showed a high convergence of the evaluations and the brain activation maps based on the data. But because the research teams had to reduce these complex results to simple yes-no decisions regarding the nine hypotheses, the deviations occurred. “This particular problem of analysis affects all areas in which highly complex data is used, which ultimately has to be reduced to a bare yes-no result,” explains Eickhoff.
This study thus underlines that the type of analysis and evaluation, especially with complex data, can have an impact on the result. “The reason for this is that researchers with such complex data sets have to make many individual decisions on how to process, organize, model, and analyze the data,” says Eickhoff. At the same time, the study confirms how important it is that raw data is shared and that analyzes and conclusions are repeated and checked by colleagues, which is a common process in many fields. “This process of self-reflection and the continuous improvement of our own methods is unique and distinguishes science”, emphasize the co-authors Michael Kirchler and Jürgen Huber from the University of Innsbruck. The fact that almost 200 scientists were willing to invest hundreds of hours in this experiment shows how willing they are to do so. “But this study also underlines how important it is that we scientists are always aware of the possible discrepancies in complex questions,” says Huber.
Source: Tom Schonberg (Tel Aviv University) et al., Nature, doi: 10.1038 / s41586-020-2314-9