Can experiments detect differences that matter?
This is the first question listed by Hanage for good reason; it is a basal question. It is relevant because there would be no point to carrying out experiments if they would not be able to lead you to a conclusion. Really, this is more of a question for the researchers than the one reading the article, but it is still valid and calls into question the purpose of the study.
Does the study show causation or correlation?
This question is important to ask as it allows you to gauge how much the results should influence your thinking. If a study is a randomized control sample with a large sample size and clear question, it is more likely to establish a causal result. However, most studies do not meet the strenuous requirements to draw causation so they must be taken with a grain of salt. This ensures an appropriate level of skepticism is applied as it is easy to get carried away when relationships are demonstrated in mice, for example. Though the relationship is very shiny and nice looking it is important to remember it is not proven yet.
What is the mechanism?
This is another important question to ask to ensure an appropriate level of skepticism is applied. This question also allows you to double check if the study makes sense. Some mechanisms are definitely proposed that are far reaching and often data hands us relationships that we cannot explain. Therefore, this is an important factor to think of when considering a scientific study.
How much do experiments reflect reality?
This question’s main purpose is to establish practicality. This is something very applicable to scientific literature as more often than not, lab conditions are far from real conditions. The only way to convert all the knowledge gained from scientific studies into real improvement is by asking this question. This allows us to apply knowledge gained to the real world.
Could anything else explain the results?
Finally, this question ensures the efficacy of results. If a confounding variable could even possibly be at play there is room for criticism. This is healthy as we want to ensure scientific data is reliable and evidence backed. It is not that the study must not have those other possible variables, but it is necessary to consider them when thinking about the study. The question contributes to the overall purpose of the other questions, to ensure data is valid and accurate.
The most helpful question when discussing controversy specifically is the last one. A piece of information cannot be held reliable if there are other explanations that could be drawn from the data. To explain it in a different way, if you could answer question five, Could anything else explain the results, with a confident no, there would be no room for controversy. It was quite disheartening to read Hanage’s editorial as his skepticism made me question many microbe studies, though the skepticism is healthy.
