In W. P. Hanage’s article, he discusses the importance of five key questions when interpreting scientific literature:
- Can experiments detect differences that matter?
Hanage discusses, through an example using microbes, that in order to detect differences that matter researchers need to be able to identify functional differences in closely related genes, genes that they may not be familiar with. This suggests that they first need to know what they are looking for. In a broader sense, researchers must be able to distinguish and define what they intend to compare, otherwise the information gathered could mean anything.
- Does the study show causation or correlation?
Correlation does not prove causation. Hanage states that all scientists are taught this catechism. He also provides an example of a study whose authors proposed a causal relationship that fit the data clearly but did not explore other factors. One of the other factors that was not considered, which is significant in interpreting scientific literature, was the reverse causality. What is there to determine that this causal factor is not correlated, or even a bystander.
- What is the mechanism?
Mechanisms provide detail for a hypothesis. When something is well defined, a mechanism will be present. Experiments may seek out this information to understand the true influence of what is being researched.
- How much do experiments reflect reality?
What good is an experiment if it does not reflect what happens in the real world? First, by controlling the environment, researchers are able to look at specific interactions. Second, researchers acquire reliable data. However, this comes with a price — reliability. While the research may be valid, the reader must consider how well the experiment explains the real world. This is hard. The real world has so many confounding variables, whether influential or not, and as Hanage discusses, with respect to microbes, germ-free mice are often used. The animals and their microbiomes are adapted to a different niche than humans, so the results may not be generalizable.
- Could anything else explain the results?
This ties in well with do experiments reflect reality. The simple answer is they do not reflect it well. Many other things may help explain the results but were controlled for in the experiment. Hanage states that it is important for a critic to ask whether other contributors to, in this case, disease are considered, compared, and reported.
Which is most helpful when discussing controversy, and why?
I believe that the most helpful question to consider when discussing controversy is the mechanism. Controversy relies on correlation, and, as Hanage explains, the use of careful experiments to determine the mechanism and biochemical activity is crucial for understanding the true causes microbial influences may have. Causation is more likely to be assumed with mechanistic evidence. A close second would be an explanation of results. Sometimes mechanisms may not be found without the adverse effects of other factors. Discussing other contributors also may facilitate explaining real-world consequences.
