The first key question W. P. Hanage discusses in his article regarding interpreting scientific literature was “can experiments detect differences that matter?” Just because a difference is detected does not mean it is necessarily important to the problem at hand. It also does rule out the possibility of a much larger difference existing that just needs to be measured through a different metric.
The second was “does the study show causation or correlation?” Just because scientists are taught this phrase, does not mean they always act according to it. They used the example of how the scientists asserted that a person’s diet affected microbiome without examining if human health partially derived from diet was what actually impacted this. The third question, “what is the mechanism” is particularly pertinent to microbiome and probiotic research because so often claims will be made regarding x affecting y, but not how x affects y. Without a nailed down mechanism it is hard to say if this is actually the case and there is not another untested variable at play.
Question four, “How much do experiments reflect reality” is a bit of a catch 22. Scientists should design experiments that reduce the number of variables outside the treatment, however, this does not necessarily mirror the environment that they are trying to make conclusions on from their experiment. This makes it harder to assert that one’s conclusions actually apply to that environment. Finally, question five, “Could anything else explain the results?” speaks to the integrity of science. Doing experiments that support a hypothesis such as x affects y are just as important as doing an experiment to show the a, b, and c, do not affect y.
It is hard to say which question is most important when discussing controversey because of how interconnected all of them are. However, I would assert that question three, regarding mechanisms, is the most important. If a paper explains the exact mechanisms as to how x affects y with significant experimental data to back it up and makes reasonable conclusions based off of it, it’s hard for scientists to disagree with it in a meaningful way. So much of the controversy surronding microbiome science, and science in general, is in regards to scientists or journalists over concluding shakey results that do little to explain what was mechanistically going on. Much of this is stripped away by simply doing good mechanistic science and making reasonable conclusions based off of it.
I felt like this paper really connected with the course content through what Dr. Massoni said in one of the first classes. He stated that the discussion and conclusion sections of a paper are an opinion piece and to look at a paper through an appropriately skeptical lens, it is best to make your own conclusions based off of the results. A lot of the questions found in the W. P. Hanage article seem focused wading through these discussions and conclusion sections, but if you just focus on the results most of what one needs to worry about is what questions does the experimental design help to answer and, by extension, if it is designed in a correlative or causative manner.
