In this post, I’ll be discussing the significance of five important questions which one should consider while interpreting scientific literature.
- Can experiments detect differences that matter?
This point is important, because it’s these differences which allow results to produced and made apparent through empirical study. Without the inclusion of statistically significant deviations from one group to another, any claims laid are unsupported conjecture, and may be considered next to useless in many forums.
- Does the study show causation or correlation?
The distinction between these two terms is crucial for determining the effects which things have on one another, and is as important to understand as the closely related distinction between fact and superstition. For example, every adult in the history of the world who has died has consumed water, but this doesn’t mean that water is responsible for their deaths.
- What is the mechanism?
This question looks more closely at the connections between observations and their results, and considers the physical conditions and process which lead to the reproducible observations under question. Ultimately, it lends deeper understanding to perceived connections.
- How much do experiments reflect reality?
The importance of this question almost goes without saying. After all, what’s the point of performing an experiment with nigh on useless results? I sure wouldn’t want a child who’s only medical experience was playing the game “Operation!” (adept as they may be at it) to start cutting things out of me. This is the same Idea.
- Could anything else explain the results?
Finally, This is a condition that has been analyzed in statistics and many other fields for quite some time. At its core, this question is related to many others, like determining causation vs correlation, or finding the mechanism. As such, I believe it is likely the most important question to consider while interpreting scientific literature. No matter how many of the other questions are adequately answered by a study, if there’s something else out that there that is equally or more likely to produce the same results, more experiments need to be performed, and more data obtained.
