Can experiments detect differences that matter?
This is significant when interpreting scientific literature because it indicates proper study design, with controls as well as controlled variables and random samples. If there were a study that happened to find a difference between two groups of individuals, but didn’t parce out what actually caused that difference, there is very little that we can draw on from that study, other than the foundations for another study.
Does the study show causation or correlation?
The significance of this phrase is more geared towards the conclusions that we can draw. There is a massive difference between the two, as well as the in the aftermath of their respective studies what we are able to do. If we determine causation, we are given clear direction, and often (but not always) the mechanisms to target, for an effective therapy. If we only find correlation, then our hands are more or less tied, as we can’t be sure that the therapies will be effective, or if they might exacerbate the issue. Overall, I feel that this is the most important when discussing controversial topics, as it helps to delineate whether or not something has been weakly associated versus strongly associated with actionable plans for the future. Also, it helps to lend credence to these more strongly associated topics, and they are less likely to be shot down by the opposing side.
What is the mechanism?
This is important for developing therapies as well. If we know that something is causing something else, but we don’t know how, we have to do further research in order to find out what to do to actually address the problem. There are certainly times where we don’t necessarily need to understand the mechanism behind a therapy to know that it works, and because of that I think that it is less important for controversial topics.
How much do experiments reflect reality?
This calls back to experimental design, but it really focuses on the applicability of the conclusions. Very rarely do scientists spend the time and money to conduct worthless, inapplicable experiments. There are times when results can perhaps be called into question, for example with animal model studies and their relation to human applications, but overall science has come to the conclusion that these are generally acceptable as experimental designs when it comes to drawing correlative conclusions.
Could anything else explain the results?
And once again, study design is in question, although this issue has to do with the thought put behind the study. I think that this is very similar to the ‘differences that matter’ and the ‘mechanisms’ issues, as it sort of combines aspects of the two. The former gives us issues with determination of results and the sample design of studies, and the latter gives us issues surrounding the next steps that we take in the future.
