A study’s external validity is threatened if there may be systematic
error in the way its results can be applied to patients outside the
precise study set. We should always be concerned about external validity
if a study deals with a group of patients who are noticeably different
in essential characteristics, relevant to the goal of the study, than
the type of patients to whom we ourselves might be interested in
applying the results.
Many studies try to evaluate the use of tests to find disease of one
sort or another. “Tests” are of course not limited to laboratory studies
or X-rays, but for the purposes of this type of research may include a
given historical finding, an abnormality on physical examination, a set
of “high-yield criteria” or anything we can add to our prior knowledge
of patients to further discriminate between those who do or do not have
the disease in question.
The importance of sample size is well known in medical research. Use
very large samples when comparing two treatments and you will find
“true” differences so small as to be unimportant. This month we are going
to explore the concept of sample size and discuss ways to read between
the lines when analyzing study results.
Some of the major problems in studies have to do with misuse of
statistics. Descriptive statistics were originally applied to medical
research to control for ways in which the study group, by chance alone,
might not exactly reflect the whole universe of patients to whom the
results might be applied.
When reading the literature, do you ever wonder who really has the
right answers? How can readers come to different conclusions reading
the same data? While it is tempting to simply agree with an author’s
conclusions, they are often invested in their own work, introducing the
potential for bias. Confounding variables are one source of bias that
can easily alter the conclusions.
Part 2 in a series - Continuing our discussion on how to understand the literature that we
read, we move to a way in which internal validity is threatened:
improper classification of results. This can be because of lack of an
adequate gold standard, because of biased estimation of results
(usually in the absence of blinding), or because of imprecise or
irreproducible measurement of results.
Much of the research that we read in medical journals should not be
taken at face value, because of a series of errors in study design,
analysis (often involving misuse of statistics), and/ or interpretation
or extrapolation. By understanding a few basic concepts we can become
adept at interpreting the validity of most of what we read.