I recently got into an online debate with some anti-vaxxers. What makes me so angry is not which side of the argument people take, but their total noncomprehension of biostatistics. People have no idea what it means to prove something scientifically. I'm going to put in an excerpt from Chapter 3 of my book. May a science journalist please read it and stop publishing nonsense....
A touch of statistical thinking is all you need to know to
see through the nonsense of many articles that incorrectly misuse statistics
and research. A real statistician might find my hints about statistics criminal
in oversimplification. But the truth is plenty of doctors can barely do
statistics. If you train at a top tier medical school there will be a heavy
emphasis on research. Lower tier medical schools tend to work as if they were
going to create clinicians only. I think this pattern is wrong on many levels.
I do not think a medical doctor should have to add research to their career
beyond training; however you really only learn to understand by doing in many
cases. When I left Penn I went to Technion because of its emphasis on research.
The amount of new material in medicine grows exponentially. There are now
always over a hundred new studies in any given field coming out. Any good
doctor who stays good past medical school has a few specific characteristics.
One is the ability to comprehend and weigh new research. In the American system
new research cannot change the way you practice medicine. In the US every
doctor dreads being hauled into court. At that point you are going to prey
whatever you did was in line with the guidelines for your specialty. But what
about those of us who want to do more, to give our patients the very, very best
medicine has to offer. When you have exhausted everything within the
guidelines, the treatments that might work are experimental. But knowing what
works and what is oh so much pharm company deception is a matter or
understanding statistics.
Let’s start with the
big picture. What is science? Science is the best method humanity has developed
to understand reality. Science is a process, not a group of static facts.
Science as a field is a sort of competition to see who can get the truth about
any given reality. The referee in the competition is always statistics.
Statistical analysis is central to the process of all sciences. The process of
science can be distilled to a sequence. Here it is: the science six step:
1. Observe stuff
2. Make some ideas about why things work the way they do
3. Refine these ideas into stuff you can actually test (hypotheses)
4. Examine the existing evidence for your hypotheses
5. If there is not enough evidence to draw firm conclusions, do some experiments
6. Evaluate the data already existing and possibly data from your experiments
So there you
have it. It doesn’t really matter that much whether we are talking about
something as esoteric as quantum mechanics or something as banal as biology.
Science is the same. The process is universal. Ultimately it’s hard to know
what reality is, but you can very easily figure out what it isn’t. If you
understand the mistakes people make in the statistical analyses of studies, you
can see through a lot of nonsense. People, and even a lot of scientists,
usually trip on step number six in my schema of science. To evaluate data you
need statistical analysis. But a really good scientist is using statistical
knowledge from at least step four forward.
Not everyone needs to or should work as a scientist. The
world could probably get along without us easier than it could get along
without farmers. Our lives are enhanced by the millions of people who are not
scientists. Artists, cake bakers, well diggers and so on are all contributing a
lot to society. The problem is that all these people, and even their leadership
frequently misunderstand science. Those we have trusted to interpret science
for them, often journalists with limited training in science, often
misrepresent it and its results. I think none of this is going to stop anytime
soon due to the poor state of education in the USA. Fortunately with a little
training even a well digger can see through the nonsense that flies into print.
Many science articles are full of mistakes.
The three most useful to understand mistakes in scientific
studies in my opinion are selection bias, recall bias and what I call
underpowering. If you can understand these three concepts, the next time
someone points out something to like the fact that her grandmother and mother
both conceived past forty as proof that you have nothing to worry about at 35,
you will be able to cut them down to size and tell them about reality.
A recent example of bad research I saw was an article a
friend posted to Facebook. The article claimed proof that nonvaccinated
children were healthier than vaccinated ones. All participants were children of
parents in a parenting group dedicated to not vaccinating their children. If
you have training in statistics at all you can see what a scientific train
wreck the article was already, and you may want to skip the explanation in the
next paragraph.
The article is a perfect example of a lot of problems.
Selection bias: is a group of parents in an anti-vaccine group representative
of the general population? What is half of them got thereafter kid number one
got an illness they linked in their mind to vaccination? Recall bias: In
recording disease and health people don’t recall things exactly the way they
were. The classic example is the mothers of deformed babies. They remember
themselves as having taken medicines during pregnancy, and wonder if they made
the wrong choice. The truth is most women take some medicine during pregnancy.
Most women who get normal kids tend to forget they took something. The woman
with the formed baby will sit there and guilt trip herself out over every time
she even looked at acetaminophen. We see what we want to see, or sometimes
don’t want to see, when we look back. The parents in the study my friend posted
to facebook recorded their children’s diseases. If they had a preconceived notion
that their unvaccinated kid was always healthy, they may have forgotten a few
illnesses. Finally the article was weak because it was not powered to prove
anything to begin with. Power, in terms of a study, could be thought of as the
ability to proclaim the conclusions of the study scientifically true. Power is, to be precise, the
probability that the study will correctly lead to the rejection of the false
null hypothesis. The null hypothesis is that the conclusion asserted was not
true. A study gets more power based on the sample size. The size of an
effect also effects power, but for a study like this one on vaccines the main
power issue was sample size. The next time someone tells you about one case of
something happening as proof, think of it as a study of one single subject. How
much power does a study of one single subject have? Very, very little;
essentially none. The next time someone makes a scientific assertion about a
whole population to you based on one single case rest assured that it proves
not much except that whoever told it to you doesn’t understand science.
I am passionate about science. But we don’t need scientific
studies to tell us everything. Many things are obvious. For example , we don’t
need a scientific study to tell us that women between 60 and 70 don’t get
pregnant naturally. It’s obvious. Nonetheless many bad scientists try to sneak
claims nearly as outrageous into the literature. I feel we would all be better off if we all
learned biostatistics; but realistically it won’t happen. To help dispel any
future myths produced in the future by bad scientists and uninformed science
writers I am including a checklist to evaluate supposedly scientific claims on
fertility.
1.
What are the author’s
scientific credentials? Circle one: an MD or DO; a PhD; an MPH or MS in
biostatistics; this author has no known scientific credentials
Anyone can write something that is
correct, but if the author is not trained in biostatistics it is highly likely
he or she might unknowingly misrepresent the situation. On the other hand
plenty of PhDs make science mistakes. Just having a degree behind your name
does not prove much.
2.
Is the author’s claim in line
with what we see in reality?
Extraordinary claims require extraordinary
evidence. The more outlandish a claim is the more proof it should require. If
someone claims most women can give birth after menopause if they simply take
vitamins, you would want to see the highest levels of scientific proof for such
a claim. In this case the highest level of proof would be several studies
including a blinded randomized controlled trial on many subjects.
3.
If there is a study cited,
does the study have enough subjects?
Single case reports, or case series prove
very little. Any one person can be an anomalie. Generally speaking reliable
trials include hundreds of subjects. If someone tells you about a study with 5
people, it could be interesting; but it couldn’t be powered to prove much
almost by definition.
4.
Was the study published in
a reputable journal?
There are plenty of nonsense journals which
will publish anything. And with the advent of the internet anything is
publishable. The peer review process is not perfect, but like democracy, it’s
basically the best system we have come up with until this point. If the study
was published in a very disreputable journal, beware!
5.
Are the studies author’s
funded in such a way that they have a financial interest in proving a certain
point?
It is possible that an author does
a great study and just happens to prove whatever point the pharm company paying
him wants proven because it is true. On the other hand it happens too often to
be a chance occurance. Dr. Ben Goldacre has written about such phenomenon
extensively. For lots of good highly readable information about the pitfalls of
bad science he can’t be beat.
No comments:
Post a Comment