Seeking Answers In A Maze Of Health Studies
We can all be forgiven these days for having a little whiplash when it comes to studies about our health. Ever hear that wine or Vitamin E or even a little pollution is good for you, only to hear that claim quickly contradicted? Statistician Stan Young speaks with host Guy Raz about searching for verifiable health patterns in a world of limitless data.
GUY RAZ, host:
Almost every day, a new study comes out that seems to prove something is either very bad for you or very good for you like chocolate.
Unidentified Woman #1: A way to your Valentine's heart may be to give a gift that is good for their heart. Some experts suggest dark chocolate fits the bill.
RAZ: Or coffee.
Unidentified Woman #2: That morning cup of Joe, cradle it, love it, appreciate it, because the caffeine in it might help you reduce your risk of Alzheimer's.
RAZ: Now, there's plenty of evidence to suggest a cup of coffee or a piece of chocolate might not be a bad idea in moderation.
But according to Stan Young, a researcher with the National Institute of Statistical Sciences, the odds are that if you look for certain patterns long enough, you're bound to find them.
Stan Young has made a project of poring through serious medical studies to find out whether their conclusions are actually conclusive. And in most cases, he says, they're not.
Dr. STAN YOUNG (Statistician, National Institute of Statistical Sciences): Probably the funniest claim that I came across was the claim made in a British magazine, the Royal Society B, that women that eat breakfast cereal in and around the time of conception are more likely to have a boy child.
RAZ: And this is a reputable publication?
Dr. YOUNG: Royal Society B is essentially the equivalent of our Proceedings of the National Academy of Science. Yes, it's a very reputable publication. It went through a peer review, went through editorial and so forth.
RAZ: So I guess the idea is if you have enough, you know, information about people's lifestyles and the choices they make, and then you combine that with the power of computers to run thousands of tests, you're bound to find some odd correlations.
Dr. YOUNG: Indeed you will. You'll find all kinds of things and then it's a matter of how clever you are at writing. If you can put a good story around the probably random finding that you found, you got a paper.
RAZ: On this program and, as you know, many other news programs, journalists go to scientists researchers who have published peer review studies in these prestigious medical journals. We trust them. Should we then just disregard these studies?
Dr. YOUNG: Well, there are really two kinds of studies and they're often sort of mashed together and they shouldn't be. In a randomized clinical trial, there's one question, you collect the data, there's no bias, you get a clean answer. Observational studies are a different kettle of fish. You can ask hundreds of questions, there can be bias and other problems. And then you can pick and choose amongst all the data, what you want to report in a paper.
The point I'm trying to make with my research is that there are systemic problems with observational studies.
RAZ: What percentage of overall scientific studies are observational versus randomized?
Dr. YOUNG: In the medical area, which I study the most, probably the majority of studies are observational.
RAZ: So, how do you propose that the scientific community address this problem? I mean, how should they re-approach the way they carry out large-scale studies when they can't do anything but observational studies?
Dr. YOUNG: Edwards Deming was a famous statistician that went into Japan after the Second World War and revamped their entire industrial process. He said that when a system is out of control - and I'm making the claim that observational studies, if they're wrong 95 percent of the time, are out of control - he said you can't correct a system by talking to the workers. You have to talk to management.
And in this case, management or the funding agencies that fund these observational studies and the peer review journals that, you know, publish the results, they're the two groups of individuals that can't fix the system.
Another very good step in the direction of fixing things is to require that when an observational study is published that the data used to make those claims be publicly available so that trained people can, you know, examine the data and see if they reach the same conclusions.
RAZ: That's Stanley Young. He's assistant director for bioinformatics at the National Institute of Statistical Sciences.
Dr. Young, thank you so much.
Dr. YOUNG: Okay. Thank you.
Copyright © 2010 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.