NPR logo Rigor Mortis

Books

Rigor Mortis

How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

by Richard Harris

Hardcover, 278 pages, Perseus Books Group, List Price: $28 |

purchase

Buy Featured Book

Title
Rigor Mortis
Subtitle
How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions
Author
Richard Harris

Your purchase helps support NPR programming. How?

NPR Summary

American taxpayers spend $30 billion annually funding biomedical research. By some estimates, half of the results from these studies can't be replicated elsewhere — the science is simply wrong. Science journalist Richard F. Harris reveals these urgent issues with vivid anecdotes, personal stories and interviews with the nation's top biomedical researchers.

Read an excerpt of this book

NPR stories about Rigor Mortis

Mick Wiggins/Ikon Images/Getty Images

How Flawed Science Is Undermining Good Medicine

  • Download
  • <iframe src="https://www.npr.org/player/embed/522262881/522826578" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript

Note: Book excerpts are provided by the publisher and may contain language some find offensive.

Excerpt: Rigor Mortis

When Carolyn Compton was working as a pathologist at Massachusetts General Hospital, one of the world's most celebrated medical centers, she never knew when a cancerous colon removed in an operating room would show up at her lab. It could take days. "I can tell you that there was no urgency" to get a colon from the operating room to the pathologist who would diagnose the disease. "A big colon gets put into a bag. It sits in the operating room until a circulating nurse gets around to putting it in the holding refrigerator in the operating room. At the end of the day the same guy who delivers the mail at Mass General comes around and puts it in a cart. He takes it two buildings over, to the pathology department. There it goes onto another bench to get logged in by a technician and it goes into a refrigerator. If it's a three-day weekend, the resident on call doesn't come in until Tuesday, opens up the colon and takes a piece of the cancer and puts it into formalin," a preservative.

This delay was not a problem for the patient when Compton worked at Harvard in the 1990s, and it isn't one today. Pathologists can still stain that piece of colon, study it under the microscope, and diagnose the type and stage of cancer. "That [drawn-out process] met the standard of care, and still does," Compton told me. But she has gradually come to realize that this rather casual attention to tissue collection and preservation spells real trouble for biomedical research that might be conducted on a sample of colon cancer. These tissues are perishable, so studies that depend on fine molecular measurements are likely to be irreproducible.

These days, scientists are trying to wring a great deal more information out of the tissue than they can see through a microscope. Precision medicine could potentially correlate specific snippets of DNA, proteins, and other molecules with disease diagnosis or prognosis. Many of these molecules are quite fragile. Compton says even the anesthesia used in the operating room to knock out the patient can affect them. These molecules can change more when surgeons cut off the blood supply to the tissue to be removed. And once the organ is out of the body, the stability of those critical biological molecules will vary depending on the room temperature and—significantly—the amount of time the tissue sits around before it's preserved. "We will not have precision medicine unless we can fix this problem," Compton said.

Compton, now at Arizona State University, says it's taken quite a while for pathologists to realize just how important all those factors could be. One wake-up call came from the lab of David Hicks at the University of Rochester. Starting in 2006, he was trying to unravel a serious medical mystery. The Food and Drug Administration (FDA) had approved a test to help diagnose a particular variant of breast cancer, called HER2-positive. The test itself was certified as very accurate and reliable. Yet about 20 percent of the time it reported that a tissue sample lacked the HER2 trait, when in fact it was present; and up to 20 percent of the time the test "found" the HER2 trait even though it was not there. That's bad either way. Either women who could benefit from Herceptin, a drug that targets HER2, weren't getting it, or women were receiving an expensive drug that was not only worthless to them but also had side effects.

But if the test itself wasn't at fault, the problem must lie in its use. Hicks's colleagues solved at least part of that mystery with a simple experiment. They let breast tissue from biopsies sit out for an hour or two before testing it. And that was enough to degrade the sample and turn a positive result into a negative one. The molecule detected by the HER2 test breaks down at room temperature. "You can have the best test in the world and still get the wrong answer if you bugger up what you are testing," Compton said.

That observation sparked action. Two leading professional societies, representing pathologists and clinical oncologists, drew up new rules in 2010 that, among other improvements to reduce false readings, required preservation of breast tissue within an hour of surgery. That has helped make the HER2 test significantly more reliable. But Compton noted with exasperation that breast cancer is the only cancer for which doctors are required to pay attention to the clock after surgically removing tissue. There are no standards for treating samples from the more than two hundred other forms of cancer. She's not simply concerned about patient care. She's thinking about how to improve the reliability of scientific research based on those samples.

Compton explained that pathologists come in two varieties: clinical pathologists, who diagnose disease, and anatomic pathologists, who do research. Clinical pathologists follow long lists of federal standards and professional practices. "In laboratory medicine, accuracy of measurement and reproducibility of measurement is everything. Everything," she said. "In fact all regulation related to laboratory medicine is focused on your ability to calibrate and reproduce reliable analytic results from run to run from day to day from lab to lab." After all, an error can lead to misdiagnosis, with life-and-death consequences. "You would never believe a result from a [blood sample] if you hadn't handled the blood tube properly. You would just order another draw."

Anatomical pathologists and allied medical researchers, on the other hand, tend not to think as much about the quality of their starting materials. "They would come to the pathology department and say, 'Can I get twenty [paraffin-preserved] blocks of colon cancer?'" Compton said. "They were glad to get their hands on anything." They would take those starting materials back to their labs, "spend a huge amount of time and money analyzing them and then get results that nobody could interpret—and they never could actually interpret!" Compton says there are no national standards for handling tissue in research labs.

Medical researchers are already getting a taste of the daunting problems that will affect precision medicine if they aren't absolutely scrupulous about sample collection. Paul Tempst and colleagues at the Memorial Sloan Kettering Cancer Center in New York City were running a study involving proteins taken from blood samples. The scientists were initially encouraged to find a difference between blood samples taken from cancer patients and those taken from healthy people, but Tempst became concerned that the result might simply reflect a batch effect. Sure enough, he eventually tracked down an unexpected culprit: the test tubes in which the blood had been collected. Tempst realized that samples from the healthy patients were collected in a clinic, while samples from people with cancer came from a hospital. And it turned out that the hospital used one type of test tube to collect blood, the clinic another. That seemingly trivial difference was enough to render his results meaningless.

Collecting the starting materials correctly is an essential first step, but it's just a start. Carolyn Compton started thinking about these issues when she was working at the National Cancer Institute (NCI). She was not alone. Her friend, Anna Barker, then NCI deputy director, was growing alarmed as well. After a dinner conversation veered deeply into the topic, "that really set me on a path to say, 'Let's create some standard operating procedures and best practices,'" Barker told me. Samples were one thing, but "how do you collect the data? How do you exchange the data? How do you analyze the data?" Individual researchers invent their own ways to do this. "Everybody wants to do their own thing," Barker said. Even though that's the ethos of biomedical research, she realized that simply would not work once scientists tried to pool their data. That way lay mayhem.

With that in mind, in 2004 Barker set out to assemble The Cancer Genome Atlas (TCGA), a massive compendium that would map out a multitude of genetic changes associated with various cancers. And to make sure the data were comparable, she didn't fund individual investigators. Instead, she contracted with researchers to perform specific tasks and to meet specific standards. "It was based on creating a situation where we would get reproducible data. So that means we controlled everything about this project," she said. Barker specified how tissue would be collected, handled, stored, and sampled, how the DNA would be sequenced, and how the results would be analyzed. The work may not have felt creative to the scientists doing it, but Barker came away from the decade long project with a sense that she had not only gathered a pile of useful data but convinced scientists to work toward one collective goal rather than pursuing individual projects.

That experience is by far the exception. Scientists are reluctant to create standards and even slower to adopt them. Something as commonsensical as authenticating cell lines has been a slog. Yet standards are hardly a new idea in science and technology. "We have tons of standards. We have more standards than you'd ever want to think about," Barker said, for everything from lightbulbs and USB ports to food purity. But they don't permeate biomedical research. "How many standards do we have in whole genome sequencing? That would be none at this point," she said. Or how about a standard to search for mutations in genome sequences? "Nobody does it the same way." Scientists have proposed many standards over the years, but their colleagues are often unaware of them, and, in any case, there's no easy way to impose them on an entire field.

Biomedical science just can't keep on going this way. "Biology has become quantitative," she said. "That's a transition that we're just beginning, and it's going to leave a lot of people behind. We haven't trained a lot of our biologists to think mathematically or to understand or analyze data. Most people understand that we're in the digital revolution. Well, your genome is digital data."

From Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions by Richard Harris. Excerpted by permission of Basic Books.