Tricare Investigation - Chapter 3: Pentagon Contractor Finds Therapy InconclusiveECRI Institute's methodology is called into question after they find limited evidence of the benefits of cognitive rehabilitation for those suffering from traumatic brain injuries.
Soon after the consensus meeting, Tricare sprang into action. In May 2009, records show, it issued a $21,000 contract to the ECRI Institute, a respected nonprofit research center best known for evaluating the safety of medical devices.
The contract called for ECRI to review the available scientific literature to weigh the evidence for whether cognitive rehabilitation therapy helped improve patients with traumatic brain injuries.
Tricare routinely hires contractors to carry out assessments to help determine which medical treatments to fund. But in selecting ECRI, Tricare had a pretty good idea of the response it would receive. ECRI had conducted a similar review for Tricare in 2007 that cast doubts on the evidence supporting cognitive rehabilitation therapy.
To carry out the new review, ECRI followed its standard protocol. It chose to include only randomized, controlled studies. Such studies randomly divide patients into groups that receive different treatments in order to compare their effects.
ECRI gave more credence to blind studies, meaning that patients did not know whether they were receiving genuine therapy or a placebo — a fake treatment. Blinding reduces bias and is considered one of the most rigorous standards that can be used in scientific testing.
ECRI also excluded studies deemed irrelevant; those studies with fewer than 10 patients; and studies where 15 percent or more of the patients were injured from a nontraumatic blow, such as stroke.
The criteria resulted in the elimination of much of the published scientific literature on cognitive rehabilitative therapy. Before applying the protocol, ECRI identified 318 articles as potential sources of information about cognitive rehabilitative therapy. The firm's final report examined 18.
Based on this limited pool, ECRI graded the evidence for the benefits of cognitive therapy as being "inconclusive" or offering only "low" or "moderate" support of improvement in patients' cognitive functions.
The final report, delivered to Tricare in October 2009, noted some areas of benefit. For instance, "tentative" evidence showed cognitive therapy significantly improved quality of life for brain-damaged patients.
ECRI's review wasn't limited only to science. The review noted one study that found that comprehensive cognitive rehabilitative therapy could cost as much as $51,480 per patient. By contrast, sending patients home from the hospital to get a weekly phone call from a therapist amounted to only $504 per patient.
Overall, the report concluded, the evidence for most benefits from cognitive rehabilitation therapy remained inconclusive, especially when compared to cheaper programs.
"The evidence is insufficient to determine if comprehensive, holistic (cognitive rehabilitation therapy) is more effective than less intensive care" in helping patients, the 2009 report concluded.
By the summer 2009, ECRI researchers had finished a draft of the study. ECRI, later joined by Tricare, asked outside scientific experts to review it.
The reviews, according to interviews and copies obtained by NPR and ProPublica, were uniformly critical.
(NPR and ProPublica obtained a copy of the ECRI reports through the Freedom of Information Act. However, Tricare denied access to reviews of the reports. ProPublica and NPR have appealed the request, but obtained copies of the reports and information on the reports from sources.)
The reviewers acknowledged that more research was needed on cognitive rehabilitation therapy. However, they noted that the Tricare report ran counter to several other so-called meta-analyses, which combine multiple, individual scientific studies to achieve greater statistical reliability.
For instance, a 2005 article in the Archives of Physical Medicine and Rehabilitation, a peer-reviewed journal that is one of the mostly widely respected in the field, examined 258 studies. It concluded that "substantial evidence" supported cognitive rehabilitation. The review included 46 randomized control studies — more than double the number in the Tricare study.
Reviewer Keith Cicerone, a leading civilian researcher who runs the JFK Johnson Rehabilitation Institute's Center for Head Injuries in New Jersey, disputed Tricare's contention that the treatment was new and untested.
"We have a significant body of evidence describing cognitive rehabilitation and showing what works in cognitive rehabilitation," Cicerone said. "The idea that cognitive rehabilitation is new and untested is simply not true. It's got a better evidence base than most things that we do in rehabilitation."
Asked to explain in plain terms, Cicerone grew animated: "The arguments that are being made against" cognitive rehabilitation "in terms of the level of research that has been conducted are hooey," he said. "It is baloney."
The outside experts also attacked Tricare and ECRI for relying upon a methodology that ruled out important research. ECRI's protocols, they acknowledged, are well-suited for drug studies, where it is easy to prevent patients from knowing which pill they are receiving.
But ECRI's protocols do a poor job in assessing rehabilitation therapy where patients and doctors constantly interact in face-to-face treatment sessions. Other well-accepted methodologies, they said, have been designed to examine the benefits of therapeutic interventions.
They also questioned the reasons for excluding studies with a small number of patients, or with differing causes for brain injury, since a stroke can produce the same types of symptoms as a blow to the head.
Malec, the research director at the Rehabilitation Hospital of Indiana, said Tricare's study sounded like it came from a private insurance company seeking to cut costs. His review said that Tricare's study "fails to represent the evidence relevant to evaluating the effectiveness of cognitive rehabilitation after traumatic brain injury."
In an interview, he said Tricare's demand for conclusive evidence was understandable, but ill-advised. While research continues, existing evidence indicates that the therapy helps, with no studies showing that it harms troops.
"They missed the forest for the trees. They missed the big picture," he said.
Some of the researchers accused Tricare of using ECRI's strict assessment protocols as a cover to justify denying troops' coverage.
Wayne Gordon, director of rehabilitation psychology and neuropsychology services at Mt. Sinai School of Medicine in New York, called the review "dismaying" and "unacceptable." He compared it to tobacco companies that dismissed studies that showed a link between smoking and cancer.
"The ECRI Institute seems to be stating that, while sufficient evidence exists for there to be consensus among diverse groups that cognitive rehabilitation is a useful service, this evidence is 'not good enough' for Tricare," wrote Gordon, who declined to explain his comments further in an interview. He wrote that the ECRI study was "designed to reach a negative conclusion."
ECRI also asked two additional researchers to examine the report, John Corrigan, director of the Ohio Valley Center for Brain Injury Prevention and Rehabilitation in Columbus, and John Whyte, the director of Moss Rehabilitation Research Institute in Pennsylvania, both leading researchers in the field.
Both men declined to comment, citing their contractual obligations with ECRI, and Tricare declined to release their reviews. People familiar with their contents said Corrigan and Whyte closely mirrored the views of their fellow critics. They recommended that ECRI use a different method to judge studies of cognitive therapy, but the institute refused.
ECRI "said thank you very much, but we're not changing anything," said one person familiar with the review process.
More Studies, More Waiting
In an interview, ECRI Institute officials defended their firm's methodology. The system is designed to provide a rigorous review free from researchers' bias, they said.
Karen Schoelles, ECRI's medical director for the health technology assessment group, acknowledged that some of the institute's criteria — such as accepting only studies with 10 or more patients — were "arbitrary." But she said they were widely accepted in the assessment industry.
She also noted that Tricare officials were aware of the criteria and made no attempt to change or adjust them. Tricare used ECRI Institute for almost 10 years to carry out health reviews, though the agency recently terminated the contract and selected a new firm to carry out assessments.
Cognitive rehabilitation "may be on to something," Schoelles said. "But it needs more research."
Schoelles acknowledged that ECRI's own reviewers had criticized the report. ECRI offered to provide copies of the reviews, but later said that Tricare ordered them not to release them.
Stacey Uhl, the lead researcher on the review, said the criticism did not change her view that randomized controlled trials were the best way to assess the quality of evidence.
She noted the review found evidence that cognitive therapy did help in some way and said she would not rule out seeking such care for a loved one.
"I as a parent would want my child to receive all available therapies," she said.
DeMartino, the Tricare official who commissioned the report, acknowledged the outside reviewers had "very, very strong opinions" that were "of concern."
He said Tricare was conducting a review to determine whether ECRI's techniques were best suited to measure cognitive therapy's benefits. He denied submitting cognitive therapy to overly-strict review standards.
"You get what you ask for," DeMartino said. "They tell us what they're going to give us, and it's our job to sort of say, "Okay, we understand that within the limitations of their methodology, this is the information that we get."
He added: "The better the information you have, the better that you can move forward and do the best thing." The Tricare reports, coupled with high cost projections, ended the legislative push to get cognitive rehabilitation for service members and veterans.
Last year, Congress ordered the Pentagon to conduct further studies to review the effectiveness of the therapy, but those studies have not yet begun and results are not expected for several years.
Tricare said it would conduct regular reviews to monitor developments in the field. DeMartino first said Tricare would carry out a new review beginning in September. A spokesman later clarified that the National Academy of Sciences Institutes of Medicine would perform the review. It is scheduled to be completed by the end of 2011.
Susan Connors, president of the brain injury association, said she was stunned by the need for legislation at all. As the Pentagon conducts yet more studies, thousands of troops and veterans may be going without the best known treatment available. Thousands more would have to rely on military hospitals or veterans clinics far from their homes, or with substandard programs. The Tricare refusal shut down access to the hundreds of civilian rehabilitation clinics nationwide.
"I'm very disappointed by the resistance," she said. "The military should want to do this."