We’ve all heard the story. An experimental cancer treatment shows promise in preliminary testing, and then larger trials find no actual improvement over the standard of care.
Everyone’s disappointed. The people living with cancer, waiting for a better — or any — treatment. The researchers hoping to bring new options to their patients. And the taxpayers, who, often, foot the bill for trials.
But there’s a positive side to negative results, those clinical trials where the treatment under study is worse or, at least, no better than currently available therapies, said Dr. Joseph Unger, a biostatistician at Fred Hutchinson Cancer Research Center. He and his colleagues found that, among all the publications resulting from large cancer clinical trials, negative trials have just as much scientific impact as positive findings — those trials where the experimental treatment tested better than the standard of care.
“Investigators want to make new discoveries,” Unger said. “There’s probably a preconception that negative trials are in some ways a failure, but it turns out as far as their scientific impact goes, they’re very meaningful.”
In a study published Thursday in the journal JAMA Oncology, Unger and other researchers from Fred Hutch, Columbia University Medical Center and Oregon Health & Science University outlined their findings, the good and the bad. To evaluate the impact of clinical trials, they looked at publication citation data, or the number of times scientific articles emanating from clinical trials were referenced by other research publications.
The researchers delved into data from a series of large cancer clinical trials conducted in the U.S. between 1985 and 2014 by SWOG, a nationwide cancer clinical trial consortium of which Unger is a biostatistician and whose statistical center is housed at the Hutch. At first, Unger was interested in examining how often trials are positive overall. He found that SWOG’s clinical trials showed similar trends to what others had seen: The rate of positive trials was about 30 percent.
Then Unger and his colleagues decided to look deeper at what happened after those trials concluded.
During the 30-year period examined, SWOG completed 94 randomized Phase 3 clinical trials — the largest and most definitive type of clinical trial conducted in the U.S. before a treatment is brought to market. Such studies are designed to show whether a treatment works, or doesn’t. Those 94 trials tested therapies for patients with a range of cancer types and involved a grand total of 46,424 patient volunteers, randomized to receive either the experimental treatment or the standard-of-care treatment.
Of those 94 trials, 28 closed early, mainly due to a lack of volunteers, so we don’t know the answers to the scientific questions those trials posited. Of the remaining trials, 26 showed positive results and 40 had negative findings, where the experimental therapy didn’t work as hoped.
Unger and his colleagues looked at the 273 scientific articles published using data from those 66 positive and negative cancer trials and used Google Scholar (Google’s academic literature search engine) to ask how many times each article was cited by later articles in the field — a rough approximation of how important other scientists found each publication.
“[Citations] are a metric that represents a construct that is kind of hard to measure, which is impact, scientific impact,” Unger said.
When it came to primary articles — those papers reporting actual trial outcomes — positive trials did tend to get more citations than negative trials, they found. From 1995 to 2014, the primary papers about positive-trial results garnered more than 15,000 citations (or an average of 43 per year for each article), while those about negative findings got just above 9,100 (an average of 21 per year per article). These primary, positive-trial articles also tended to be published in higher-visibility, higher-impact medical journals.
But when the researchers added in the “secondary” articles — that is, publications about anything learned from the trial’s results, from new biomarkers to new hypotheses — the picture changed. On average, there were 55 citations per year for each article for positive trials, and 45 per year for the negative trials, a difference the researchers found was not statistically meaningful. The sum of all the citations for both primary and secondary articles was nearly identical for positive and negative results, with over 21,000 citations for each.
All those numbers add up to one overarching theme: The redemption of the negative finding.
“Negative trials matter because they tell us what doesn’t work — which can be as important as what does,” Dr. Dawn Hershman, a Columbia University oncologist and one of the study co-authors, said in a statement.
And it’s not just academic. Negative results can have real, immediate impact for patients, too. Take the example of a drug that’s being prescribed off-label for a given condition, a relatively common situation, Unger said. Now, imagine that drug is tested in a rigorous Phase 3 trial for that condition.
“If a trial like that turns out to be negative, and shows that the new drug doesn’t work, that can be beneficial to patients,” Unger said. “They may have been spending a lot more money on this drug that people think is working when, in fact, it isn’t, or they can be experiencing some toxicities from this drug that they wouldn’t otherwise, or they can have their hopes raised about a cure in a way that turns out to be unreasonable and not borne out by the results.
“So it’s another reason, I think, patients would be comforted by the idea that contributing to a trial, regardless of how it turns out, is really important. Because the information is vitally important, whether it ends up being positive or negative,” he said.
There is a third class of trials that Unger views as true failures — and the study bears this out. These are the trials that close early, usually due to a lack of volunteers. Among the 94 SWOG trials in the study, nearly 30 percent fell into this category.
“The trials that have the least impact are those that are never finished,” Unger said.
He hopes his study will change perceptions of clinical trials, both the positive studies and negative findings, and help all communities involved in medical research — patients, clinicians and the researchers themselves — understand that contributing to a clinical trial will have value, no matter the outcome, as long as there is an outcome.
“Sometimes, studies fail after a lot of resources and a lot of patient time and effort and investigator time and effort were put into them,” he said. “The most important thing is that when a study is launched, it completes, because if it does complete it’s going to have a meaningful impact either way.”
Funding for the study came from the Dr. Charles A. Coltman Jr. Fellowship of The Hope Foundation, the public charity for SWOG; and from the National Cancer Institute's Community Oncology Research Program Research Base grant.
Rachel Tompa is a former staff writer at Fred Hutchinson Cancer Center. She has a Ph.D. in molecular biology from the University of California, San Francisco and a certificate in science writing from the University of California, Santa Cruz. Follow her on Twitter @Rachel_Tompa.
Every dollar counts. Please support lifesaving research today.
For the Media