Scrutinizing the study on cancer & exercise

"It was a great effort. The story is delightful. But liking the conclusion is not a reason to stop thinking."

This is what greeted many of us in our morning mail and social media accounts today. Overwhelming enthusiasm about what was described as:

• a “landmark” study (by the 1440 newsletter)

• a first-of-its-kind study

• a definitive test

• exercise better than drugs in cutting the risk of cancer returning

• a new standard of care for cancer

The senior author of the paper in the New England Journal of Medicine (NEJM) said:

“We now have definitive evidence that exercise is not just an intervention for quality of life and fitness. This is an intervention that improves survival and should be standard of care,” said Dr. Christopher Booth, the senior author of the paper and a professor of oncology at Queen’s University in Canada.

Dr. John Mandrola, in his Sensible Medicine newsletter, began with what looked like more of the same:

The CHALLENGE trial tested cancer outcomes from a structured exercise program. The NEJM published the study, more than a 100 news sites covered it, and hundreds reposted it on social media.

So you know the results were positive. Hugely positive, in fact. And who, I ask, does not love the story that exercise vanquishes cancer?

But Mandrola, who is known for his in-depth analyses of studies and claims made about them, then shifted.

While I want this story to be true, there are at least seven reasons to be cautious. These are mostly internal validity concerns. But there are also major external validity issues as well.

You should read his complete analysis, but his seven reasons include these concerns:

1. He calls the 37% reduction in all-cause mortality “implausible and rivals many proven cancer therapies.

2. For a study looking for the benefits of exercise, this one reported “almost no between-group differences in typical exercise parameters…zero differences in body weight, waist circumference and a mere 30 meters longer distance in the 6-minute walk test.”

3. Questions about how well the two groups were randomized, “which is not surprising given the fact the ambitious complicated trial took 15 years to enroll.”

4. “Poor adherence to the exercise regimen further reduces plausibility. Nearly half of the patients in the exercise program did not complete the treadmill protocol at three years and a third did not complete the 6-minute walk test. These patients were included in the intention-to-treat findings—and would have the effect of reducing between-group differences in exercise.

5. Despite their best efforts, the researchers struggled to recruit enough people. “This reduces statistical power and raises the possibility of false positive findings—which is consistent with the biological implausibility.”

6. In the study, “the structured exercise group received an incredible amount of intervention in both behavioral modification and exercise. This makes performance bias highly likely, as evidenced by the large differences in the quality of life questionnaires.”

7. Questions about the “external validity or generalizability” of the trial. “Even if you accepted the results as presented it would apply to a fraction of patients with colon cancer.”

Mandrola concludes:

The cost and healthcare system implications of accepting this protocol would be massive. In the same way that regulatory trials for drugs or devices require multiple positive trials, we should feel the same about CHALLENGE.

It was a great effort. The story is delightful. But liking the conclusion is not a reason to stop thinking.

If you didn’t see any of this kind of analysis in your news stories or social media accounts, that’s not surprising. Mandrola has a really good eye. Of all the news stories I saw, a couple came close to touching any of the stated concerns.

The New York Times included:

Of course, the real-world impact of such an intervention will depend on how many people can take up and stick with an exercise program. This was a clinical trial in which patients were slightly younger and healthier than the typical cancer patient might be, and they were also already motivated to exercise. Even the control group increased their exercise levels…though the treatment group increased them by much more.

NBC News offered this:

Still, it will be challenging to help patients integrate more consistent exercise into their lives after cancer, especially if they don’t have access to a coach or were not very active prior to their cancer diagnosis. It’s also unclear what benefit exercise would have on extending the life of someone with later-stage colon cancers, which can be incurable.

I applaud any caveats that pull in the reins on breathless enthusiasm when big study results are published and when all news sources and social media publishers seem compelled to jump on the publicity bandwagon.

Yes, exercise is a common sense approach. Yes, there are few downsides to exercise done reasonably. But no, no single study is the be-all and end-all. And I think Mandrola’s scrutiny is a good lesson in how to not stop thinking.

---