I recently exchanged some tweets with Stan Kutcher and the “Knowledge Exchange” manager of the Mental Health Commission of Canada (MHCC), Christopher Canning. The heart of the matter was the Hot Idea or Hot Air paper released by Kutcher whose purported “results” were re-tweeted by Canning and journalists Tom Blackwell and Andre Picard:
Purchasers of [either the SOS or Yellow Ribbon suicide prevention] programs should be aware that there is no evidence that their use prevents suicide. (Kutcher et al, 2015)
Now, I have no knowledge of either
the SOS or Yellow Ribbon suicide prevention programs beyond what can be gleaned
from a few minutes on their websites. I would be open to their thoughts on how
they measure their own success (and what they think of Kutcher’s pronouncements
in the name of “science”) but am at the moment in no position to defend them. I
do not know how effective these programs are.
What I do know is that a reading of
Kutcher’s paper begs at least as many questions as it answers.
Methodological tinkering or critical
thinking?
In a characteristic section on the
methodology of a database search, Kutcher drones detail:
The results from the two search terms were combined using the Boolean AND, which were further filtered by key terms “SOS suicide prevention” and “Yellow Ribbon suicide prevention”. Meanwhile, we repeated the first two steps but filtered the results with key words systematic review and meta-analysis, to capture existing reviews/meta-analyses… (Kutcher et al., 2015)
But at the
end of it, we have no actual idea what resources might have been filtered. In
another section, Kutcher announces, “Systematic review or meta-analyses dealing
with suicides linked to physical illness were excluded,” but we have no idea
why such an exclusion was
considered necessary or desirable, or whether the results would be different
had “suicides linked to physical illness” been included. We do not even
know how the suicide-physical illness link is defined or how it was applied.
What we do know
is that two “team members” did data searches, found about 13K+ abstracts after
de-duping, and then narrowed those down to 5, which were then examined
critically resulting in the “no evidence” claim. How did they do this? If
I am understanding it right, the “team” filled out a “data extraction form,
developed a priori” on each abstract
and then applied standards to select a much smaller group of 32 (7 intervention
studies + 25 systematic reviews). After examination of those 32, they applied
more standards and then selected 5 of the studies.
It’s boring to recapitulate tweet
battles, but I have to mention I made a mistake when I said that Kutcher had
performed a meta-analysis in this Hot Idea or Hot Air paper. In fact, Kutcher
never got to the meta-analysis. Instead he reviewed 5 papers after eliminating,
first, thousands he deemed irrelevant, and then 27 he deemed unqualified,
according to certain standards.
One of the curious things about
Kutcher’s methodology is that it utilized standards adopted from a team assembled by George W. Bush shortly after the
passage of the Patriot Act. Bush’s White House Task Force for Disadvantaged Youth developed the OJP
What Works Repository used by
Kutcher’s team. The Obama administration has certainly embraced similar approaches. It’s too tempting for any politician
to pass up: Get up and make a stern speech with a
“we-are-not-going-to-take-it-anymore-attitude” then conclude, “When spending
tax money, measure results and get rid of bad performers.” It’s hard to argue
the opposite.
But it is important to note that
“evidence-based” evaluations are primarily bureaucratic cost-cutting tools.
They can be directed against enemies and skewed to help friends. Minimal
critical thinking dictates that we ask whether evidence-based standards are
being applied evenly. Are “evidence-based results” demanded of every
government-funded sector, or just a choice few? Does the MHCC, for
example, a government-funded organization that supports the work of Canning,
Kutcher and Picard, have any double-blind experiments establishing the
effectiveness of spending millions of dollars on studies and marketing
campaigns, or justifying its claim to be “a catalyst for
improving the mental health system”?
Putting aside these general
observations, let’s look at a particular case.
Evidence debased practice
Let’s say a medical researcher
accepts money from pharmaceutical companies and then seems zealous to dispose
of evidence for the positive effects of non-pharmacological
interventions. Let’s say he was also found sitting on evidence for the
ineffectiveness of pharmacological interventions. Perhaps we should question
his categorical assertions that “there is no evidence” for the effectiveness of
what turns out to be the product of a competitor.
Stanley Kutcher and his team attempt to dispose of evidence for the positive effect of one non-pharmacological
intervention here:
Further, although all the SOS studies failed to find reduced suicidal ideations, they did report reduced attempts. The reported discrepancy between suicide attempts and ideations is in contrast with previous reports (Bridge, Goldstein, & Brent, 2006; Perez, 2005) and the authors’ clinical experience suggest that the two measures should be significantly associated. (Kutcher et al., 2015)
Apparently, in this version of
“evidence-based” theory, the “authors’ clinical experience” trumps data.
Kutcher’s team can simply announce “there is no evidence” because a reported
decrease in suicide attempts (which is data, or “evidence,” regardless
of the weight or interpretation one gives to it) does not accord with expected
results based on personal experience.
Suicidal ideation: The point prevalence of suicidal ideation in adolescence is approximately 15–25%, ranging in severity from thoughts of death and passive ideation to specific suicidal ideation with intent or plan (Grunbaum et al., 2004). The latter is much less frequent, with annual incidence rates of 6.0% and 2.3% in adolescent girls and boys, respectively.
…Longitudinal studies have shown that the more severe (high intent or planning) and pervasive (high frequency or duration) the suicidal ideation, the more likely such ideation is to eventuate in an attempt (Lewinsohn et al., 1996). Attempters who show persistent suicidal ideation, particularly with a plan or high intent to commit suicide or both, are at increased risk to reattempt (Goldston et al., 1999;Lewinsohn et al.,1996)
Kutcher’s paper makes no mention of
these two types of ideation, one type non-predictive (passive ideation),
or of the fact that Bridge, Goldstein & Brent consider the best predictor of suicide to be previous suicide attempts:
Previous suicidal behavior: A prior suicide attempt is the single most potent risk factor for youth suicide in both case–control and prospective studies, elevating the risk of asubsequent completion 10–60 fold. (Brent et al., 1999; Kotila & Lonnqvist, 1989; Marttunen, Aro,& Lonnqvist, 1992)
Yet Kutcher et al seem to be
saying that a decrease in reported attempts in the absence of a decrease in
suicidal ideation should be ignored because it is not in accord with the
authors’ personal experience.
None of this is to say that the
paper does not raise legitimate questions about “self-reported” suicide
attempts as opposed to documented interventions. Again, I am in no position to
argue the effectiveness of the programs or the reliability of the data
supporting them. But Kutcher and team do have a knack for turning the
lack of a measurement into evidence for the absence of effectiveness.
So, it would seem that Kutcher et
al might be a bit zealous to dispose of evidence for the positive effects
of non-pharmacological programs. But is there any indication that Kutcher
ever failed to publish evidence for the ineffectiveness of
pharmacological interventions? As a matter of fact, there is.
Paxil Studies 329 and 377
Back in 2011 Stanley Kutcher was running as the federal Liberal Party candidate in Halifax when he
Paxil Studies 329 and 377
Back in 2011 Stanley Kutcher was running as the federal Liberal Party candidate in Halifax when he
Why was it
lucky? Because if Thomas Bousquet, in the Halifax weekly The Coast, within days
of an election, had not quoted
Alison Bass, award-winning author of the book Side Effects, when she said that Study 329, and implicitly Kutcher as
co-author, had “essentially lied,” it would have been much harder for Kutcher’s
legal team to force The Coast to apologize and remove Bousquet’s entire
article, which it did, using legal threats referencing special libel conditions
before an election.
This too was
lucky, because it has allowed Kutcher to appear vindicated ever since when the
main point of Bousquet’s article is unassailable: Study 329, co-authored by
Kutcher, was funded by GlaxoSmithKline, and when raw data tended to show that
GSK’s anti-depressants were no more effective than placebo, and that teens who
took the drug became more suicidal than the placebo group, a way was
figured out to manipulate the data and write the experiment up as a success.
The evidence emerged back in 2004 when NY AG Elliott Spitzer filed a consumer fraud suit against GSK, the brainchild of NY AAG Rose Firestein. As the WSJ put it:
The New York lawsuit describes five Glaxo studies of use among children and adolescents. Two of the studies failed to show that Paxil was more effective than a placebo for treating depression in children and adolescents, according to the suit. In one, the placebo actually outperformed Paxil on a primary efficacy measure, the suit says. And three of the studies showed that certain possibly suicide-related behaviors were about two times more likely among the Paxil users than the others. In one study that wasn't published, 7.7% of the youth on Paxil had behavior that included "suicidal thinking and acts," compared with 3% of the placebo group, according to the suit.
An internal Glaxo document said the company would have to "effectively manage the dissemination of these data in order to minimize any potential negative commercial impact," according to the suit. That document recommended that Glaxo publish a full article only on the one study out of the five that had some favorable conclusions. Soon after, that study was published in Journal of the American Academy of Child and Adolescent Psychiatry.
I think that
“full article” would be the
report on Study 329, co-authored by Stan Kutcher and the
very subject of Bousquet’s retracted article in the Coast.
The suit was
settled for a pittance, but Spitzer forced GSK to release all the buried Paxil
studies. Some embarrassing GSK internal memos also emerged:
According to an internal email obtained by the BBC, GSK executives were well aware that Paxil wasn't performing…"Essentially the study did not really show it was effective in treating adolescent depression, which is not something we want to publicize," reads the memo (Halifax Media Coop, 2011)
One of
the un-publicized findings was Stan Kutcher’s Study 377:
Kutcher was also involved in study 377. The results were dismal, showing that Paxil was no better than sugar for treating depression in youth. While 329 was subject to a market-friendly makeover, 377 was suppressed by the drug company. It didn't see the light of day until the New York district attorney's office forced GSK to release it.
377 discloses each author's involvement in the pharmaceutical industry. Of Kutcher, it lists that he had been a paid consultant for GlaxoSmithKline. It also says that he had "received research grants from, has been a consultant for, or participated on advisory boards of" pharmaceutical heavy-weights GSK, Pfizer, Eli Lilly. He disclosed nine drug companies in total. (Halifax Media Coop, 2011)
Bousquet’s
article didn’t even mention Study 377. In 2012 the U.S. Justice
Department announced that GSK had agreed to plead guilty and pay a $3
billion fine, in part for promoting the use of Paxil for children. The
suppression of Kutcher’s Study 377 was a major part of that case.
Tweeting knowledge or tweaking truth
Now to
answer my own tweet, “Is Stan Kutcher doing bad science?”
First I
would say that Kutcher has certainly appeared capable of “bad science” in the
past based on his involvement in a famous case of withholding evidence
resulting in a guilty plea by a major drug company and, arguably, in suicide
promotion.
Second, I
don’t know yet whether Kutcher’s current attempt to de-fund two popular suicide
prevention programs is “bad science” because I am not sure it is science at
all. It does appear somewhat “nasty
and brutish” to this critic. It is certainly questionable and does not
deserve the unquestioned re-tweeting it gets from Blackwell, Canning and
Picard.
The bigger story: none of the antidepressants reduce the suicide rate, either.
ReplyDeleteRight! In a nutshell.
ReplyDeleteSuperbly written article, if only all bloggers offered the same content as you, the internet would be a far better place..
ReplyDeleteWilmette Psychotherapy