Adam Zivo: New ‘landmark’ safer supply study is junk science

Last month, the B.C. Centre for Disease Control (BCCDC) announced it had conducted a “landmark study” showing that access to “safer supply” drugs—free, government-funded substitutes for potentially tainted illicit substances—reduces mortality rates among drug users. While the study has been widely promoted in the media and by harm reduction activists, some experts say it is actually inconclusive and misleading.

The study arrives at a particularly sensitive time for safer supply advocates, who claim that the government can reduce overdoses and deaths by providing safer supply.

Over the past year, dozens of addiction experts have said that not only are safer supply drugs being widely resold on the black market, but that there is no actual evidence that safer supply works given that supporting studies typically use weak methodologies that would be unacceptable elsewhere in health-care research.

Federal funding for over 20 safer supply programs is set to expire next month with no word of renewal. As criticism of safer supply continues to intensify, there has been increasing pressure to prove that the controversial strategy is worth supporting.

The BCCDC researchers behind the new pro-safer supply study, which was published in the British Medical Journal, examined the anonymized health data of 5,882 individuals over an 18-month period (between March 2020 and August 2021)—all of whom were diagnosed with opioid or stimulant use disorders.

Some of these individuals received safer supply drugs while others did not. Those who received safer supply were given either opioids (i.e. hydromorphone, which is roughly equivalent to heroin) or stimulants (i.e. Ritalin and Adderall) or a combination of both. The opioids were intended to replace street fentanyl or heroin, whereas the stimulants were meant to replace potentially-contaminated cocaine or meth.

Upon analyzing this data, the BCCDC researchers determined that receiving safer supply stimulants had no significant associations with reduced mortality. However, recipients of safer supply opioids were 61 percent less likely to die (and 55 percent less likely to die of an overdose specifically) within one week of receiving their “safe” drugs. Individuals who received safer supply opioids for four or more days in a single week were 91 percent less likely to die over the following week.

The researchers thus concluded that safer supply is successful.

“This paper is the strongest evidence we have so far, by a large margin, supporting the idea that this can be an effective strategy for reducing overdose death risk,” said Paxton Batch, an expert in addictions medicine at St. Paul’s Hospital and a co-author of the study.

The study immediately received somewhat glowing media attention from several media outlets, including the Globe and Mail and CBC, which repeated the BCCDC’s conclusions and added positive quotes from associated researchers and harm reduction activists. No apparent attempt was made to include real critical voices or interrogate the study’s data and methodology.

Two weeks later, the Toronto Star’s editorial board published an op-ed arguing that funding for safer supply must not be discontinued—and cited the BCCDC study as strong evidence of success.

But had the journalists and editors behind these articles done their jobs properly, they would have noticed that something was off. When I forwarded the study to seven addiction physicians and a professional scientist with working knowledge in statistical analysis, they determined that it was misleading and that its conclusions were unsupported.

Dr. Meldon Kahan, one of Ontario’s leading addiction experts, noted that the study’s authors failed to adequately address the fact that half of the patients who received safer supply had also been co-prescribed traditional, evidence-based medications, such as methadone and Suboxone (also known as “opioid agonist therapy” or “OAT”).

It has been conclusively proven that OAT medications significantly reduce opioid overdoses and mortality among drug users. It would thus be incumbent upon any responsible researcher, when studying a population that is using both safer supply and OAT, to verify whether mortality reductions are caused by one or the other.

While the study made some attempts to use statistical analysis to filter out the effects of OAT medications, there were conspicuous gaps in these efforts. For example, no attempts were made to measure what dose of OAT medications study participants received, even though dosage has a significant impact on mortality.

As safer supply patients are generally prescribed stronger doses of OAT compared to other drug users, this raises an obvious question: were they simply dying less often because they were receiving more OAT?

There are strong reasons to believe that the answer is yes.

The study data showed that safer supply patients who did not receive any OAT medications in the preceding 30 days showed no statistically significant improvements in mortality—which suggests that the benefits touted by the researchers were likely primarily driven by OAT, not safer supply.

This issue was actually flagged in the notes generated by the study’s peer review process (which are publicly available), where I found evidence that external evaluators had repeatedly raised concerns about unaddressed data bias. One reviewer, who had “major concerns” about the paper, worried that it was a “big issue” that safer supply seemed to actually shorten survival once OAT was removed as a confounding factor.

I emailed Bohdan Nosyk, the study’s lead public contact, with a list of questions about his team’s work. He provided a detailed response, which I forwarded to several experts for further review. Upon sending Nosyk a second round of questions and concerns, he declined to provide further answers, citing “a need to attend to our other obligations.”

Nosyk claimed that controlling for variations in OAT dosing was something his research team “couldn’t reasonably build into this initial study” given the complex and dynamic ways that OAT medications are used during treatment.

However, Dr. Kahan counter-argued that Nosyk’s comments “don’t make sense” given that administration of safer supply is similarly complex and dynamic. “If they could control for (safer supply) dose and frequency of pharmacy dispensation, then they could also have controlled for the (OAT) dose and frequency of dispensing,” he wrote to me in an email.

While Nosyk acknowledged safer supply appeared to provide no statistically significant improvement for patients who had not received OAT in the preceding 30 days, he pointed out this did not necessarily mean that there was no improvement at all.

In our correspondence, he linked to an article that argued that “statistical significance” is a harmful concept and that less stringent evidentiary standards should be normalized in science.

Nosyk’s argument was technically correct. An effect is only “statistically significant” if it is strong enough for us to be confident that it didn’t occur by chance. Typically, that means ensuring that there is a less than five percent chance that the effect was a fluke. While these evidentiary standards are the norm in most statistical analyses, they can lead scientists to dismiss very weak effects whose existence is unclear.

However, a debate about the meaning of statistical significance seemed to miss the point. Nosyk’s study had claimed that safer supply led to a 55-91 percent reduction in mortality—a huge effect. But now it seemed that, once an obvious confounding factor was fully removed, benefits were actually so weak that they were barely detectable if they existed at all.

Why weren’t the researchers more forthright about this tension in their paper? Why did they not flag this when speaking to the media? If the benefits of safer supply are so weak that we need to abandon normal evidentiary standards to accept their existence, that doesn’t bode well for the harm reduction movement.

The physicians I consulted with also noted that it was highly unusual that Nosyk’s study exclusively focussed on mortality rates after just one week.

“One-week mortality rates are an unusually short time period to study, and to not report on mortality rates for longer periods leaves one wondering if the data for longer periods did not show any difference between the groups; for example no difference in mortality at a month or year which would make the inferred benefits much less significant,” wrote Dr. Michael Lester, a Toronto-based addiction physician.

Two days after I received Dr. Lester’s note, the scientist who volunteered to assess the study’s statistical analysis confirmed that survival rates for all study participants appeared to have been more or less the same after 52 weeks, regardless if they had accessed safer supply, which suggested that “whatever effect (safer supply) had at the very beginning, hardly mattered at all after a year.”

I asked Nosyk why his team excluded long-term outcomes and only analyzed one-week mortality rates.

“It was important for us to capture the effects of (safer supply) while individuals were receiving it, and measure outcomes in the immediate aftermath of receipt. Just like OAT, or insulin for diabetes, or any number of other medications for chronic conditions, their effects dissipate after discontinuation,” he responded.

His comparison to insulin was illuminating, but perhaps not in the way that he intended.

If a team of researchers examined the impacts of a new type of insulin, it would obviously be preferable to measure both: 1) the immediate impacts of that insulin; and 2) the long-term impacts of its repeated administration. If a research team were to examine only immediate effects, despite having enough data to examine long-term ones, surely that would raise eyebrows?

And surely it would be strange for the researchers to argue, in this case, that this insulin is a promising intervention solely based on immediate impacts, especially if the data suggests hardly any long-term benefits at all.

And yet it seems this is exactly what Nosyk’s research team has done, in its own way.

The physicians I consulted, along with the notes generated by the peer review process, raised a number of other serious concerns regarding the study’s data and analysis.

For example, the British Journal of Medicine’s editors were “puzzled” by the fact that, after one week, safer supply was simultaneously associated with fewer deaths but not fewer overdose hospitalizations. “One would expect these results to be in the same direction,” they wrote.

Some experts I spoke with felt  Nosyk’s team failed to adequately discuss findings on safer supply stimulants—as mentioned earlier, providing such stimulants was associated with no improvements in mortality after one week.

Does this simply mean that safer supply only works for opioids, but not stimulants? Or does this reinforce the argument that the mortality reductions seen with “safe” opioids were actually caused by OAT, as OAT is not used to treat stimulant addiction and thus would not be a confounding factor with “safe” stimulants?

One might expect a fulsome discussion comparing the divergent results of opioid and stimulant safer supply, but that was largely missing.

Interestingly, the fact that safer supply stimulants had no apparent impact on mortality was omitted from the study’s conclusion, which focused exclusively on the benefits of “safe” opioids. As there is a responsibility to spotlight negative results alongside positive ones, one could argue that this omission is subtly misleading, especially considering that many people, including journalists, tend to skip to most studies’ conclusions to understand their findings.

One might ask: why does it matter that this study is misleading? Why does it matter that some researchers claim that safer supply saves lives when their data suggests that this is not actually the case?

The answer is simple: thousands of people are dying every year amid an unprecedented overdose crisis, and if this problem is to be solved, Canada must invest in addiction interventions that work. When bad science creates false hope, it costs Canadian lives.

Despite its flaws, the quality of this study is much higher than what is typically produced by harm reduction activists, so it is unsurprising that many of them are now aggressively promoting this “landmark” research, which they believe vindicates them. But it seems that, as is so often the case with safer supply, the truth is not entirely what it seems.

This study is indeed a landmark, but not because it shows that safer supply works—quite the opposite. A critical look at the data reveals that it is ineffective and that a new path is needed.