Photo credit: Pexels (open source) https://www.pexels.com/photo/person-holding-black-vape-3545426/

The Cochrane Library’s Tobacco Addiction Group has just published its third update of the evidence about the usefulness of e-cigarettes (ECs) in quitting smoking. The update considered 50 completed studies, which together had 12,430 participants. Twenty six of these studies were randomised controlled trials (RCTs). However, just four studies were considered at low risk of bias (such as non-randomization) and these four formed the basis for the report’s main comparisons. They considered only reports which assessed smoking status at a minimum of six months from when the participants started using the ECs

The review concluded that there was “moderate-certainty evidence, limited by imprecision, that quit rates were higher in people randomized to nicotine EC than in those randomized to nicotine replacement therapy (NRT) (risk ratio (RR) 1.69, 95% confidence interval (CI) 1.25 to 2.27; I2 = 0%; 3 studies, 1498 participants). In terms that ordinary folk might better understand “For every 100 people using nicotine e-cigarettes to stop smoking, 10 might successfully stop, compared with only six of 100 people using nicotine-replacement therapy or nicotine-free e-cigarettes, or four of 100 people having no support or behavioural support only.”

Put another way, if we take 100 smokers participating in an RCT, 90 would still be smoking six months later if they used e-cigarettes, compared with 94 who used NRT, and 96 who just tried to quit alone or got some “behavioural support”.

I’ve tried hard to think about this, but I cannot come up with any drug, used for any purpose, which has any remotely more dismal success rate than e-cigarettes or NRT in achieving its main outcome. If you went along to your doctor for a health problem and were told “here, take this. It has a 90% failure rate. But I’m describing it as successful.” …what would you think?

Well, here’s how the new Cochrane report was reported:

“Vapes more effective to quit smoking than gum or patch, review finds” Reuters

“Updated Cochrane Review shows electronic cigarettes can help people quit smoking” Eureka Alert

“E-cigs better than gum or patches” Hippocratic Post

And below, how indefatigable vaping advocate, Alex Wodak, greeted the news.

But it’s far worse than this. Far worse. There are many reasons why randomised controlled trials in smoking cessation tell us very little about how well these products perform in the real world, away from trial conditions.

And that’s the only world that’s worth thinking about when we ask basic questions about whether e-cigs are game changers in helping people quit smoking.

Problems with RCTs

Randomised controlled trials  (RCTs) are revered in experimental and clinical science as being “gold standard” evidence about whether an intervention (often a drug) makes a difference to outcomes of interest, such as smoking cessation.

RCTs can compare a drug with another drug used for a similar purpose, with a placebo, or with “usual care”. Usual care in smoking cessation RCTs can be the sort of advice that a doctor or other health professional might ordinarily offer to a smoker when they were not participating in a study. Given that such advice is often given, especially when a medication is involved, it is important to assess whether the medication has any additional cessation effect on top of the advice or routines to which smokers would ordinarily be exposed in their interactions with a health care provider or service.

But when smokers access drugs in real world circumstances, they may receive no support or advice (for example, when buying NRT from a supermarket or e-cigs over the internet) or only brief, sometimes perfunctory advice when a health care provider or pharmacist is too busy to spend much time with a customer or patient. A NSW survey of 700 pharmacies reported that pharmacists claimed to spend an average of five minutes discussing stop smoking medications with smokers. meaning that many would spend less time than that.

Once a sufficiently large number of participants have been selected to participate in an RCT, they are randomly allocated into treatment or comparison/control arms of the trial. Ideally, the allocation should be done by randomization software and by someone not associated with the trial but by third parties with no interests of any sort in the outcomes of the trial.

Those conducting RCTs can recruit their participants in a variety of ways, some of which introduce important biases in the study population. In the smoking cessation field, we often see subjects recruited from sources like quit smoking clinics, telephone quitline callers, general practitioner and other primary health care patients, smoking cessation or vaping website and chatroom visitors. With each of these, we need to ask whether smokers recruited in such ways are different in important ways to randomly selected smokers in the population at large. Self-selection bias is very relevant here. We are likely to be dealing with those who are more help-seeking. This may mean they are more motivated to quit than smokers in the general population, and it may mean they are people with lower self-efficacy (lower confidence in their ability to quit unaided).

Often researchers attempt to address this concern by demonstrating that those who have been recruited into trials are comparable to smokers in the whole population on a range of variables like demographics, smoking history, level of nicotine dependency, intention to quit and so on. But, beyond all these comparabilities, in a very important respect they are different: they have often taken steps at help-seeking in their hopes to stop smoking. But the great majority of smokers who quit don’t seek help to do so. So those who volunteer to take part in trials recruited in these ways are help-seeking volunteers.

Trial exclusion criteria

Those running trials will often exclude people from trial participation for a variety of reasons. Those who have language problems are often excluded as interpreters are expensive to add to study budgets. Those with drug or alcohol dependency, serious mental health problems like depression, psychosis or bipolar disorder can also be excluded, as can those with no fixed address, or who move addresses often, are in prison or who have a serious illness which might reduce their life expectancy (and so participation in the study down the track. Those with low motivation to quit can also be excluded.

One study reviewed 54 RCTs smoking cessation trials for criteria for exclusion and found 25 separate criteria being used across these trials. They then applied 12 of the most commonly used of these criteria to 4,962 adults with nicotine dependence in the past 12 months from a US national survey  on alcohol  use  (NESARC) and to a subgroup of participants motivated to quit. (see table below).

They found two-thirds of participants with nicotine dependence would have been excluded from clinical trials by at least one criterion, with 59% of the subgroup of motivated to quit smokers  also  excluded.

Exclusion variable	Current nicotine dependence (n=4962)	Motivated to quit smoking (n=4121)
Pregnancy	3	3
Cardiovascular disease	7	7
Smoking <10 cigs/day	32	34
Current/past 6m use of any psychotropic medication	NA	NA
High alcohol consumption	14	13
Not motivated to quit	18	0
Use of other drugs	3	3
Current depression	17	16
Current/past 6m use of bupropion and/or NRT	NA	NA
Eating disorder	NA	NA
History of psychosis	2	2
History of bipolar disorder	10	10
Exclusion by any criterion	66	59

Trial subject retention strategies

Those running trials put a lot of effort into maximising trial cohort retention rates. If lots of people drop out of the groups being studied, this can greatly compromise the integrity of trials, as important questions can be asked about whether those who pulled out or were lost to follow-up differed in important ways to those who remained in a study across its entire course.

Real-world studies have found high levels of premature discontinuation of medication use. A four nation study of 1,219 smokers or recent quitters who had used medication in the last year (80.5% NRT, 19.5% prescription only found most (69.1%) discontinued medication use prematurely (71.4% of NRT users and 59.6% of bupropion and varenicline. NRT users who obtained their patches or gum over-the-counter without prescription were particularly likely to discontinue (76.3%).

Lots of wisdom has accumulated in professional trial communities about cohort retention. Strategies include reducing any barriers to participation, efforts to building a sense of community and belonging among trialists, follow-up and reminder strategies, and tracing techniques. Community building strategies can be particularly important and trial staff who have good “people” skills are particularly important. This often fosters positive attitudes and a sense of responsibility among participants toward helping the trial avoid low levels of dropout. They can be made to feel important that they are contributing to the advance of science and the health of communities.

Trial staff often include young investigators whose PhD work is focussed on a trial. They have particularly strong motivation to develop good personal relationships with participants as the work they do will be assessed by their thesis and publication reviewers and major problems like high dropout rates can be fatal to publication. Someone mildly irritated with the on-going demands of a study to complete questionnaires, provide biological samples and keep personal data records may feel a sense of “that lovely young researcher who contacts me every few months would be very unhappy if I pulled out”. Strategies like sending thank you, birthday and holiday cards, trial newsletters, supplying trial logo material like caps and T-shirts are also often used. None of this happens in the real world when people start vaping or using NRT they might buy in a supermarket.

Trialists are often paid and drugs are free

The drugs used in trials are given free-of-charge to participants. Even where governments subsidise the cost of approved prescribed medications, the drugs are never free, and to those on low incomes, can still constitute a significant outlay. This may inhibit them being used into the medium or longer term by those who feel they need to continue using them.

It is also increasingly common for trialists to be paid for their participation in trials. This is intended to act as both fair compensation for their time and any out-of-pocket expenses like travel to the research unit, but may also act as an incentive to continue participation, particularly for those on low incomes or who are unemployed.  In real world, unmonitored or unsupervised quit attempts, smokers are never paid to use quitting aids. These differences may give an extra boost to full compliance across the recommended course of smoking cessation aid use, something that is often far from the case in real world use.

Blindness integrity problems 

In most RCTs, participants are not told whether they have been randomised to receive the active or placebo (control) drug. This is called subject “blinding”: they are blind to whether they are getting the active drug or the dummy, inert, control drug. Sometimes, investigators are also blinded as to whether subjects have been allocated to a particular treatment arm to which each study participant has been added. This is called double blinding and is undertaken to remove the possibility of researchers actively or inadvertently communicating expectations of effects to study participants. A researcher who might have hopes that a particular treatment is efficacious and who knows that particular study participants have been allocated to the active drug, may make comments to these patients that suggest to them it is likely that they are on the active drug. Researchers with expectations that successful outcomes of a trial (ie where the active drug is shown to be far better than a placebo) might lead to valuable, career-enhancing opportunities may sometimes be tempted to compromise the integrity of the blinding of a trial.

Nicotine replacement therapy and vaping are  strong candidates for a failure in blindness integrity. Nearly all smokers have often experienced interoceptive cues that they are craving nicotine. Here, we need only think of the speed with which many smokers light up a cigarette soon after waking each morning, the common sight of smokers rushing to light up after alighting from non-smoking public transport and standing outside office blocks and restaurants. These commonplace sights tell us that smokers are very familiar with sensations that remind them  of their need to re-dose with nicotine and the relief and pleasant sensations they experience shortly after doing this.

So when smokers, who may have had their brains pickled in nicotine for years, get allocated to the placebo (non-nicotine) arm of a trial of NRT or e-cigs, guess what? Many of them very quickly twig that they are in the control arm of the trial: they are not getting the “good stuff” that the trial is testing to see if it is effective. Their body tells them they are not getting nicotine.

 This paper looked at this issue. Of 73 trials it reviewed, only 17 made any assessment of blindness integrity (the others didn’t even consider it). And of the 17, 12 reported that the participants guessed correctly which arm of the trial they had been assigned to.

When considered together, all the above problems make RCTs on smoking cessation a very, very far cry from the way smokers use NRT and ecigs in the real world. But this, as we saw, will not stop the headlines about effectiveness, as if these artificially constructed trials bore any resemblance to the spread and conditions of use in the real world.

When we look at the best of the cohort studies that follow groups of smokers over time, we get a very different perspective of how well these nicotine replacement methods work in reality.

A US paper by Coleman et al using PATH data reported on a 12 month follow-up of 2932 vapers. The table below shows that for every person vaping at the start of the study (wave 1) who benefited across 12 months by quitting smoking, there were 2.1 who either relapsed to or took up smoking. By far the most common outcome was that those who were smoking and vaping at the beginning of the 12 months study period were still  vaping and smoking at the end of the 12 months. That might suggest that the vaping holds far more in smoking than it tips out of it.