Bayesian? Or data driven?

The Watchman is a clever widget, implanted via the femoral vein, for stopping atrial fibrillation-related blood clots floating off and causing strokes. It is already licensed for those who cannot tolerate warfarin, but the big money will come if it works better than warfarin and can be used for everyone.

The first trial (click here for the 2009 Lancet paper) had been registered here with a planned sample size of 1550. With a randomisation ratio of 2:1 device:control, we should expect approx 1000 intervention:500 control.  The published numbers were 463 intervention: 244 control. On the face of it not good.

However a trial description had also been published in 2006 (click here) which pre-specified a Bayesian analysis testing for device non-inferiority. The minimum of 300 participants followed up for one year, and 100 for two years pre-specified there, was achieved. So OK. Back on track.

But the trial failed to convince the FDA. Perhaps they were annoyed that the analysis plan was not clear on the trial registration site. Perhaps they just found the Bayesian analysis confusing. Forgivably so! Here’s an extract:

“The pre-planned primary analysis of efficacy and safety used a Bayesian Poisson model, stratified for CHADS2 score, with a non-informative gamma conjugate prior distribution.”

Urgh!  Anyway the FDA demanded another trial. It’s called PREVAIL and the researchers are making a complete hash of it.

It was registered here on August 12 2010.  The original primary endpoint was:
“stroke, cardiovascular death and [presumably or] systemic embolism” by six months.

On March 1st 2013 this was changed to three co-primary endpoints of:
“7-Day procedure rate of death, ischemic stroke, systemic embolism and complications requiring major cardiovascular or endovascular intervention;”
“Comparison of the composite of stroke, systemic embolism and cardiovascular/unexplained death”;
“Comparison of ischemic stroke and systemic embolism occurring greater than 7 days post randomization”.

These three new co-primaries were to be measured at 18 months. i.e they’d altered the timing of the original primary outcome, and added two more!

On the face of it this is cheating. The trial started in Nov 2010, with 18 month follow-up by Dec 2013. By March 2013 the triallists would have had the original, six month primary outcomes, so a change at that stage is data driven!

However, according to this blog:

“[…] Boston Scientific told me that the endpoint changes had been put in place several years ago by Atritech, the company that developed Watchman and which was acquired by Boston Scientific. Atritech, the representative said, had neglected to update the PREVAIL ClinicalTrials.Gov website.”

I guess cock ups happen. But the trial is not double blind, so the investigators would have known how the data were going long before any formal analysis. Let’s hope the relevant email trails can show the changes were not data driven.

There are other problems. The planned sample size was 475. But Boston Scientific are already leaking data to their shareholders here (accessed 17 March 2013). According to that 407 patients were randomized 2:1 device v warfarin control.

The first co-primary <7 day safety data were not presented by group, but the event rate in the device arm was 2.2%. For the second and third co-primaries data were limited because only 88 patients had reached 18 months, but risk ratios were presented!  For the new second co-primary, i.e. the original primary endpoint measured at 18 months, the risk ratio is 1.07, device versus control.  For the third co-primary, the observed difference in adverse event rate between the device and control group was 0.0051 per 100 patient years. It’s is not clear in the data released to investors if this favoured the device or control. However, a slide presentation for the American College of Cardiology meeting in March, which was never presented because a press embargo had been broken (That’s another story. Details here!) gives a clue.  I got my copy from here.  Here it is, PREVAIL slides pdf. The results favour controls.

Although I can’t find any published analysis plan, this slide set suggests that PREVAIL is also Bayesian. Slide 28:

“Bayesian piecewise exponential technique used to model 18-month rates, with the historical priors based on data from the previous pivotal trial, PROTECT AF”

So that’s it.  PREVAIL is a Bayesian trial using the PROTECT trial’s posterior credibility intervals as historical priors.

In theory that’s exactly how Bayesian trials should be analysed. But we still need a data independent analysis plan, and we need the triallists to keep to it. Otherwise they have too many chances to get the “right” result.

Here’s one area of possible muddle. Since PROTECT is still accumulating data, which results will form the priors for PREVAIL?  The Lancet ones, or the ones on the day of analysis?

Eventually PREVAIL will be published in all its glory, and perhaps all these questions will be answered. But unless the authors have some date-stamped analysis plans, I suspect the FDA will ask them to try a third time.

Personal note

I’m an obstetrician, not a statistician, but I’ve had my own fingers burned by the reverend! Ten years ago my colleagues and I conducted a trial (GRIT) of timed delivery for premature babies failing to thrive in utero. Click here for the main Lancet report. Full text grit-Lancet-2004 for those without institutional access.  The trial was registered here, and we analysed it in a Bayesian way, following a published analysis plan (click here, or GRIT Bayesian analysis). It’s still the only trial of this important intervention, but it’s influence has not been great. One reason was that it was a negative trial, but another may have been that the Bayesian analysis was confusing. In our case we used three different prior beliefs (enthusiastic for delivery, for delay, and sceptical), which we said different obstetricians held, so we ended up with three different results!

Randomised trial have two jobs to do. The first is to give an unbiased estimate of treatment effect. Frequentists and Bayesians can both do that.  The second is to convince practitioners. Frequentists are better at that.

Jim Thornton